Clojure and aplisms

The more I do APL and K, the more I see forks and hooks everywhere else.

That's just a couple of little helpers I've added in some clojure projects. One is "apply-if", which more than from APL, I'm taking it from any procedural language, where you can do `x= if p(x) then f(x) else x`. We used to do this a lot in lua (using ands and ors). But in a combinator style.

Second one is an attempt at the fork pattern.

Look how the avg looks like. it's verbose, but it captures the idea very close to how APL does it.

2 classes of system designers

Here's a devastating quote from Andy Kinslow (NATO SOFTWARE ENGINEERING CONFERENCE ,1968) 

There are two classes of system designers. The first, if given five problems will solve them one at a time.

The second will come back and announce that these aren’t the real problems, and will eventually propose a solution to the single problem which underlies the original five.

This is the ‘system type’ who is great during the initial stages of a design project. However, you had better get rid of him after the first six months if you want to get a working system.

I found it in the Software Engineering conference pdf , but I didn't know anything about that "Andy Kinslow". And looking for that person, led me to some interesting findings, like an APL paper, or finding out that Andy Kinslow pioneered work on Time-Sharing-System, following leads of Corbato and Strachey

DuckDB's arg_max

In kdb+ there's a great way of showing aggregated information called `aj` for asof join. This way of joining data is a neat way to get the "latest record" in a join. 

These kinds of operations are already quite painful to do in plain sql when grouping by. Ever had to get the latest invoice of each customer? probably you had to nest a query on top of another one, or use a fancy "distinct on".

I realized though, that some analytical engines have something called `arg_max`, that can alleviate the pain when doing nested group by. And Duckdb already implements it! (discussion here)

First, let's create a table with a few values ready to group and get the max. Then, we'll want to get the id of the max of every group.   At that point, arg_max just does the right thing.

 

Btw, look at that nice Duckdb's group by all. 

Sum scan in sql

`scan` is an adverb in apl/j/k which is used to reduce an array over an operation, while keeping all the intermediate results. It's used in lots of very creative ways in the array programming languages. Here's some sample examples of "scan (\)".

:x:1+!9
1 2 3 4 5 6 7 8 9
+\x
1 3 6 10 15 21 28 36 45
*\x
1 2 6 24 120 720 5040 40320 362880

In most of those programming languages, scan acts as an adverb, which is a higher order function that modifies a verb. In clojure, you can get similar results with `reductions`.

Anyway, the topic of today is how to accumulate intermediate results like these, in sql.

Well, in the AoC 2022 (day 1), the first part was about summing numbers by paragraphs. 

Some people at duckdb discord are sharing their solutions, and we see it's kinda normal to use `row_number() over() as idx` idiom to add ascending numbers to any projection.

Another trick, to indicate the paragraph number in each line (for future group_by purposes), this is what Teggy writes:

 `select sum(i.value is null :: int) over (order by num)`

Here I show generate_series, row_number, and the sum scan based on "is null".

More on Writing, Notation, Symbols, and Emoji

Since My "On Writing" post, a few posts came through HN that touched somehow on the communication styles, context, and missconceptions on sharing knowledge.

- What happened when I stopped using emojis. That reasonates with my thinking a lot. I'm trying to use only the modern version of '+1' and simple ':)'. The other ones dumb down the expressivity to a baseline that brings nearly 0 info. On the other hand, knowing that old reddit's '/s' has its own family of symbols and meanings made me think that well used, a short symbol can convey the a lot of info (https://toneindicators.carrd.co/). On Context and the power of understanding "where that person is coming from", there's this great example on Greg Wilson's talk What Everoyone in Thech Should Know About Teaching and Learning where it explains how in examining the novice's mistakes, you can infer what he/she didn't understand, and what's the best explanation or example to give in order to solve the enigma and make it 'click'. (As an aside: I think it's a great example of why human interactivity is not redundant once you have docs and wikis.)

Which brings us to APL (again), and its notation. I've been more and more excited by APL, and I'm following now some APLers on youtube, for example Rodrigo Girao which does great short beginner videos on APL solving leetcode problems. Also, subscribed to the shakti mailing list, and reading 'mastering dyalog' when I have some time.

 

On the strict writing side, I found a couple of courses and docs in https://developers.google.com/tech-writing and https://developers.google.com/style/highlights that deserve a good read. I just skimmed them, but well.... something >> nothing.

The Array Cast

There's a new podcast about array programming languages!

https://www.arraycast.com/

In the first chapter, they mention Richard Park's articles and videos porting perlweekly challenges to APL.

https://perlweekly.com/a/richard-park.html and https://www.youtube.com/channel/UCOx-h5m9MeV14rG4PicRB7g . Very very cool stuff around this magical programming language style

Some Thoughts on Ergonomics and Compression

A recent personal take on ergonomics and compression:

Why is vim so addictive, and somehow it's difficult to explain to non vim fanatics? I think it's a reification of the process of walking through a text file and doing stuff to it. A concrete way to talk about processes onto text files. Some feel it is strictly superior to GUIs because you can get ahold of it. you can write it down, put it in a postcard, and you have a handle to it. You type it again and it works. there's no syntax or fuzziness or stuff that can go wrong (TM).

Next question: Can you ahold an algorithm in your head? in the same concrete way you hold 'yypVr-'?

I'm lately quite interested in Array Languages and I'm reading some snippets of Apl/J/K. And the feeling is the same. Everything else looks so wasteful and error prone like the point-and-click looks to vimmers.

Example:
Shuffling a vector – breaking ⍵ down into ⍺ pieces from which another vector is built by merging. E.g. if is 'abcdefghij' and is 3, the pieces are 'abcd', 'efg', and 'hij', and the result is 'aehbficgjd':

⍵[⍋⍋(⍴⍵)⍴⍳⍺]

The ergonomics are pretty different from what we're used to, but there's definitely an emergent property there. It's not just "it's shorter to type".

Perl & Ruby do APL

So recently I saw a couple of posts that mention APL approaches to problems. The cool thing is that I saw those in a Perl related forum and a Ruby one.

- https://www.youtube.com/watch?v=UBl6t7zNfwE
- https://zverok.github.io/blog/2020-05-16-ruby-as-apl.html

John Conway's Game of Life

John Conway passed away recently, so let's look at his Game of Life from the APL/J/K perspective.

https://www.jsoftware.com/papers/eem/life.htm
https://news.ycombinator.com/item?id=22845249
https://www.youtube.com/watch?v=a9xAKttWgP4

And a recent ode to J, that explains the first prototype that Arthur Withney and Ken Iverson wrote in an afternoon. I remember having great fun deciphering it 3 years ago on the flights back and from $WORK at that time, and people staring at me reading that printed code and WTFing.

Transitive closure in k, and more

I found this gist on finding reachable nodes in a graph super entertaining. Every time I read on APL/J/K I feel this intense deep puzzle solving that heats my brain a little bit, but feels awesome when you arrive at the end. And it all makes sense there.

About 10 years ago I was having those feelings with the Shwartzian transform and JAPHs around the net.  Perl is the gateway drug.



Btw, if you want some more, https://www.youtube.com/watch?v=v7Mt0GYHU9A or https://www.youtube.com/watch?v=hzPd3umu78g .

J's awesome

So here are a couple of nice reads that, although they're not new, they somehow teached me something new today about J.


First one is this 9 minute video on the "i." integers monadic verb.  It's super simple. So simple I almost dismissed it as trivial. "Monadic verb that returns an array from 0 to the parameter you pass it", right? Well... There's more than meets they eye.  The trick is that for non-array programmers it's not obvious that the parameter can have many dimensions. And what zeroes mean? and negative numbers?

Next is this super nice interactive J demo.

And a classical: Loopless programming.

some more apl/j/k

 So here are some links about this amazing language, that I'm starting to like more and more, although I see it nearly impossible to do something "useful" with, appart from data crunching. I don't see array languages as general purpose languages but as DSLs for very specific tasks.

Here's some of the links

1 apl/j/k

1.1 https://news.ycombinator.com/item?id=16842378

1.2 http://www.eecg.toronto.edu/~jzhu/csc326/readings/iverson.pdf

1.3 https://github.com/hypotext/notation

1.4 https://github.com/fastai/fastai/blob/master/docs/style.md

1.5 https://github.com/fastai/fastai/blob/master/docs/abbr.md

1.6 http://www.cogsci.ucsd.edu/~nunez/COGS152_S12/TWWT_Ch1.pdf

1.7 https://www.youtube.com/watch?v=9xCJ3BCIudI

1.8 http://code.jsoftware.com/wiki/Community/Creative_Commons

1.9 https://www.reddit.com/r/apljk/comments/7s7evt/an_introduction_to_array_programming_in_klong/

1.10 https://greenjenny.com/2018/05/28/lets-write-quicksort-in-j.html

1.11 https://greenjenny.com/2018/05/28/lets-write-life-in-j.html

1.12 http://code.jsoftware.com/wiki/NuVoc

1.13 http://wordsandbuttons.online/apl_deserves_its_renaissance_too.html

1.13 https://github.com/phantomics/april

1.13 https://alexshroyer.com/posts/2018-06-01-J-is-readable.html

k/Apl info

• APL self hosted compiler https://news.ycombinator.com/item?id=13797797
• APL self hosted compiler screencast https://www.youtube.com/watch?v=gcUWTa16Jc0 
• q manual http://code.kx.com/q4m3/
• ok manual https://github.com/JohnEarnest/ok/blob/gh-pages/docs/Manual.md
• k main page http://kparc.com/
• kona https://github.com/kevinlawler/kona