Just use Postgres for everything!

 Just discovered this post https://www.amazingcto.com/postgres-for-everything/ which is short, and to the point.

Copying it verbatim, because it deserves to be read even if you don't click my links

"

One way to simplify your stack and reduce the moving parts, speed up development, lower the risk and deliver more features in your startup is “Use Postgres for everything”. Postgres can replace - up to millions of users - many backend technologies, Kafka, RabbitMQ, Mongo and Redis among them.

Use Postgres for caching instead of Redis with UNLOGGED tables and TEXT as a JSON data type. Use stored procedures to add and enforce an expiry date for the data just like in Redis.

Use Postgres as a message queue with SKIP LOCKED instead of Kafka (if you only need a message queue).

Use Postgres with Timescale as a data warehouse.

Use Postgres with JSONB to store Json documents in a database, search and index them - instead of Mongo.

Use Postgres as a cron demon to take actions at certain times, like sending mails, with pg_cron adding events to a message queue.

Use Postgres for Geospacial queries.

Use Postgres for Fulltext Search instead of Elastic.

Use Postgres to generate JSON in the database, write no server side code and directly give it to the API.

Use Postgres with a GraphQL adapter to deliver GraphQL if needed.

There I’ve said it, just use Postgres for everything.

"

Also, there's a link to radical simplicity, which seems like another iteration of use-boring-technology.

This one is also great gist about using pg for basically everything https://gist.github.com/cpursley/c8fb81fe8a7e5df038158bdfe0f06dbb

A Few C++ Talks?

 Here are a few talks from the C++ community that when I watched I thought I'd rewatch again because they either are very cool even though I don't grok C++, or they talk about general programming concepts. Here's the list:

- https://www.youtube.com/watch?v=V5SCJIWIPjk Optimizing for Change in C++ - Ben Deane

- https://www.youtube.com/watch?v=2ouxETt75R4 Easy to use, hard to missuse - Ben Deane

- https://www.youtube.com/watch?v=sWgDk-o-6ZE CppCon 2015: Sean Parent "Better Code: Data Structures"

- https://www.youtube.com/watch?v=W2tWOdzgXHA&list=PLM5v5JsFsgP21eB4z2mIL8upkvT00Tw9B Sean Parent "Seasoning" talking about algorithms

- https://www.youtube.com/watch?v=JELcdZLre3s Function Composition in Programming Languages - Conor Hoekstra - CppNorth 2023

Naive option picker (fzf) in pure shell

 I've been a big fan of option pickers since.... forever. I've used ratmenu, ratmen, dmenu, dzen, percol, fzf, helm, vertico, consul...

For me, it's so fundamental that I need this functionality in every piece of code/script I write.

Finally I got to a working version of a very small version of it.

As always, pure shell, but with some advanced features can give you a succint function that works predictably.  The /dev/tty part was the trickiest to get working, as I didn't know how to get input from a pipe, and at the same time be able to read data from the keyboard.

Yet another example of "code the stupidest thing first"

 I've read a post called "Code the shortest path first", and I couldn't agree more, really. The core idea is a well known one to the reader of this blog (me, mostly). It's not even about YAGNI, or "The Wrong Abstraction". It's about having that "tracer bullet" or "steel thread" that guides you through the implementation phases. Also, if the feature looks too big from the beginning, you can try "SPIDR".

Also, what I find is that my first code, it's usually highly compressed, and that also gives a first soul to the implementation that lives on during the implementation. 

Clojure and aplisms

The more I do APL and K, the more I see forks and hooks everywhere else.

That's just a couple of little helpers I've added in some clojure projects. One is "apply-if", which more than from APL, I'm taking it from any procedural language, where you can do `x= if p(x) then f(x) else x`. We used to do this a lot in lua (using ands and ors). But in a combinator style.

Second one is an attempt at the fork pattern.

Look how the avg looks like. it's verbose, but it captures the idea very close to how APL does it.

2 classes of system designers

Here's a devastating quote from Andy Kinslow (NATO SOFTWARE ENGINEERING CONFERENCE ,1968) 

There are two classes of system designers. The first, if given five problems will solve them one at a time.

The second will come back and announce that these aren’t the real problems, and will eventually propose a solution to the single problem which underlies the original five.

This is the ‘system type’ who is great during the initial stages of a design project. However, you had better get rid of him after the first six months if you want to get a working system.

I found it in the Software Engineering conference pdf , but I didn't know anything about that "Andy Kinslow". And looking for that person, led me to some interesting findings, like an APL paper, or finding out that Andy Kinslow pioneered work on Time-Sharing-System, following leads of Corbato and Strachey

"delete limit X" is not possible in postgres...

And how to fix it.

 

So I was pretty sure that I had done some sort of large-ish destructive (as in side-effect-y) operations in batches of small numbers.  And today I had to delete a bunch of rows from a bunch of tables.

It turns out that postgres doesn't support "LIMIT" in UPDATE nor DELETEs.

 First of all, the misunderstanding when I remembered doing destructive operations in batches comes from calling postgres functions via select.

Let's say you have to kill the db connections to a db:

SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='db_1234567890abcdef';

This, being a select, you can do things like:

SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname like 'db_%' limit 100;

So yes, you are batching operations, but it's because "selects" are not always side-effect-free.

So now, back to our real problem of doing updates or deletes in batches.

 It's not that it hasn't been asked or thought about. Here's one of the mail threads about it. Tom Lane's answer highlights some of the issues from the correctness perspective.

Ok, but if I really want to do it... what's the best way? Crunchydata has a blog post explaining multiple ways of doing it, but the main takeaway is to use CTEs to select the amount you want (using limit), and then delete by id "where exists" or "where id in".

WITH rows AS (
  SELECT
 something
  FROM
    big_table
  LIMIT 10
)
DELETE FROM big_table
WHERE something IN (SELECT something FROM rows);




Cool stuff. Nothing magical, but it's a good pattern to be familiar with.

Naming, Notation, and Idioms conventions

Variable names is a big point of discussion in communities, projects, PRs...

I'm on the "terse" side of the spectrum, and I think many times variable names are hindering more than helping. I lay on the "more inline anonymous things" rather than "everything-has-to-have-a-very-specific-name-in-the-toplevel-of-a-file". Of course, names should convey their meaning, and if they fail to do so, it's a bad naming, but I want to talk about the more subtle process of choosing active/passive names, short/long,etc... that conveys how one thinks about the code while navigating it.

Here are some of the concepts that inspire me on naming variables and organizing code "on the small".

Misscomunicating via overconstraining and overcommunicating: Sometimes, we create a piece of code thought for a particular use case, and the more descriptive we name the variables, the less obvious it is that that functionality can be used in other situations. In strongly typed languages, this interchangeability of use cases and discoverability is already killed by the language "features", but in the languages I like to use, it's perfectly ok to call the same functions to objects of different semantic meanings, even of different type or structure.

Huffmanization:"huffman coding" principle, meaning that things that are mentioned more often should be shorter.

Perl: has so many good tips about naming, and how to understand the dimensions of concepts in programming.  To name something, "topicalization" is introducing a topic you wanna talk about, and then be able to refer it with another (shorter) name (usually a pronoun) for some time, until that topic is out of scope.

APL, SmallerCodeBetterCode, Suggestivity: Showing the guts of the algorithms inline has its benefits (see Notation as a Tool of Thought paper), and the apl family is a great showcase of that. Variable names and the symbols used for the operators have a great importance there too. Also, not binding names keeps the algorithms generic and reusable (at the human level, not binding an algorithm to a usecase but to the concrete thing it does to a data structure has way more suggestivity than a triple nested structure with its own accessors).

K:  The great k's style.pdf has so much unconventional knowledge I can only suggest to go and read it with fresh and curious eyes. So bold it's what got me into k.  Also, "Stages of denial encountering K"

fast.ai naming conventions. Jeremy Howard's insights on programming are again, a bit controversial, but his tips have been always enlightening. If he says X, doesn't mean X is the only truth, but it means you should look at X with fresh and curious eyes, and maybe you'll discover a treasure.  He comes from an APL and Perl background, and writes so nice terse code I can only agree with what he says over and over.  Look at fast.ai style conventions, and the fast.ai abbreviations list. "One line of code should implement one complete idea, where possible"

Picolisp. I don't know how Alexander Burger chose all the names for picolisp's core functions, but there are lots of cool ideas both in the namings, and how the language api makes the code flow in a very natural way. (For example, @, or the `do` flexibility).  Picolisp code tends to be very dense. Not apl/j/k-dense, but very dense still.

k/q: Also, lots of names to pick from, and the information density is pretty high. From k I picked the massive overloading of functions to do "the right thing" depending on the types. I've never seen such a compact core, and it's funny to think that the functions and their overloads seem quite intuitive after having learned them for some time.  I never thought I'd say that, but here we are.

Forth and Factor: A bunch of words to pick from. If you're looking for a word, chances are that there's a Forth or Factor dictionary with a related concept.  dup, dip, swap, bi2,.... If you get familiar with those, there's a bunch of names and name compositions you can start using in your code, and things will just make sense.

Clojure: Clojure is of course very nice namewise, and it has very carefully choosen words. Even the threading macros, which are not names, but allow writing code in a way that leans to a very flow-y way. I never understood why `assert` doesn't return the first value in case of not throwing, but apart from that, no complains.

Red: Red language is the one I know the least of this list (maybe along with forth/factor). The language has such impressive properties also, not found anywhere else that I know of.  In that link I pasted, there's a function reference. Also, lots of interesting names to pick from.

Unconditional Code: Michael Feathers did a very down-to-earth talk in 2018 about Unconditional Code. The concept is not the same as the "anti-if" movement from the smalltalk community. In this case, Michael proposes to solve corner cases of our code by either changing the specs a little bit (many times making the code more generic), assume sane defaults (see Sussman's NYCLisp talk when he talks about flight autopilot, or John Ousterhout's Philosophy of Software Design), and some times, look for generality in the algorithm by exploiting data properties (examples of this is to make functions always process lists of things, and allowing empty lists so the function does nothing)

Other notations

Where to stop in the extensibility ladder

Here's a snippet where I explored different places where to "stop" on the ladder of extensibility/abstraction/suggestivity. It's at a very small scale, but those things add up.

Code is in clojure, which makes it even easier to mix and match approaches, because symbols and sets are functions (Like K, where list access and function application looks the same) 

State machines

 I love state machines, and I use them in real world scenarios at least once a year.

The more I'm using them, in different languages or different situations, the more I get to "it's just an if" conclusion.

In this HN post about xstate, there's a minimalistic implementation in swift which I find a very nice distillation of how I use state machines usually. 

switch [current-state, event ]

  case ['foo', 'bar' ] -> ...

And that's 90% of it.

I've used ruby's statemachine, clojure's tilakone and clj-statecharts (there's also this new maestro lib which I haven't used yet), and home made lua ones. 

But the essence is, having a clear distinction of where am I, what I get, and what do I have to do. For the "what do I have to do", here's what I usually do:

- Updating the state of an object (usually the one bound to the state machine) is fine.

- I tend to run actions only for things that need to be immediate. Sending an alert in a UI has to be synchronous I guess.

- But if I can, I like to run some idempotent process that will run the actions. That process should notice that the state machine has stuff to do. That process is the one that has the knowledge of actions that should happen in XYZ situation. This process runs them, and logs them somewhere, so that next polling iteration doesn't redo them. These actions are things like "cancel a subscription after being unpaid for x days". The state machine will set the status to "unpaid", and set the cancel-at, but the polling process will check for "unpaid" subscriptions with cancel-at<now() that are still running, and kill them for good.

Further reading on xstate and fsm and one/two/three level fsms:

- http://www.findinglisp.com/blog/archives/2004_06_01_archive.html

- https://www.industriallogic.com/patterns/P22.pdf

- https://hillside.net/plop/plop2003/Papers/Adamczyk-State-Machine.pdf

Finding Approximately Repeated Patterns in Time Series

I found this talk about time series which seems super cool, about smart things people can do with time series

https://www.youtube.com/watch?v=BYjOp2NoDdc

And here are some more details in time series discords

Going meta until it hurts, with Brian Cantwell Smith

I realized today I haven't mentioned Brian Cantwell Smith in this blog before.  I'm not intimately familiar with his work, but I've read some parts of his papers, thesis, and talks.

And I can tell you, dear reader... if you're into meta/circular concepts, you'll have a great time reading his work.

Here are some links. The first one is a new talk by him, and later are works I've read and caused some impression on me. Look for more of his talks. All of them are pretty deep, and not " run on the mill" talk you find in youtube.

• https://student.cs.uwaterloo.ca/~cs492/11public_html/p18-smith.pdf <- limits of correctness . I loved it. it's the most light of the links here
  
• https://www.youtube.com/watch?v=0eE-CTX96v8
• https://www.ics.uci.edu/~jajones/INF102-S18/readings/17_Smith84.pdf
  
• https://dspace.mit.edu/handle/1721.1/15961

 I find these themes pretty dense, but it's enjoyable to have a peek from time to time. Similar to Dennet's, Spolski's, and Hofstadter's

Have fun

Oral History of Dan Ingalls

Dan Ingalls and Brian Kernighan are the few people that I can listen for N hours straight, no x2, just x1, and just listen and be in awe.

 

There's this new interview to him, and I think it's amazing. Nothing to do with Alan's talks. The speed, the body language... many different things, and equally amazing.

https://www.youtube.com/watch?v=EMwHeqQZgFw

 

Btw, at 2:30 is when he talks about Steve Job's demo.

Also, 2:45 or so, squeak implementation and portability, blowing your mind.

jq assignment of multiple fields

Jq has become the de-facto json console tool. Apart from slicing and dicing your jsons, one can do things like assigning new fields to it, or modifying existing ones.

As I explain in my scripting field guide, changing epoch times to a readable timestamp can be done like:

echo '{"date":1643192480}' | jq '.date|=todateiso8601' #  {"date": "2022-01-26T10:22:48Z"}

 

To assign multiple fields in jq, you must interleave '|' inbetween, effectively piping the result to the next assignment.

And here's the bash snippet of the day:

join_by() { local IFS="$1"; shift; echo "$*"; }
pretty_dates() {
  jq "$(join_by "|" ${@/%/|=todateiso8601})"
} 

`curl http://..... | pretty_dates .start_date .end_date` will build the proper assignments and use jq so that we get a nicely formatted json output.

Again, for small helpers like these, a bit of advanced bash can get you very very far.

 

The explanation of the weird line is: from the default (${@}) params as an array, append every element (/%/) with |=todateiso8601). That, joined by |. And that becomes the "filter" to jq.

Reinventing the Queue

Yesterday, this HN thread Write your own task queue got me thinking again on the NIH and "build vs buy". 

I'm sometimes an advocate for using own implementations of subsystems if you don't need the whole complexity of premade solutions. But...

I usually wouldn't write a task queue, as this is rarely my core business, and that I don't think my needs are anything custom, but I'll take whatever comes with the package.

That said, I've built one task queue in the past, the excuse being I was working on CommonLisp, and the platform already used all the AWS stack: (DynamoDB and Kinesis Streams) for similar things, and no ready-made solution was available. 

In that case, workers were really an active object with an active loop that kept polling the correct stream, and when a job came, an attribute in a dynamoDB table would be used to track the status. It was simple, and it allowed for reimplementations of the queues for local testing. So yeah, I've done it also...

But I can't see myself doing it when using Ruby/Python/... I feel it's very rare I'm gonna have a need that is not covered in your run off the mill job system.

Anyway, wakatime did the same thing, they implemented their own queue. And they all seem to complain about celery.

And more recent projects that approach queues:

- https://github.com/drpancake/chard. async, no threads/process

- https://github.com/thread/django-lightweight-queue. django tailored one.

Vim and Vscode to camelcase

Earlier today I saw a post on vscode subreddit where the author explores a plugin to transform parts of the code from/to camel/snake/kebab case.

Automatically I started thinking how I'd do it in emacs/vim. Given I'm lately not doing so much text munging as I used to, I feel a bit rusty wrt editing-fu. Let's see...

In the video, the guy uses something like multiple-cursors to select the area from the beginning of line till the colon. That was the most difficult thing to me, because neither vim nor emacs have multiple cursors without installing a plugin.

So, I though there had to be a "declarative" way (whatever it means) to do it, and thought of a regexp that would convert the _x to X , for all _ before a colon.

It turns out I could do it quite successfully with vim, just researching a bit how to do lookaheads.

I took 

foo_bar_f: foo_bar,
foo_bar: foo,
foo_bar_: foo  

as my test case, because it has underscores in both sides, it has missplaced underscores, and different lenghts (so we can't use visual block)

This is what I came up with. Visual select and

:s/_\(.\)\ze.*:/\U\1\L/g


I was a bit surprised that it took me a bit longer than expected, and that I had to use a few "advanced" regex features.  

I haven't tried to do it in emacs yet.  Any ideas? Any simplification to my solution?

java time

This is so helpful that I'm just putting it here for future reference.

https://stackoverflow.com/questions/32437550/whats-the-difference-between-instant-and-localdatetime

A better watch: viddy

I just found this thread in HN, and I think I'm sold already on aliasing watch to this improved version of the watch command. https://news.ycombinator.com/item?id=31829343

 

Also, nice trick of aliases, where

A trailing space in value causes the next word
        to be checked for alias substitution when the alias is expanded

Hugo documentation

Some time ago I created a new homepage for https://raimonster.com, where I was supposed to migrate my blog to. The migration never happened really (that's why I'm writing this post here), but I learned hugo on the way, and that was already something. 

During the learning process, it took me quite a bit of time to understand how hugo would render the site, what were the templates, the posts, the index pages, the home page, and how they "yielded" from one to the other. I even wrote a post about it. It was nice to find I was not alone finding the docs confusing, and this link explains why it is so confusing. Reading that post was like reading my own mind, explaining the reasons of the confusion.

Parsing args with python (alt version)

Parsing cli arguments is a solved problem. Every language has a library to get the flags from your command line. Maybe not in the stdlib, but for sure there blessed libs for it.

In Python, argparse is the official way. It's very complete and everything, but I've found it's api quite hard to follow, and I've read code that feels like written in a copypaste style, and instead of giving you more intuition on the program as a developer, it hides the knowledge of which flags go where, and which defaults they have, and which options make sense to each subcommand.

I recently rewrote a tool that had exactly that problem. Unused flags that no one removed from the argparse parsing code, hidden defaults...

When I rewrote that code, I started with this yosefk post in mind (and some reminiscence of picolisp's cli handling). I'll just do the simplest thing. Even against the standard official way (so kids, don't do it at home, or yes, do it only at home. Or not).

Here's the first version

What's already something super good is that there's 0 overhead for getting your parameters.  Of course, the parameters are not processed at all, so you'll have to cover for defaults or wrong number of args. But the language already is going to help on some of those. It will just complain if you try to call testing without an argument.

Another nice thing is that you can predict what's going to happen without checking any docs or anything. For someone that doesn't create cli apps every day, this is a nice pro IMHO.

Second version:

What can we say about this? This one takes --flag=value, and you can pass them in the order you want because the nice splat operator will take care of it. It's very rewarding to use the language features to work alongside your goal.  **kwargs, does also a great job, providing for optional parameters, that every function will make sure to validate (It should be their job anyway, not the argparser's IMHO).

Of course, in the end, the overengineering takes over:

And we end up feeling envy from --help. In this case we also can use reflection to fill in the parameters and docs from the signatures and docstrings.

That was a nice exercise!

(In the end, I just went with the first solution, because KISS and YAGNI and Suckless.)