miércoles, 30 de enero de 2019

inifinite history

This is a very nice hack combo that makes shell history really useful.

Add precmd to log the last command into a given ~/zsh_history.log file that will log EVERY command you ever type in the console. With some context like date, and $cwd.

rghist finds commands from any of you history files, allowing you to find many words in the same line but not necessarily one after the other. But necessarily in the same order.

rghist is 100% mine, and comes without guarantees. The precmd command has been sitting in my .zshrc for years. Not sure where I got it from, but the idea is very nice.

Given that rghist can plow through gzipped files, gzipping the log (with password), could be a nice improvement.

function rghist(){
  local str="$(echo ${(j:.*:)*})"
  rg --no-filename -v rghist ~/.zsh_history.log ~/.zhistory | rg "$str"      

function precmd {
  if [ "$(id -u)" -ne 0 ]; then
    echo "`/bin/date +%Y%m%d.%H%M.%S` `pwd` `history -1`" >> ${FULL_CMD_LOG};

so, rghist will join $*  with '.*', making it seamless to find for many words in the same line. For example, sometimes I wonder: "How was the command to run a bash inside a running docker, was it run? was it exec?", then I just "rghist docker bash" and get a list of examples of commands I've run in the past like that. It's dead easy, it's something like a grep with infinite memory, but having this ever growing scratchpad has immense value to me.

EDIT: rghist above tried to be an improved version from my original one(better color managemen). But it's failing to produce output in some cases (I suspect binary strings in file). Here's the older and more robust way.

function rghist() { 
  local str="$(echo ${(j:.*:)*})"
  rg --no-filename --color always "$str" ~/.zsh_history.log ~/.zhistory |
  rg -v rghist

martes, 22 de enero de 2019

Formatting JSON

Here's something you probably had to do at some point. You have some big json blob and you want to format it nicely.

I had done this in vim and honestly, it comes very very easy.  In vim, if you have a buffer (/tmp/foo.json) you want to reformat, and you have jq, the nicest way I've found is something like:

:r !jq %

This is really hard to beat, and I'm not sure you can be much faster in any other thing.

But I live inside emacs, and I when I have this need, until now, I was selecting the region, then `M-| jq .`  The problem is I always forget about m-|, and it's really annoying to type it anyway.

So I was about to create my elisp function that would reformat my json region, and before coding it, I timidly tried M-x json-form..  and yes, of course, json-reformat-region is already there (you can download it from melpa).  Job done.

Btw, if you use evil, you can also use ':r !jq /tmp/foo.json'.  For some reason, % doesn't work as a substitute for the current buffer-file-name, but at that point, I don't care much. :)

miércoles, 16 de enero de 2019

rubber docker

 Thousand different blogs and people rambling about docker, and whatnot, and years later of its boom, I slowly get the knowledge of wtf is docker and why it might be useful. 

I still think it's super complex and flexible, and that most people do not fully understand the nuances between the different situations you can get at with it.

Here I leave some links that in one way or another have made me "tick" something about docker.
  • https://courses.cognitiveclass.ai/courses/course-v1:IBMDeveloperSkillsNetwork+CO0101EN+v1
  • https://towardsdatascience.com/learn-enough-docker-to-be-useful-b7ba70caeb4b
  • https://dockerbook.com/TheDockerBook_sample.pdf
  • https://blog.sixeyed.com/windows-dockerfile-26/
  • https://medium.com/@nagarwal/lifecycle-of-docker-container-d2da9f85959
  • https://medium.com/the-code-review/docker-run-vs-exec-deep-dive-into-their-differences-19a1041735a3

One of the differences that were somehow harder to grasp was what exactly makes a container different than an image.

As of what I know today, a docker image has its read-only layers. A container that you spawn off this image will have a rw layer on top of that, and will point to the top layer of the image, that's why you can have 2 containers off the same image.

If you want to create an image out of a container (that means, start some image, fuck around manually as much as you want, and then store it to a runnable image), you can use docker commit.

Another option to create an image out of another image is to create a dockerfile and make that dockerfile start FROM the initial image. Then, in the dockerfile you fuck around, and after that you'll have an image where you can spawn off many containers.

The image doesn't have any state. You build it and that's it.  A container, apart from the filesystem itself is a bunch of processes that are running, and a network setting, and mounted volumes.

You can start a container from an image with different network settings, or exposed ports. That belongs to the container dimension.  And containers can be in multitude of states. halt,run,stop,pause,....

Another weird thing is what does it mean to run a command in a container.  If you have a container running, you can use 'docker exec' and you'll run a command inside that container. no new container will be created, and if this command changes some internal state, or creates a file, it will create a file in that already existing container.  If instead, you 'docker run', you should pass an image name, not a container name, and if the command you run is stateless, and just does something to the outer world (run migrations), you can run it either with run or with exec inside an already running container (if you have one)

docker dilema

martes, 15 de enero de 2019

tmux session navigation

https://tmuxcheatsheet.com/ tells us that c-b ) and c-b ( lets us cycle through different "sessions". And I'm afraid that's pretty cool, and in fact I can import that to ratpoison and move through the different workspaces with c-t ) and c-t (.

I guess that's pretty basic tmux, but I found it very nice and I'm already using it to swing between my work session and the "slacks" session (using slack-term)

lunes, 14 de enero de 2019

Postgres compression of jsonb

Postgres supports JSON data since loong time ago, but until jsonb (9.4) all solutions were somewhat limited in features.  Jsonb gives you most of what a document storage engine would give you, but...  what about performance? and space efficiency? Unfortunately I don't have real answers to those questions, but I've started doing some research, and for now, here's a nice takeaway:

First of all you should be aware of "\dt+", which gives us the size of a table and "\d+" which describes a table. The "+" suffix adds more info to the descriptions.

CREATE TABLE foo AS SELECT '{"f":true}'::jsonb FROM generate_series(1,1e6);
CREATE TABLE bar AS SELECT '{"f":true}'::text FROM generate_series(1,1e6);
\dt+ foo;
\dt+ bar;  

We see that the text field takes about half of the size of the jsonb. So if you don't need json features in a field (you're only archiving data that happens to be json), think twice when giving it the jsonb type.

Jsonb is TOAST-able, but will probably be toasted  (and compressed) when it's bigger than a page, so don't count on that if you're jsons are <4kb .="" p="">

  • https://dba.stackexchange.com/questions/161864/do-long-names-for-jsonb-keys-use-more-storage
  • https://stackoverflow.com/questions/23120072/how-to-check-if-toast-is-working-on-a-particular-table-in-postgres
  • https://postgrespro.com/list/thread-id/1849114

So, don't think spacewise it comes for free. If you're concerned about space, there's a big impact on the size of the table.

domingo, 13 de enero de 2019

docker run top & docker exec

Here's how I test docker things:

docker container run -t ubuntu top

Then you have this top window  that shows only a single process.

In a different terminal you can then do things like "docker container exec 123456789abcdef sleep 10", and see it in the top window. It's a stupid thing, but I need instant feedback to solidify my knowledge of what's happening where.

jueves, 10 de enero de 2019

Remapping Keys "Up to Eleven"

Remapping keys on your linux is a "normal" thing to do amongst hardcore vim or emacs users.

The most standard hack is to remap CapsLock to Escape or to Ctrl.

And with emacs' key-chord and key-seq plugins, you can have really fancy combos to do the most obvious save-buffer, m-x, switch-to-buffer...

But I just discovered (via this reddit thread) that there's this "xcape" thing that allows you to bind keys to different keycodes depending on whether a key is pressed and released on its own or it's pressed as a modifier key along with other keys.

Here's how to "Remap left and right shift, when pressed alone, to left and right parens on keyup": xcape -t 250 -e "Shift\_L=parenleft;Shift\_R=parenright" &

That's super cool. And the readme of the project has some hardcore hacks for even more crazy conditional remappings.

As a side note, but kinda related, here's a great recent article about all kinds of glue (and duct tape) for X windows. And here's another awesome post about wizardy linux usages. The guy uses dmenu as an interface to a barebones plumber and gave me an idea to improve on my home made plumber (be smart and let you trim down witn dmenu when there are multiple options)

miércoles, 9 de enero de 2019

docker-compose, multiple yml files and environment variables

Did you know that docker-compose can merge multiple yml files into one? it seems like a very cool way to override configs and have something like a linear composability.

Just for fun, try

And do a few tests using

  • docker-compose -f docker-compose.yml config
  • docker-compose -f docker-compose.yml -f docker-compose-over.yml config
  • FOO=1 docker-compose -f docker-compose.yml config
  • FOO=1 docker-compose -f docker-compose.yml -f docker-compose-over.yml config
  • docker-compose -f docker-compose.yml up
  • docker-compose -f docker-compose.yml -f docker-compose-over.yml up
  • FOO=1 docker-compose -f docker-compose.yml up
  • FOO=1 docker-compose -f docker-compose.yml -f docker-compose-over.yml up

In addition to this, you can use the .env file to add defaults to the variables, and you can take a look at a special .override.yml suffix in https://docs.docker.com/compose/extends/. Plenty of ways to configure, reconfigure, and override variables. The difficult part is to make it understandable at all for the next guy touching your setups.

Also, in your app, you can have defaults for the env variables (docker will set them to empty strings if you don't set a value for them).

Also, remember that "docker-compose run" and "docker-compose exec" accept -e to set environment variables. I still haven't tried all the possibilities with those to see exactly what do they overwrite and what is fixed.