Shell Oneliner Compression

I'm back to Lisp. Now, Clojure. For real. But this post is about zsh

Part of my relearning of clojure is to read and watch everything clojure that crosses my path. And I discovered https://github.com/clojure-cookbook/clojure-cookbook , which seems super useful.  But I have too many open tabs and books already. So I thought I'd do a 'Tip Of The Day' like thing, and I'd pop a different page every day.

 Relevant files have .asciidoc  extension. And the ones with content start with a number. The other ones are index pages or headers/footers.

Here's a version of what I typed. and it worked (at the second try!)


random-clojure() {
   e $(ls ~/clojure-cookbook/**/*asciidoc(.) | awk -F/ '$NF ~ /^[0-9]/' | shuf -n1 | xargs realpath)
}

Quite a lot of things going on there, so I'm gonna explain the oneliner, because there's a lot of compression there.

• e. e is my shellscript that opens files in emacs, moving to the line, and massaging input. in this case, it's the same as emacs, vi, $EDITOR.
• ~/clojure-cookbook/**/*asciidoc(.). This one expands recursively files that contain 'asciidoc'. excluding directories. '**/' means 'recursively' in zsh, and the '(.)' globbing is for files only.
• awk. '-F/' makes slash a field separator. Then, we pick the latest field, and we regex match it with '/^[0-9]/', beginning with a number. Cool!
• shuf -n1 picks a line at random
• xargs realpath. As we'll be running this function from all kinds of directories, expand the file path to an absolute path. as realpath asks for a parameter, we either nest the whole thing or use the xargs trick.

 

While writing this, I noticed that the whole thing is much simpler than that:

 e $(ls ~/clojure-cookbook/**/[0-9]*asciidoc(.)|shuf -n1| xargs realpath)

pipes on steroids

I though I had blogged about that before, but I can't find it anywhere, so I'm just gonna put it here (again?).

Pipes are great. You know that, I know that, everyone knows that. Because/But they are restricted to linear, non conditional flow.

Sometimes, I'd like to have an out-of-band pipe that bypasses a command in the middle, and there's no clear way how to do it.

So here are a few links on how to use file descriptors for advanced use cases. You can use them for this, and for other smart stuff in shells.

• https://catonmat.net/bash-one-liners-explained-part-three
• https://wiki.bash-hackers.org/howto/redirection_tutorial
• http://tldp.org/LDP/abs/html/ioredirintro.html
• http://catern.com/posts/pipes.html
• https://mosermichael.github.io/jq-illustrated/dir/content.html
• https://news.ycombinator.com/item?id=21700014 ( https://www2.dmst.aueb.gr/dds/sw/dgsh/ )
• http://dongyuxuan.me/posts/pipeline.html 
• https://stackoverflow.com/questions/2990414/echo-that-outputs-to-stderr
• http://www.tldp.org/LDP/abs/html/io-redirection.html
• http://wiki.bash-hackers.org/howto/redirection_tutorial 
• https://news.ycombinator.com/item?id=22704774

Making "docker run ... bash" Remember

Here's a small docker+shell hack I've never seen around and feel it makes a difference if you're getting in and out different (but related) containers. Very hackish, but works surprisingly well! Don't try it at home or rely on anything from it for other tools, as it doesn't compose very well, but hey... it's free :)

It's shell monkeypatching, and some kind of command parsing that looks like tcl/lisp-y list munging. Enjoy!

bypass zsh commands in bash

Here's a small trick I came out with, when trying to run some scripts that were thought for zsh in bash.

I use 'noglob' in many places, and sometimes they leak into my bash scripts, or are called from bash somehow.  As bash doesn't know about noglob, usual result is an error.

But! you can use this

cat <<-EOF >~/bin/noglob
#!/usr/bin/env bash
$*
EOF

So a bypass file is called in case the command 'noglob' is not catched by the shell.

EDIT: Now I remember why the f I created this.

You know I'm a heavy user of zsh's global aliases. my aliases that contain pipes are always UPPERCASE, because it gives a hint that something strange is happening there, and I also see it entering the realm of pipes. 

The thing is that when pairing with others, if I write that when they are looking, people have no clue what's happening when I type that, and it's pretty unintuitive.  Also, as my zsh and bash share part of the history, if I reach some command that contains one of the magic aliases, I've to manually fix them by expanding by hand.

1st fix: magic expansions

globalias() {
   if [[ $LBUFFER =~ ' [A-Z0-9]+$' ]]; then
     zle _expand_alias
     zle expand-word
   fi
   zle self-insert
}

zle -N globalias

bindkey " " globalias

This expands the previous word if it's an alias, but I only want to expand the ones that are ALL CAPSLOCK. Because I have very nasty aliases I don't want to expand as I go. (This has extra an benefit of allowing expansions of "dynamic aliases", which I'll show in some other post)


With this, I end up with noglobs scattered around my history, and if for some reason I execute those in bash, it'll try to run `noglob http ....`.  Here is where  ~/bin/noglob works fine by just bypassing everything.

inifinite history

This is a very nice hack combo that makes shell history really useful.

Add precmd to log the last command into a given ~/zsh_history.log file that will log EVERY command you ever type in the console. With some context like date, and $cwd.

rghist finds commands from any of you history files, allowing you to find many words in the same line but not necessarily one after the other. But necessarily in the same order.

rghist is 100% mine, and comes without guarantees. The precmd command has been sitting in my .zshrc for years. Not sure where I got it from, but the idea is very nice.

Given that rghist can plow through gzipped files, gzipping the log (with password), could be a nice improvement.

function rghist(){
  local str="$(echo ${(j:.*:)*})"

  rg --no-filename -v rghist ~/.zsh_history.log ~/.zhistory | rg "$str"      
}

function precmd {
  if [ "$(id -u)" -ne 0 ]; then
    FULL_CMD_LOG=~/.zsh_history.log;
    echo "`/bin/date +%Y%m%d.%H%M.%S` `pwd` `history -1`" >> ${FULL_CMD_LOG};
  fi
}  

so, rghist will join $*  with '.*', making it seamless to find for many words in the same line. For example, sometimes I wonder: "How was the command to run a bash inside a running docker, was it run? was it exec?", then I just "rghist docker bash" and get a list of examples of commands I've run in the past like that. It's dead easy, it's something like a grep with infinite memory, but having this ever growing scratchpad has immense value to me.


EDIT: rghist above tried to be an improved version from my original one(better color managemen). But it's failing to produce output in some cases (I suspect binary strings in file). Here's the older and more robust way.

function rghist() { 

  local str="$(echo ${(j:.*:)*})"
  rg --no-filename --color always "$str" ~/.zsh_history.log ~/.zhistory |
  rg -v rghist

} 

TIL: dabbrev-expand in zsh

So just by chance I found that M-/ does a kind of dabbrev-expand in zsh.

And it took me a few seconds to realize that shouldn't happen in that context.

Or should it?

What's for sure is this is one of the coolest things you'll learn today about zsh.

TIL: no more git clone

I knew about suffix aliases but usually don't need them because my launchers already use xdo-open. But I came out with this, and feel that it has a twist:

alias -s git="git clone"

It makes github links more copypasteable.

grep for two words in the same file

Let's go for some more csv (or any text file) fixing, grepping and slicing and dicing.

The problem is simple:
Find a file (among a vast amount of them) that contains 2 or more words, not necessarily in the same line.

That makes piping greps onto other greps useless. The solution is quite easy, but it might not be obvious:

grep -l word1 **/*csv(.) | xargs grep -l word2 

Thanks for watching.

guerrilla csv and xlsx

I like to have a huge toolbox so that I can always find the right tool to do any task. But I'm also a big fan of composability, and orthogonality.  So it's a bit like vim vs emacs, or small languages vs big languages, or scheme vs CL, or Python vs Perl.

On the command line, I also like to find tools that compose.  Although pipes and xargs are the way to compose commands, the interfaces have to be compatble, by using stdin/stdout, or file names ( <() comes to the rescue by helping with the plumbing).

So today I had to count the appearances of a given word in different xlsx files. Each xlsx had many sheets, and we only want to count the appearances in column 9.  It was kind of a checksum to make sure that all appearances of  $KEYWORD were still there.  So, task the task is:

Aggregate counts of appearances of 'keyword' in the ninth column in all sheets of each one of those excels.  Get the sum per file.

Apparently, after 5 minutes of typing in trance, this did the trick.

for i in **/*xlsx ; do echo $i ; csvfix write_dsv -f 9  <(xlsx2csv.py --all $i ) G 'keyword' WC ; done

We can't get much further with debugging this. The pity with these kind of approaches is that they either solve your problem in the first shoot, or it gets exponentially difficult to treat for special cases, or add debugging info.

I got to, at least, compare the results themselves using vimdiff.

vimdiff <(csvfix write_dsv -f 9  <(xlsx2csv.py --all file1.xlsx ) G 'keyword') \

        <(csvfix write_dsv -f 9  <(xlsx2csv.py --all file2.xlsx ) G 'keyword')

This is totally not rocket science, but I love the feeling of power and accomplishment you get when this magic incantations work.  You run that, you get the result, you use the result, and you throw the whole thing away.

And you keep doing what you were doing.  Or go write a post about that.

Zsh global aliases for command chaining

(The motivation for this post is my recent discovery of oilshell, a very promising new shell.  The author has been posting regularly for some time, and he has quite insightful ideas. Some of them I had them too and I didn't know they conformed to some defined concepts (I discovered Bernstein chaining two weeks ago). Whatever.... here it goes, as a way to open more discussion topics)

There's a small zsh trick that is unmatched by any other shell. It's maybe just syntax sugar, but it enables some pretty neat tricks.

The feature is global aliases (or "alias -g").

They are alias substituted anywhere on a line. Global aliases can be used to abbreviate frequently-typed usernames, hostnames, etc.

I think of them as some kind of reader macro, that applies substitutions to strings at read time, so there's no need to substitute a full command by another full command.

alias -g

 

The simplest case is to make an alias for a bunch of flags for a command you use many times (Also look at zsh globbing to see other examples of 'simple' alias -g).

alias -g GLWEEK=' --since=1.week.ago --author=kidd'

Given this ("git last week"), we can use it in different contexts. In this example, the only useful ones that come to my mind are "git log" and "gitk", but to get the feeling of what it does internally, you can 'echo GLWEEK' and you'll get the flags printed.  It allows for a lot of composability, and Bernstein chaining because everything concatenates and tries to apply.

Piping through

 

Another way to see it is as a way to Forthify your shell a bit more with pipes. This is how I use the feature the most:

If you make aliases expand to pipes (left, right or both), you can make many unix commands compose in a seamless way. Under the hood, you know that the connexion happens through stdin/stdout, but visually it eliminates most of the clutter.

Let's see some examples:

alias -g T='| tail'
alias -g T1='T -1'
alias -g X='| xclip '
alias -g G='| grep '
alias -g GV='| grep -v '
alias -g DM='| dmenu '

Let's see a few ways we can compose this:

ls -ltr G foo GV bar T1 X
evince $(ls *pdf DM)

With some training, it's very easy to see that this will copy the last file that matches 'foo' but doesn't match 'bar', or it will open the selected pdf file.

XARGS

 

There are some commands that do not accept arguments from stdin, but they require them to be as command line arguments. in those cases, one starts doing things like:

mplayer $(ls *(mpg|mp4|avi) DM )

Which works, but it breaks the pipe flow.  For that, we can use xargs (as we saw a few posts ago), and we can use, again, another global alias:

alias -g XA='| xargs '

now we can do "ls *(mpg|mp4|avi) DM XA mplayer" .

FUNCTIONS 

 

Here's an example of a uniq function on steroids. it accepts a field number to uniquify, and the rows do not have to be sorted to work.

function uc () {
    awk -F" " "!_[\$$1]++"
}

alias -g UC=' | uc '
alias -g P1='| awk "{print \$1}"'

Let's see how to use this one: For the sake of the example, let's pretend we want to know the users that have some process running in my machine.

ps -axuf GV USER UC 1 P1

Of course, this is the same as typing the vanilla commands yourself, but IMO, it makes quite a lot of difference on the experimentation easiness.

There are some more fancy tricks, but we'll leave them for some future post.

The whole point of this is that some features allow for greater composability, and let you build your ad-hoc tools in a much easier way than others.  Oilshell has plans to have '{', '}' and '[', ']' as operators, so you can have block like syntax (maybe the semantics of tcl's quoting would be enough?).  Anyway, it's great that the shell space is active and there are interesting things happening. I'll keep oil in my radar :)

Command line bookmarks

I saw a Gary Bernhardt's talk where he explains a few console tricks. Most of the tricks themselves I already knew, but the biggest outcome of seeing the talk is the motivation for scripting everything and using small functions to use bash/zsh the fastest possible way.

I'm a big fan of zsh, and I have a fairly big .zshrc (101 aliases, for example).

At work, we use a quite strict and orthogonal way to tag all project issues, so that one can find out easily which issues are waiting for merge, which are halted, or which are being worked on.  As we use github, the usual way is log into github (or waffle) and search .

But when it starts getting repetitive, I usually think how to do it faster. one option was bookmarks in conkeror, but it didn't quite work.

Now, I'm using this in my .zshrc :

BROWSER=/home/rgrau/bin/conkeror
PROJECT=projectname

function ghnext {$BROWSER "https://github.com/3scale/$PROJECT/issues?q=is%3Aopen+label%3AT-core+label%3AB-Next"}
function ghmerge {$BROWSER "https://github.com/3scale/$PROJECT/labels/needs%3A%20merge"}
function ghmy-issues {$BROWSER "https://github.com/3scale/$PROJECT/issues?utf8=%E2%9C%93&q=is:open+assignee:kidd"}
function gh3scale {$BROWSER "https://github.com/3scale/$PROJECT"}
function ghissues {$BROWSER "https://github.com/3scale/$PROJECT/issues"}
function ghcurr {$BROWSER "https://github.com/3scale/$PROJECT/issues?q=is%3Aopen+label%3AB-current+label%3AT-core"}

And here is gary's talk:

Metaprogramming Zsh - Poor man's autojump (or J (or Z))

There's autojump, there's also J, there's also Z.... Each one of them with its own fans.
I've been trying some of them on and off, but mostly ditched them because I don't need the complexity and I don't get used to type 'z' when I mean 'cd'.


An easy and smart alternative is to autogenerate aliases on boot. It's easy, you can understand all the logic behind it, and your shell will provide the autocompletion. Pretty darn simple.

function aliasgen() {
    for i in ~/workspace/*(/) ; do
        DIR=$(basename $i) ;
        eval "alias $DIR='cd $i'";
    done
}

aliasgen

For your usual projects, this should be more than enough. "But, but sometimes I want it more dynamic aliases, like, for random directories", I hear you say. Ok, then there's this nifty functions that also creates aliases on the fly.

function a() { alias $1=cd\ $PWD; }

When you're in a directory you wanna keep for later, type "a foo", and an alias "foo" that will go to the current directory will be generated.
Even if you wanted to persist them you could create a symbolic link from the "~/workspace" directory in the previous snippet with the choosen name.
These 2 little tricks just show how using old tools and some wit can get you going a long distance.
I hope you enjoyed this. Cya next time!


EDIT: Post deprecated in favour of CDPATH .  At least I learnt a new thing . Thanks Toni!

Renaming 'used' directories in Zsh and Bash

I can't understand why, oh why, neither bash nor zsh can apply the same policy to mv that they apply to umount, so that when trying to umount a volume that is in use, it tells you so.

When the directory you're in is renamed (moved),the shell keeps showing the old path you were in without notifying you in any way that this directory you see in the prompt is not that anymore.

 Probably it's an inode thing: When you change the name to a file/dir, you probably just have to change the inode's name. And I'm fine with it. Anyway it'd be a waste of resources to try to communicate to other open terminals/shells that there's been a change in a remote directory in the system.

That, and probably for thousand reasons I don't even know its existence.

So the question would be. Why couldn't we make bash/zsh check for the existence of the cwd when displaying PS1 if in the PS1 itself there's the metachart to show the directory path (%~ in zsh)?

Sorry if it sounds a bit rude, I just had an amazing debugging session for 2 hours because I was debugging the wrong thing. My bad, I know, but.....


So here's the way to reproduce

[ /tmp ] %mkdir test
[ /tmp ] %cd test 
[ /tmp/test ] %ls
[ /tmp/test ] %mkdir foo
[ /tmp/test ] %echo OHAI >foo/bar
[ /tmp/test ] %cd foo 
[ /tmp/test/foo ] %ls
bar
                                     In another terminal
                                     [ ~ ] %cd /tmp/test 
                                     [ /tmp/test ] %ls
                                     foo
                                     [ /tmp/test ] %mv foo bar
                                     [ /tmp/test ] %ls bar 
                                     bar
[ /tmp/test/foo ] % ls
bar
[ /tmp/test/foo ] %pwd
/tmp/test/bar

Is there a reason for that?


PS: My real case was something even more tricky as I changed 'reponame' to 'reponame2', and then recreated 'reponame', so I had different shells pointing to 'the same' path, without being actually the same.

Self modifying bash script

Last month, my ten years old lappy said enough. Well, it can boot, but it doesn't get the plug correctly, so it has to stay pretty static. It came with windows 98 installed, and I've done most of my uni tasks there, as well as installing tenths of linux distros (until vectolinux).

While moving files from its HD to a safer place, I found a bash script (works on zsh too) I did about 5 years ago, which emulated a 'mute' function, and could be called through a shell (or through xbindkeys / keylaunch / ratpoison binds).

The funny thing about it is that it modifies itself to remember the last volume that was set, to restore it afterwards.

I know it could be done using a file to store the previous volume value (and in fact, I use a trick of the same kind), but well, I liked to see that trick again. It uses sed to substitute a variable that is later tested if it's 0. Pretty easy stuff.

It makes me remember my first questions about self modifying code. Now, with lisp and smalltalk, living in a life environment, everything is clearer (sure?).

Comments and improvements are obviously welcome. Enlighten me!

zsh spell checking

I was talking to David about bash vs zsh features, and looking at the wikipedia page on comparison between shells We saw They only mentioned a couple (maybe 4) things where zsh is superior to bash.

One of the things that seem to make a very little difference (if any at all) is spell checking. Is it useful?

Well, the answer is: Definately!

If the next command outputs a zero, then you don't need this feature

history| grep 'ks\|,ale\|male\|gi\|' | grep -v "history" | wc -l

Otherwise, keep reading.

For short commands you don't use tab, you can misspell them and zsh guesses what you meant.

rgrau@ares [ ~ ] %,ake
zsh: correct ',ake' to 'make' [nyae]? y
make: *** No targets specified and no makefile found. Stop.
rgrau@ares [ ~ ]:2 %ks
zsh: correct 'ks' to 'ls' [nyae]? n
zsh: command not found: ks
rgrau@ares [ ~ ]:127 %

Great, If you ask me.

zsh saved my home

zsh has just saved my whole /home/rgrau directory .

I was hacking like crazy, listening to some techno music, typing as if I were mad, and that's what happened:

rgrau@ares [ ~/Desktop/pharo ] %cd ..
rgrau@ares [ ~/Desktop ] %cd ..
rgrau@ares [ ~ ] %rm *
zsh: sure you want to delete all the files in /home/rgrau [yn]? n
rgrau@ares [ ~ ]:1 %

Obviously, what I wanted to do is delete all files under Desktop/pharo dir , but I typed the 2 cd .. commands without thinking (I suppose I wanted to do a cd ..;rm -fr pharo )

Anyway, zsh asked me and I read what I was going to do... cold sweat... typed 'n', and went to smoke a cigarette.

Thank you zsh. I really love you.

Btw, if you don't want zsh to save your life, you can add setopt rmstarsilent to your .zshrc .