domingo, 20 de marzo de 2016

console tools for structured trext

The lingua franca of unix is plain text and raw files. That's why we can do so many things using its standard tools and pipes as combinators.

But there are some special formats of files which have a concrete structure, and we can extract meaning from its structure. For example CSV.

JQ

jq is like sed for json files. It allows you to parse, grep, replace, match and join json files. For example, /tmp/issues.json being the output of a github repo issues:

cat /tmp/issues.json | jq '.[] |select(.labels[].name | in({"S-zendesk": 12})) | {labels: [.labels[].name | match("^A-.*") | .string] }'

Select issues which have label S-zendesk, and pick the label A-* of it. To know which are the areas that have zendesk issues.

Xmlstarlet

Xmlstarlet is the same as jq but for xml. Allows you to print and match fields from xmls. For an xml file like this:
We can list title of files and id's with:

cat files.xml | xmlstarlet sel  -T -t -m 'files/file' -v 'title' -o ' => ' -v 'id' -n


dateutils  

I just discovered dateutils. But it seems a very good companion for tail, or just to do standalone date calculations.

 pbmtools

Part of netpbm. compare and operate on images from the commandline. Sucks less (apparently) than imagemagick.

Praising MJD

Mark Jason Dominus is one of those guys that is worth following everywhere in the net.

  • Web: His site is quite old school. It reminds me of c2, or norvig.com, that have really old look but is full of insights and no-bullshit content. Also, loads instantly.
  • Blog: Wide range of topics. From systems programming to functional programming, haskell, monads, git, books, alien codes, mathematics.... 
  • Book: Higher Order Perl is a great book on functional programming. It uses Perl, but the concepts are really language agnostic. It clearly follows a similar approach to SICP, maybe with a more practical approach.  Higher Order Perl is on my top 10 list of tech books. Top5 if we exclude lisp books.
  • Talks: He excels at giving talks also. There are very nice talks about. 

viernes, 18 de marzo de 2016

programming quizzes with org-babel

When trying to write some proof of concept, or an algorithm in isolation (or programming quizes), I usually create a new file (foo.lua, for example), and run it with my patched version of shell-execute that replaces '%' by the current file name .

But sometimes I want to give more context or write the code as a story, or a question-answer dialogue.  In those occasions, org-babel is the way to go.
This piece of elisp enables a bunch of languages to be evaluated in org file blocks:

(org-babel-do-load-languages
 'org-babel-load-languages
 '((ditaa . t)
   (tcl . t)
   (ruby . t)
   (sqlite . t)
   (clojure . t)
   (lisp . t)
   (http . t)
   (sql . t)))


Now, in the org file itself, place this kind of block:
#+BEGIN_SRC sqlite :db pieces-and-providers.sqlite
#+END_SRC

Now, when pressing c-c c-c inside a block, a new block called RESULTS will be created underneath with the result of the eval'ed block. 

 There are many other options on exporting org-blocks and other options for fine tunning how results are shown, or formatted. For me, this is a nice way to have my experiments documented, or leave exercises for my future me to solve. For example, I copied most of the sql exercises wikibook and created this repo with the stories/exercises as org files so I can try them in an interactive way.