Fast feedback loop with emacs (Lua)

Here's a long overdue post about emacs fiddling. All this belongs to my previous job time, during the past 3 years doing Lua.

One of the major test frameworks in Lua is Busted. It uses an rspec-inspired syntax and the cli accepts flags to filter the tests you want to run, by file and regular expression.

To iterate as fast as I could, I'd want to lock on a particular test, and be able to run that same test for a while. That's super useful while hacking on a feature or fixing a single test.

So the solution I came with was to use our beloved emacs, to find the current test the cursor is on, and remember it in a global variable. Of course, I want it in the most dwim-y way, so the same binding does "the right thing" most of the times.

Here's the decision making proces, that starts calling rgc-run-test (bound to f7):

rgc-run-test

If I'm in a *_spec.lua file, the process sets a global var with the current file path. The process also looks for the closest previous line to my cursor that looks like a test definition and sets another variable with the string of the test. To recognize the test definition, I match the lines against "it(", "describe(", or "it_content_types(". I use 'rx' library for that, which is pretty cool, check that out if you're doing complex regexes in elisp.

rgc-test-shell

After that, we find a buffer with a shell.

If there's none currently open, I create one, using the right parameters (so it lands on the correct directory).

An extra nicety is that I call 'highlight-regexp' with a string that will highlight any debug line.

You can enable (add-hook 'shell-mode-hook 'compilation-shell-minor-mode), making all shell buffers to run compilation-shell-minor-mode. That means that every line that looks like a path/to/source.file:line becomes clickable. That means that you can navigate to the source:line from a stacktrace.

With this, you can press F7 in a test file, and this snippet will make sure that a shell is in the right place (opening an existing one if it's already there) and will run the current test. 

But, many times, you are not just editing the test, but you're touching the code. No problem. F7 remembers the latest test you run when you were in a test file, so it will run that same test in case you're not in a test file.

When things seem to be solved, you then want to run the tests in that same file, but not restricting it to that single test, but run the whole file, to make sure you didn't mess up anything else. C-u F7 will do just that.

rgc-test-flags

Last cool thing. Sometimes I want to lock extra flags for the tests, rgc-test-flags is a quick way to overload the flags and keep them around for the next runs also.

ffap

Also, sometimes, the test errors in lua are marked like 'path/to/file @ 98'.  This makes find-file-at-point miss the line number.  But yeah, emacs. you defadvice find-file-at-point, with an extra case and off you go.

 That's it

I know this whole thing is quite hackish, and there's a lot to chew in just 50 lines, but this was so useful to me that I wanted to share before I forgot about it (changed job recently so I'm not using this snippet anymore)

It's this kind of workflows that the holistic approach of emacs allow for. And we love it so much :)

Here's the gist of it:

 

so JIT, WOW

This is so amazing I don't want to lose it:


A Smalltalk-80 VM written in LuaJIT, that boots the images from the eighties:
https://github.com/rochus-keller/Smalltalk#a-smalltalk-80-interpreted-virtual-machine-on-luajit

Approachable tutorial on JITs. The second part talks about metatracing.
https://news.ycombinator.com/item?id=23740655

I recently discovered this guy and he's awesome. Haven't particularly examined this repo, but please, also check out his other stuff.
https://github.com/spencertipping/jit-tutorial

Checking Password Strength in 10 Lines

Talking about dependencies, there's this simple case:

To check password strength, we want different min lenghts for passwords depending if they have lower, upper, numbers, and simbols.

For this, there's passwdqc that allows you to do it in a very simple way,  but, do you really need a library?
Here's the minimalistic implementation I came up with, which I think is pretty decent, and again, has some nice property I can't quite describe.  The code is lua, but it can of course be translated to anything.

It has minimum lenghts for the passwords depending on the amount of different classes of characters it contains. If it only contains 1 type, we don't accept. for 2, minimum length 24,....
   local str = io.read('*l')
   local d = str:match("[0-9]") and 1 or 0
   local down = str:match("[a-z]") and 1 or 0
   local up = str:match("[A-Z]") and 1 or 0
   local s = str:match("[!@#$^&*()_=+-]") and 1 or 0
   local l = #str
   local defs = {math.huge, 24, 11, 9}
   print(d,down,up,s,l,l>=defs[d+down+up+s])

Some Forth Implementations

To finish up one of the geekiest weekends in the last months, I just wanted to point to a few Forth implementations, from where I learned some nice tricks (from both forth and the implementation languages)

- Bashforth/Perlforth
- Rubyforth
- Miniforth
- ItsyForth

I spent a lot of time grasping Bashforth, with its usage of some smart tricks that you don't usually see in bash scripts. Also, the fact that array indices are used as a sort of pointers makes total sense, but it wasn't obvious to me at first (I had to research a bit about "delcare"). Perlforth is done by the same guy, so it uses a similar approach.

Miniforth is Lua, and uses a different kind of implementation, and gets to bootstrap something forthish in 40 lines of lua.  Not bad :)

Rubyforth is the one I touched less, but looks like a variation of miniforth with more Forth compliance.

Now a quick bash quiz: What does it echo? AnswerHere

#!/bin/bash
fun() {
        local var="local value of var"
        echo "$ref";
}
var="global var"
declare -n ref=var
fun 

Well, there are two more canonical forth implementations that is worth mentioning them here, because if you're gonna just look at one implementation, you should take these ones as more "de facto" implementations.
- pforth
- jonesforth

Some Lua articles

So at my job we're heavy Lua(jit) users. I've always liked Lua since I discovered it a few years ago.

It has a scheme-y feeling to it (with paredit unfriendly syntax, unfortunately). So here's a new article that has been recently published in the ACM. And some other paper from 2011. And another very nice paper that shows how lua can be used in different programming styles. Like the old "Implementation of Lua 5.0" or the lpeg one (you search that link yourself), Lua maintainers have a very concise style of designing and writing. Both code and papers. I always enjoy them.

fixing indentation of lua (busted) in emacs. A nasty hack

In general, indentation is not an issue in emacs.

But there are some exceptions.  For example, in Lua, one of the de facto testing libraries is busted, which tries to mimick rspec in many aspects.

A typical busted test looks like this:
Lua mode tends to indent code in a very lisp-y way (which I personally like) by aligning parameters to the same function using as a "starting point" the offset of the first parameter.  In this case, there's also an opened function that gets indented in addition to that base column.
This is unacceptable in most codebases, so I had to write some hack for those particular cases.

As the indentation code for lua-mode is quite complex and this is an exception to the general rule, I wrote this very ugly hack, that seems to solve my problem at hand.
As with all defadvice uses, it looks like a hack because it is a big ugly hack, but at least it lets me deal with it, and move on with my tasks without manually reindenting stuff.



Another +1 for emacs hackability :)

spying on lua function calls

Following on lua, there's been some trick I've been using for some time, and it's quite useful and (as usual), doable with tiny piece of code.

If we want to have a trace of function calls with their parameters and results, there's a super easy way to do it in lua.  The functionality is basically inspired by lisp's trace or elisp's trace-function.  The code is ridiculously simple, it's a basic case of rewriting key-values in modules and wrapping functions.

local function make_tracer()
  local indent = ""
  return function (mod, f_name)
    local old = mod[f_name]
    return function (...)
      print( string.format("%sCall: %s: params: ", indent, f_name), ...)
      indent = indent .. "   "
      local ret = {old(...)}
      indent = string.sub(indent, 4)
      print(string.format("%sRetn: %s: ",indent, f_name), unpack(ret))
      return unpack(ret)
    end
  end
end
local trace = make_tracer()

... 

for m,_ in pairs(M) do M[m]=trace(M, m) end

It's great that with so simple code we can have a basic debugging tool like this one (btw, this tool is probably not very robust if we put coroutines in the mix, but for simple cases it works quite well).  All this is possible because lua embraces the Universal Design Pattern.

Test spies with Lua metatables

Dabbling with Lua metatables, I tried to write a minimal testing library that does not impose you any funny 'describe(...)' or 'it(....)' nesting, and one can just organise the tests as he pleases.

What

I called it spacesuit.lua as it wraps your functions and gives you minimal support to write tests (assertions and spies) in the wild. If you need your tests to be TAP compliant, runnable from any platform, and a well known solution, I can recommend busted, but for me, I tried to keep it minimal so I can put it in my bag and run the files I need from my console, using some silly bash/zsh script using globbing. no need for luarocks, native compilation of lfs or anything.

Apart from providing some sugar for assert_equal (which I'll probably delete in favour of plain assert(foo==42)), it provides:

• assert_raise(fun): runs the function and asserts an error is thrown during its execution.
• spy(fun): returns a proxy function (it's a table with __call in its metatble) that logs all the calls (both actual parmeters and results). The usage is quite simple:

s = spy(function(x) return x+1 end)
s(42)
s(45)
-- inspect the log
s.called_with(42) -- true
s.called_with(42).and_returns_with(43) -- true
s.called_with(43) -- error
s.called_with(42).and_returns_with(44) -- error

--number of times called
s.called() -- true
s.called(1) -- error
s.called(2) -- true

• make_spy(Module, 'fun_name'): hijacks Module.fun_name so that you can track executions of functions inside modules. It provides a clean() method that releases the hijacking.
• stub(Module, 'fun_name', fun): hijacks Module.fun_name and substitutes it for 'fun'.

The whole ungolfed code is (without tests) about 100 lines of lua, which is very impresive for a non-batteries included language.

How

The how is what is interesting. When you make a spy out of a function, spacesuit creates a func table which responds to called_with. called_with  returns a table with and_returns_with key which will do the matching. It's quite a nice usage of lexical scope juggling.

For the hijacking part, I wanted to wrap everything into another table which would have the 'clean' method, and use __call to call the spy table (that would cascade to its __call entry in its metatable, but lua doesn't let you chain __call's. So you have to write the outer one as a function that calls the inner one (and then the __call is run).

Bootstrapped metacompiler using Perl5 and lua

I wrote a Shchorre's metaII implementation myself using perl regexes.

The whole code that is run is just a recursive regexp match against a string (/$bootstrap/ =~ /$program/), which makes it even more mindfucked than usual. It's a simple way to create recursive descent parser just using regexes and perl extended patterns.  The string that tries to match is a representation in meta-II of the very same syntax the string is written on.  Yes.  :-)

I'm taking advantage of the Perl5 extended pattern '(?{})' that runs perl code whenever the regex reaches that point.  The idea is pretty similar to how metaII outputs work themselves even syntax-wise, so I thought it was a nice way to implement it as it's using the same idea that is going to use metaII after being bootstrapped (sorry if this post is difficult to read, but I can't find easy ways to write about without it in clear non-chained-and-recursive-and-self-referent-way). 

To be able to run recursive regexes, we need what MJD calls a proxy parser which is just a delayed 'thunk' that will be evaled just at runtime. We can achieve it in the regex world with (??{}).

If you're not familiar with metacompilers, my advise is to google a bit about them, and find out about them. It's an amazing piece of technology.  Basically you can get a compiler build itself in very few lines of code, and then augment it step by step by modifying the rules it consumes, and creating a slightly more evolved copy of itself, that you can use as a stepping stone to create more advanced compilers.

I added a makefile that shows the process of compiling a compiler using itself and a description of itself.

Here's the repo where there  are more insights in the readme file. Also, check my other posts on metacompilers.

MemoYzing: memoize using Y Combinator

Lately I've had to speed up an openresty-lua application.  As most of the code is just applications of transformations to data, and it's mainly functional, I thought that memoizing would be the easiest way to go.

After generating a flamegraph for the code, I spotted a couple of functions that could be memoized. Problem solved.

While looking for a nice way to write the memoize function, I remembered the shortest memoizing code ever in lua. Also I googled a bit and found kikito's memoize library. So far so good. But they both share a problem. What about recursive functions? They will get catched only on the top level, because the self referencing calls , after memoizing are not self referencing anymore, and they point to the old function.

Perl memoize module overwrites the symbol table to alias the functions. In ruby 2.0 you can memoize a recursive function using Module#prepend. With the Y combinator

Here's this article from Matt Might about how YCombinator makes it possible to turn a recursive function into a memoized recursive function caching the intermediate results, using the indirect self-reference that it provides.


EDIT: I just published the button and then thought "what if I wanna convert a doubly  recursive function (fib) into a iterative one (tail call) by using accumulators? I can obviously memoize according to the two args, but it gets pretty useless, as the results can be hardly reused. I found this series of articles "from recursion to iteration" that provide some tricks. Haven't fully understood it, but I'm on it.

dash docset for nginx lua

Here's a quick hack I wrote to fetch the reference doc for nginx lua module and convert it to a dash docset. Which I view using our beloved helm-dash.

Quick, and kinda easy.  It's totally hackish in the way that I'm parsing HTML with regexps. Not even recursive regexps.....

But the thing is that this html is generated by markdown, and as it's fetching for a very concrete type of lines, they are all formatted the same way.

Another option (less hackish but for this concrete case, equally fragile) is using xmlstarlet. The code is also in the import.sh file (for reference).

So here's the kidd/HttpLuaModule.docset repo.

Improving Lua support in etags

It seems it's Lua time again!

So for my new project I'm starting in openresty+lua, I needed some kind of support for tags. Lua is a very simple language syntaxwise (it's whole grammar fits in a screen of code).

Etags support is quite poor, in fact, if you read the emacs-devel message when it got added, you'll see how basic is it.

+ * Lua tag functions
+ *  look for function, local function.
+ */
+
+static void
+Lua_functions (inf)
+     FILE *inf;
+{
+  register char *bp;
+
+  LOOP_ON_INPUT_LINES (inf, lb, bp)
+    {
+      if (bp[0] != 'f' && bp[0] != 'l')
+       continue;
+
+      LOOKING_AT (bp, "local");
+
+      if (LOOKING_AT (bp, "function"))
+       get_tag (bp, NULL);
     }
 }

The regex version of this would be:   /^(local)?\s+function\s+(\w)/ .

So I wanted to add support for lines not in the beginning, and also to add support for things like

local foo = function (p1, p2) .... end

So it turned out to not be very difficult to augment etags to do that.

tags:
 etags --language=lua --regex='/.*\([^. \t]\)*[ \t]*=[ \t]*function/\1/' \
 --regex='/.*\(local\|\)[ \t][ \t]*function[ \t]\([^ \t(]*\)[ \t]*(/\2/'  **/*lua

Just adding this to the Makefile allows me to catch the other forms of lua functions. Again, regex to the rescue! :)

More JIT and luajit links

Here's another of these list of links post: About JIT and Luajit.

This is hardcore stuff. I've read all the articles but honestly, I barely get what I'd have to do to write something with this on my own. Anyway...

• Hello JIT world
• Unofficial DynAsm docs and tut
• luapower DynAsm page

The luapower site is full of nice tricks for lua libraries. Highly recommended (at least skim it). Specially the design of glue and lua&luajit tricks.

edit: Also, a brainfuck compiler using dynasm
Cya!

multiple value return in lua

Lua functions can return multiple values, and the language will natively assign them to the variables on the other side of the equal sign.

local a, b = (function() return 1,2 end)()
print(a,b) => 1     2

That's fine, but when things get a bit more tricky is when the values are not returned in tail call position.

local a, b = (function()
               local res = (function() return 1 , 2 end)()
               return res
             end)()
print(a, b) =>  1    nil

The catch is that lua assigns the 'rest' of the values only to the last element of tables, or argument lists. If we want to wrap a function into another while not being in tail position, we have to use a little trick. This trick is unpack.

local a, b = (function()
               local res = {(function() return 1 , 2 end)()}
               return unpack(res)
             end)()
print(a, b) =>  1    2

Presenting Eva

I've hacked a tiny little lisp interpreter in lua, and I called Eva, for obvious reasons.

It's not pretty, it's not complete at all. Hell, it doesn't even have strings, you can't call native lua functions, and there are no conses (although you could get them using functions for that).

The whole point of it was to have some fun implementing a tiny lisp-like thingie, and to make it in lua, which is a language that I like quite a lot (although some of its table missfeatures make me cry sometimes)

Without further ado, Eva .

More Lua iterators

After the permutations post, I thought that we needed something like a 'take' function, that would allow us to use infinite streams safely.

local function take(n, it , param, state)
  local count = 0
  return function()
    count = count+1
    if count <= n then
      return it()
    end
    return nil
  end
end


local function take_while(fun, it , param, state)
  return function()
    if fun() then
      return it()
    end
    return nil
  end
end

Simple but nice.

Permutations in Lua. An iterator example

Here's some code I just wrote when messing with an algorithm to generate permutations. The code is just perfect to be used in the form of an iterator, and although I was reading the example in Perl (I read it in Higher Order Perl, an amazing book no matter what's your programming language of choice), I thought Lua would be a good candidate for that.

Along the way, I wrote a few utility functions you can see in there. Mostly tests on function composition and mapping over iterators.

Here's the code. If you need to generate permutations, I found this algorithm (which I don't know the name, but let's call it 'odometer counting') very easy to implement and understand. At least easier than Randal's way of doing it.

local inspect = require'inspect'

function map(f, t)
  local r = {}
  if type(t) == 'table' then
    for _, x in ipairs(t) do
      r[#r+1] = f(x)
    end
  else
    for x in t do
      r[#r+1] = f(x)
    end
  end
  return r
end

function count()
  local c = 0
  return function()
    c = c + 1
    return c
  end
end

-- for x in count() do
--   print(x)
--   if x > 10 then break end
-- end

function permutations(...)
  local function inc(t, pos)
    if t[pos][3] == t[pos][2] then
      if pos == 1 then return nil end
      t[pos][3] = 1
      return inc(t, pos-1)
    else
      t[pos][3] = t[pos][3] + 1
      return true
    end

  end

  local sets = {...}
  local state = map(function(x)
                      return {x, #x , 1}
                    end , sets)
  state[#state][3] = 0

  local curr = #state

  return function()
    while true do
      if inc(state, curr) then
        return map(function(s)
                     return s[1][s[3]] end,
                   state)
      else
        return nil
      end
    end
  end
end

function compose(f,g)
  return function (...)
    return f(g(unpack(arg)))
  end
end

pinspect = compose(print, inspect)

map(pinspect, permutations({1,2,3}, {5,6,7}))

local c = 0
for i in permutations({1,2,3,5,6} , {3,4,5,6,7,6}) do
  c = c+1
  if c == 10 then break end
  pinspect(i)
end

lua and luajit

In the lua workshop, lots of nice topics were raised, but something that striked me was when everyone agreed that writting lua for luajit or for the official lua VM was totally different, and ppl had different mindsets when writting for one or the other.  I know there's a big difference on how to write bindings to C (luajit being more fond of ffi and stock lua prefering the Lua C Api.

If you wanna know a bit more on how luajit works and where it gets its blazing speed, here you have some links to explore it, and to get a grasp of differences between lua implementations.

Agentzh talking about why he (and cloudflare) are targeting mostly luajit:

Nice introduction to luajit. In fact it's mostly lua, but maybe the second part will be more targeted to luajit

Mike Pall has written a few times about how luajit tracer works.

And here it's a functional library which, not being directly linked to luajit, it makes a really nice usage of iterators to build code structures lazily and make the traces jittable by luajit. nice code read.

If you're on the stock lua vm, here you have a paper with a good overview of how lua 5.0 was implemented.

Also, there are a lot of libraries and apps that are targeting just the luajit implementation. It creates some nasty splits on the community, but you know... these things happen.

Reading lua source

As I'm digging deeper into lua, I'm from time to time looking at the lua C source code itself, to see how something is implemented.

It's nice to see a highly commented code with some quite clear parts which goes right to the point. Quite tough though and dense at many other parts.  The whole lua 5.2.2 has a bit more than 14K lines. Not bad for the language, compiler, vm, repl, C Api, and libs.

Anyway, here's a couple of links I found useful in case you want to have a deep look at the lua source:

• Mike Pall suggestions on a sane reading order of the lua 5.1 source
• Guide from the Lua Wiki

Lua Workshop 2013

I was lucky enough to be in the Lua Workshop 2013. Held in Toulouse.

I had lots of fun there, both in the conference and outside it. For me, one of the most important topics raised were the speciation of the Lua world.  Stock Lua, LuaJIT, openresty, luvit ... Many different environments for which lots of packages do not work in different environments (lua-redis and lua-resty-redis, for example). 

This 'problem' extends to other parts of the language and comunity, like packaging. You cannot use luarocks for Luvit, or, the openresty packages are not in LuaRocks. The community is aware of that, and trying to find some compromises to create a healthy ecosystem.  I loved when LuaDist, LuaRocks and the debian packager of lua started an 'impro' discussion on issues they had, pros and cons.

Other talks were also amazing, like Roberto's one: 'Lua, past present and future'. Great way to expose the Lua history and philosophy and reasoning behind some of the features that Lua came to have nowadays. (Spoiler for the future: They are working on Macros!!!)

People modifying the vm to adapt it to their needs were also really wicked cool things we saw there.

Thanks to 3scale for sponsoring my trip, and to the Lua community for being so awesome :).