awk is cool (again)

Awk is lately appearing in HN more often.  I quite like awk, but having gone through Perl, bash, awk, I see people praise awk for being a POSIX compliant enhanced scripting language.  And while being true, I don't think this matters much to the commenters. They see it as an improved bash. And I think we're repeating the history.

Shortcomings that people see in bash, or awk, are solved in Perl. But people dismiss Perl for some reason, and they (we) are going to rediscover it in different forms. Perl is so optimized for scripting that it's really hard to beat. And it has a awk-like mode, so you can easily do the "/pattern/ {action}" thing.

 

Still, here are some threads on awk, with mentions to Perl.  I specially liked K (from K&R) awk.help file. That guy knows how to write!

https://news.ycombinator.com/item?id=28707463

https://earthly.dev/blog/awk-examples/

https://www.cs.princeton.edu/courses/archive/spring19/cos333/awk.help

 https://ferd.ca/awk-in-20-minutes.html

https://www.gnu.org/software/gawk/manual/gawk.pdf

http://www.cs.unibo.it/~sacerdot/doc/awk/nawkA4.pdf

https://ia803404.us.archive.org/0/items/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf

Oh Yes You Can Use Regexes to Parse HTML!

This is Perl, and regexes, and parsing, so if you enjoy those kinds of things, you'll love the comment on this HN Thread. that points to this insane "oh yes, you can use regexes to parse HTML" .

Wow. We've seen all the "you can't parse html with regexes",  and if you were into Perl and knew about the superpowered regexes, you knew it was possible. 

And you might even remember that Regex::Grammars was some amazing Damian  Conway's thing that twisted regexes to their limits.  

Or, my Meta-II compiler implemented in a Perl regex...

 

But it's great to see all those twistings of the common tools. 

Perl & Ruby do APL

So recently I saw a couple of posts that mention APL approaches to problems. The cool thing is that I saw those in a Perl related forum and a Ruby one.

- https://www.youtube.com/watch?v=UBl6t7zNfwE
- https://zverok.github.io/blog/2020-05-16-ruby-as-apl.html

Temporally Quaquaversal Virtual Nanomachine

I've enjoyed this talk so much I'm gonna post it here: Great as always, Damian Conway doing his particular 'GEB' on physics and programming: https://www.youtube.com/watch?v=ORjyXcLDd9M


Also, Keep Ruby Weird 2018, the same as !!Con, very interesting talks.
https://www.youtube.com/watch?v=v32XHJxljKI&list=PLE7tQUdRKcyaxIbXihF5bRdMumqVtGrT4&index=5

 

EDIT: 2020-11-10. Found https://marak.com/blog/2013-05-13-time-loop-software

Indeterminate dimensionality

From: http://www.wall.org/~larry/natural.html
 

Scientists like to be able to locate things by giving a ``vector'', that is, a list of coordinates in a space of known dimensionality. This is one of the reasons they like orthogonality--it means the various components of the vector are independent of each other. Unfortunately, the real world is not usually set up to work that way. Most problems, including linguistics problems, are a matter of ``getting from here to there'', and the geography in-between has a heavy influence on which solutions are practical. Problems tend to be solved at several levels. A typical journey might involve your legs, your car, an escalator, a moving sidewalk, a jet, maybe some more moving sidewalks or a tram, another jet, a taxi, and an elevator. At each of these levels, there aren't many ``right angles'', and the whole thing is a bit fractal in nature. In terms of language, you say something that gets close to what you want to say, and then you start refining it around the edges, just as you would first plan your itinerary between major airports, and only later worry about how to get to and from the airport.

Regex madness

I'm a big fan of regexes, the problems they solve, and the problems they create. It reminds me of Perl :p

Lately I've encountered a couple of very twisted uses of them worth a post.

First of all is a very nasty way of using the regex engine as a sintactic transformer that ends up validating arithmetic operations. Totally nuts.

• http://www.drregex.com/2018/09/a-regex-i-submitted-to-reddit-climbed.html 

Second one is a kind of "sed for power users". It's about internals of sed at a user level (as strange as it seems).

• http://tech.bluesmoon.info/2008/09/programming-patterns-in-sed.html 

After reading drregex post I started hunting for that Abigail's slides on how to solve sudoku using regexes.

• http://perl.abigail.be/Talks/Sudoku/HTML/index.html 

And here is Brian d Foy explaining how to parse json with a single regex.

• https://www.youtube.com/watch?v=-O-Uajd2WzU 

Last one is my metaII bootstrapping parser. A single regex to bootstrap the metacompiler. I guess it also counts as crazyish.

• https://github.com/kidd/meta-II/blob/master/metaII.pl

Praising MJD

Mark Jason Dominus is one of those guys that is worth following everywhere in the net.

• Web: His site is quite old school. It reminds me of c2, or norvig.com, that have really old look but is full of insights and no-bullshit content. Also, loads instantly.
• Blog: Wide range of topics. From systems programming to functional programming, haskell, monads, git, books, alien codes, mathematics.... 
• Book: Higher Order Perl is a great book on functional programming. It uses Perl, but the concepts are really language agnostic. It clearly follows a similar approach to SICP, maybe with a more practical approach.  Higher Order Perl is on my top 10 list of tech books. Top5 if we exclude lisp books.
• Talks: He excels at giving talks also. There are very nice talks about. 
• Systems programming (also, unix process model, unix history, )
• Hash tables
• Parsing combinators
• Programming red flags
• Perl pseudohashes. Nice story of an un-feature 
• Giving talks tips

Bootstrapped metacompiler using Perl5 and lua

I wrote a Shchorre's metaII implementation myself using perl regexes.

The whole code that is run is just a recursive regexp match against a string (/$bootstrap/ =~ /$program/), which makes it even more mindfucked than usual. It's a simple way to create recursive descent parser just using regexes and perl extended patterns.  The string that tries to match is a representation in meta-II of the very same syntax the string is written on.  Yes.  :-)

I'm taking advantage of the Perl5 extended pattern '(?{})' that runs perl code whenever the regex reaches that point.  The idea is pretty similar to how metaII outputs work themselves even syntax-wise, so I thought it was a nice way to implement it as it's using the same idea that is going to use metaII after being bootstrapped (sorry if this post is difficult to read, but I can't find easy ways to write about without it in clear non-chained-and-recursive-and-self-referent-way). 

To be able to run recursive regexes, we need what MJD calls a proxy parser which is just a delayed 'thunk' that will be evaled just at runtime. We can achieve it in the regex world with (??{}).

If you're not familiar with metacompilers, my advise is to google a bit about them, and find out about them. It's an amazing piece of technology.  Basically you can get a compiler build itself in very few lines of code, and then augment it step by step by modifying the rules it consumes, and creating a slightly more evolved copy of itself, that you can use as a stepping stone to create more advanced compilers.

I added a makefile that shows the process of compiling a compiler using itself and a description of itself.

Here's the repo where there  are more insights in the readme file. Also, check my other posts on metacompilers.

Permutations in Lua. An iterator example

Here's some code I just wrote when messing with an algorithm to generate permutations. The code is just perfect to be used in the form of an iterator, and although I was reading the example in Perl (I read it in Higher Order Perl, an amazing book no matter what's your programming language of choice), I thought Lua would be a good candidate for that.

Along the way, I wrote a few utility functions you can see in there. Mostly tests on function composition and mapping over iterators.

Here's the code. If you need to generate permutations, I found this algorithm (which I don't know the name, but let's call it 'odometer counting') very easy to implement and understand. At least easier than Randal's way of doing it.

local inspect = require'inspect'

function map(f, t)
  local r = {}
  if type(t) == 'table' then
    for _, x in ipairs(t) do
      r[#r+1] = f(x)
    end
  else
    for x in t do
      r[#r+1] = f(x)
    end
  end
  return r
end

function count()
  local c = 0
  return function()
    c = c + 1
    return c
  end
end

-- for x in count() do
--   print(x)
--   if x > 10 then break end
-- end

function permutations(...)
  local function inc(t, pos)
    if t[pos][3] == t[pos][2] then
      if pos == 1 then return nil end
      t[pos][3] = 1
      return inc(t, pos-1)
    else
      t[pos][3] = t[pos][3] + 1
      return true
    end

  end

  local sets = {...}
  local state = map(function(x)
                      return {x, #x , 1}
                    end , sets)
  state[#state][3] = 0

  local curr = #state

  return function()
    while true do
      if inc(state, curr) then
        return map(function(s)
                     return s[1][s[3]] end,
                   state)
      else
        return nil
      end
    end
  end
end

function compose(f,g)
  return function (...)
    return f(g(unpack(arg)))
  end
end

pinspect = compose(print, inspect)

map(pinspect, permutations({1,2,3}, {5,6,7}))

local c = 0
for i in permutations({1,2,3,5,6} , {3,4,5,6,7,6}) do
  c = c+1
  if c == 10 then break end
  pinspect(i)
end

Kiss the cuke

I don't care if you're cuking it right or wrong.

But you can use cukes whatever language you use.

• Ruby, obviously
• Perl? sure, look in CPAN.
• Python.. Yeah, they have it too.
• Smalltalk. It seems we have it too.
• Elisp. WTF? Yeah, why? because we can.

So, there's no excuse, if you like BDD, not to use it.

Invincible Perl

Have you seen the talk about DoSsing websites just using crafted data on forms (POST)? It's the trending topic of this week.

Well, here you have a couple of links related to this talk and some extra info.

• http://www.ocert.org/advisories/ocert-2011-003.html
• http://permalink.gmane.org/gmane.comp.security.full-disclosure/83694
• https://cryptanalysis.eu/blog/2011/12/28/effective-dos-attacks-against-web-application-plattforms-hashdos/

The interesting thing that surprised me (or not so) on that article/talk is that the only language they tested that's not vulnerable to this attack is.... guess what? Perl. Here's the extract of the article:

Julian and Alexander did a great job with checking many programming languages used for web applications for their hash table implementation and hash functions. For all of them they checked, they managed to find a lot of keys mapping to the same output, except for Perl. Perl uses a randomized hash function, i.e. the hash doesn’t only depend on the key, but also on an additional value, that is chosen at startup of the application at random. All other languages also store the query parameters send in an HTTP GET or POST request in an hash table, so that a request with many query parameters all mapping to the same hash value will slowly fill such a hash table, before the first line of code written by the application programmer will be executed. Filling this hash table will usually take several minutes on a decent CPU, so that even a fast web server can be kept busy using a slow connection.

And here you have HN comments

+1 for Perl!

Between offtopic and related here's another nice talk from 28C3 that's having place these days in Berlin.

Introduction to (modern) Perl (course)

Barcelona Perl Mongers are organizing a 8 hours course on Perl intended for programmers that have no or little experience in Perl. It's not an introductory course to programming.

The course will be on November 5th, at UPC (Barcelona). 25 Euros for the course, and an exclusive T-Shirt.

I'll be there, maybe just listening, or helping answering questions.

More info in: http://barcelonapm.wordpress.com/2011/10/21/curs-dintroduccio-a-perl/

I still think Perl

The other day, a friend asked how to split a string by commas, but only if they aren't inside a tag, say enclosed by '<' and '>'. I tried python, but somehow regexes are so hardwired in my head, that I can't work with languages that do not have regexen so hardwired in their heart.

Here it is. Tiny, but gets the job done. And that's what perl is for. a regex with lookbehind stuffed into split. No problem sir!

git push -f to master and you're a dead man

Just came back from the half of my holidays, and came with a quite long TODO list.

One item of the list was a wrapper for git that avoids pushing -f the master branch.

At my new $job (yes, I changed again, you'll be noticed properly when I have more time to write), we use a minimal version of git-flow and we run tests on remote (personal) branches.

That means that sometimes, you have to git push -f to your personal remote branch, and possibly delete history. There's no problem if you do it in your personal branch, but as dumbans we err, and myself, being new to all this git fancy stuff, I was a bit worried about it.

I've come with a solution that should work fairly well and transparently. A wrapper for git that nops dangerous commands.

It's perl, and here it is.




Despite being destructive, I like the equals method, kind of lispy, but using shift as a car/cdr solution.

To make the wrapper work, rename it to 'git', make it executable, and put it somewhere in your path, before /usr/bin/git executable.

Sending links when you cannot send files.

I've been reviewing my ~/bin directory and felt like sharing some of the little scripts there.

Today, here's a little Perl script that helps me when having to give a file (screenshot or something) to someone @work. Most people do it by mail, or using jabber protocol. Unfortunately, emacs-jabber does not allow sending nor receiving files, and opening mail client and attaching files is waaay to slow and boring.

The way I work with this situation is having a simple http server in my computer, and moving the file there. then, I paste the link to my coworker, and he/she just has to click it and download.

The http server lives inside emacs. Oh yeah, that's irrelevant. But I had to say it.

Well, here's a script I wrote in 2 minutes.



Hashing the name is to avoid name guessing. I think I got this idea from a Mark Fowler's post that was doing something like this with dropbox.

Perl, paths, local::lib and why I got stuck for a week

When writting ratfinder2, I started testing it in console, and when the main structure kind of worked (the plugin system is for Yet Another Post(tm) ), I wrote the gui stuff, and plugged alltogether. Everything worked ok when run from console. but if the program is executed directly through a ratpoison binding, the script didn't work.

After some debugging time, and nothing seeming to make sense, trying && failing while(1..Inf);... the solution is here.

The thing is that when executing the perl script directly, .zshrc doesn't get executed, so the path isn't added. Be able to run the script we can use 'use lib "/home/rgrau/perl5/lib/perl5";', or otherwise call the script with the -I flag setting the appropiate path.

I suppose if you bind an icon on gnome/kde/windows to a perl program, you should be aware of that, and bind the icon to the appropiate command.

so now, my ratpoisonrc line is like


bind j exec perl -I/home/kidd/perl5/lib/perl5 ~/bin/rat-finder.pl
bind C-j exec perl -I/home/kidd/perl5/lib/perl5 ~/bin/rat-finder.pl


I know it seems obvious now, that you should tell your perl where the modules are, but once you forget about local::lib.... It can't be unforgotten :). The opposite of THIS

Open2 for dummies

Some days ago I started rewriting ratfinder, and I'm trying to make it cleaner, smarter, more extensible, and (why not) have more fun programming it.

The language of choice is still Perl5, because I'd like to keep in touch with the language, and keep learning new tools and libs for it. Ruby is ok, but honestly, if we're not talking about god's purity (those two old languages), Perl is my favourite bitch.

The first design decision was to make it extensible via a plugin system, where you provide a couple of methods, consume a role (Moose, yeah), and ratfinder2 will do 'the right thing'(tm).

I'll talk about the plugin system in another post, but today, I had a simpler problem (at least at first sight).

One of the multiple ways to spawn new processes in perl, is using the super-duper-overpowered-open primitive, with a pipe attached into the name of the process. I still remember One of Casiano's example on it. Quite mindblowing when you first see it.

But what happens when you want not only INput OR OUTput pipes but INput AND OUTput pipes for a process?

Hey, it's Perl, you know man, TIMTOWTDI. IPC::Run, Capture::Tiny, IPC::Cmd.... but the standard way is, for the moment, IPC::Open2.

It's usage may be trivial for you, but for me, the doc is a bit behind the times, as it uses GLOBS for handles, and the examples didn't clarify much for me.

The key concept for me is knowing that the whole thing works when your write pipe is closed (at least it seems so), so if you can't make your host process close it, you should close it by yourself.

Here's the part of relevant code in my Dmenu.pm



As usual, everything is public, undocumented, buggy, and it's on github.

Modern Perl Toolchain, dip your toe

Yesterday, we had another Bcn PerlMongers meeting, and AlexM showed us some nifty tools he's using in a project he's into.

Basically these are what in the perlsphere is called Modern Perl Toolchain.

Forget about cpan asking tons of questions, forget about only one
location for modules.

With the help of:

• cpanm


• App::Perlbrew


• Dist::Zilla

And some plugins for dzilla, you get isolated perl instalations, with
their own modules (@INC), and a trully easy maintenance of
Boilerplate. In fact, it's as easy as it can get, because, you don't
have to do any dedicated maintenance for them.

I had tried some of the old ways to package apps, and so, and when I
saw the power of these 3 modules (I had already used cpanm, but..), I
felt we got an autoshaving yak.

Here are the slides that AlexM kindly uploaded today.

During the talk, some modules arouse to the conversation, and I
took a look at some of them:

• lib::xi

similar to Casiano's Remote::Use, but probably easier in its usage. a big win.

• Devel::REPL


• Carp::REPL

That's the killer one for me. It hooks on die signal, and spawns a
REPL and you have then access to the whole environment. Great great great.

If you want to set 'breakpoints' you can hook warns instead of just exceptions.

And another cool feature is that you use it just enabling it from the commandline

perl -MCarp::REPL

Perl Higher Order Functions

I've been chit-chatting with a friend who's learning Perl5, and has written some little apps for his own needs. While reviewing his code (not that I'm a programming guru, but I help with what I can), I talked about Higher order functions, and we ended writting some throwaway code to explain the idea.

To clarify the concepts, I wrote some trivial scheme functions-as-data munging in scheme, and translated them to Perl.

What surprised me, was the "wow!" comments about concepts that aren't perl unique features, but general higher order procedures. Certainly, Java and C++ are 'a bit' behind Perl, Ruby, or Lisp, but it's fun to remember how mind blowing are these concepts to newcomers. I discover mind-blowing concepts quite frequently too.

We had a good afternoon, reviewing code, Quines, concurrency, talking about GEB (The book I'd take to a desert island along with SICP)...

Here's part of the code we've written (as a reminder). Nothing useful, but tiny proof of concept of HOP. Ah, btw, I recommened him to read chromatic's Modern Perl book. I recomend it to all of you in the Perlsphere. Great book.

generate Moose classes from CLI

It seems I've been a bit out of Perl lately (last 2 or 3 weeks), but that's not completely true.

I have a couple of projects in mind (one bigger than the other, but both interesting nonetheless), but I have not been in my best moods to program... difficulties to focus. At work, I do exactly 0% perl :(, so I have to push myself to use it...

In one of my projects I'm using OOP intensively (maybe overengineering?, we'll see), and I always tend to forget how to correctly set up a bare Moose class.

So here's what I've written:


Just a simple script that generates scaffolds for Moose classes and gives a couple of hints on attributes' properties. A good friend of the script template I did some time ago

I've tried to make it work the most DWIM-Y way, so you call it like "genclass.pl lib/My/App/Entity" , and generates My::App::Entity class in the appropiate path and with appropiate namespace.

SeeYapp! ;p