Reinventing the Queue

Yesterday, this HN thread Write your own task queue got me thinking again on the NIH and "build vs buy". 

I'm sometimes an advocate for using own implementations of subsystems if you don't need the whole complexity of premade solutions. But...

I usually wouldn't write a task queue, as this is rarely my core business, and that I don't think my needs are anything custom, but I'll take whatever comes with the package.

That said, I've built one task queue in the past, the excuse being I was working on CommonLisp, and the platform already used all the AWS stack: (DynamoDB and Kinesis Streams) for similar things, and no ready-made solution was available. 

In that case, workers were really an active object with an active loop that kept polling the correct stream, and when a job came, an attribute in a dynamoDB table would be used to track the status. It was simple, and it allowed for reimplementations of the queues for local testing. So yeah, I've done it also...

But I can't see myself doing it when using Ruby/Python/... I feel it's very rare I'm gonna have a need that is not covered in your run off the mill job system.

Anyway, wakatime did the same thing, they implemented their own queue. And they all seem to complain about celery.

And more recent projects that approach queues:

- https://github.com/drpancake/chard. async, no threads/process

- https://github.com/thread/django-lightweight-queue. django tailored one.