Building Clojure services at scale
blog.josephwilk.net
blog.josephwilk.net
For building out a RESTful API, I found Liberator to be very clever and original:
http://clojure-liberator.github.io/liberator/
For handling internal queues, I use Lamina:
https://github.com/ztellman/lamina
For instance, when saving something to the database, I usually put the data in a closure, put the closure in a Lamina "channel" and let some workers execute the closure when the thread-pool and IO constraints allow.
I just started experimenting with the Ribol conditional restart system, for error handling:
For the integration of frontend with backend (MongoDB) we were originally going to use DerbyJS, but I talked the team into trying Sente instead:
https://github.com/ptaoussanis/sente
We are not done, so the jury is still out, but I am very impressed, so far, with Sente.
For dealing with MongoDB, we use Monger, a library that just keeps getting better and better:
In terms of my personal development, I have struggled to find a workflow that allows me to be efficient at the REPL. I am just now starting to use Daniel Szmulewicz "system" which I am hopeful about:
[from TFA]
> This however does come with a cost:
> - We cannot use Clojure’s concurrency primitives (futures/promises/agents).
Running my own large-ish Clojure service (CircleCI), I'm pretty convinced that futures are an anti-pattern in many situations. If your future is for side-effects, rather than for computation, the future call should probably be going on a durable queue being processed by a cluster of machines. (post "/user/foo" [request]
(update-db)
(future (send-user-email))
{:status 200
:body (render-page)})
What happens if email delivery fails? What happens if this box powers off? etc. Queues don't solve all your failure scenarios here, but they're a great first step.As soon as you want to make the use of futures robust, you will basically start reimplementing parts of a work queue library.
E.g. futures assume they will be handled within the lifetime of the current process: they do not survive catastrophic process failure. If you want to overcome that assumption with persistence, you can either reinvent a wheel, or use a work queuing library that has already implemented this for you.
E.g. you can reinvent your own generic on-fail-retry or you can submit the work to a queue managed by library code that has already implemented retrying for you.
Scaling RabbitMQ at SoundCloud: http://vimeo.com/69960105
Edit: usually with durable queues on SSDs
Edit: text version of the talk: http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-r...
What is it about RabbitMQ or Redis durability that you find lacking? What alternative do you recommend?
http://aphyr.com/posts/315-call-me-maybe-rabbitmq
But all systems like this have weak points, just look at the other systems aphyr has analyzed.
http://aphyr.com/posts/293-call-me-maybe-kafka
We chose RabbitMQ, but we're achieving durability by storing messages in a backup database (CouchBase) that appears to handle partitions better. Still not ideal, and more complicated than we'd like.
6) Open your hand. 7) Paste this to 15 comments n the day you said In step 4 he/she will tell you they like you. If you don't do this you will have bad luck in the next hour.