Monthly Archives: August 2013

Systems that Run Forever Self-heal and Scale

I recently saw a great presentation by Joe Armstrong called “Systems that run forever self-heal and scale” . Joe Armstrong is the inventor of Erlang and he does mention Erlang quite a lot, but the principles are very much universal and applicable with other languages and tools.

The talk is well worth watching, but here’s a few quick notes for a busy reader or my future self.

General remarks

  • If you want to run forever, you have to have more than one instance of everything. If anything is unique, then as soon as that service or machine goes down your system goes down. This may be due to unplanned outage or routine software update. Obvious but still pretty hard.

  • There are two ways to design systems: scaling up or scaling down. If you want a system for 1,000 users, you can start with design for 10 users and expand it, or start with 1,000,000 users and scale it down. You will get different design for your 1,000 users depending on where you start.

  • The hardest part is distributing data in a consistent, durable manner. Don’t even try to do it yourself, use known algorithms, libraries and products.

    Data is sacred, pay attention to it. Web services and such frameworks? Whatever, anyone can write those.

  • Distributing computations is much easier. They can be performed anywhere, resumed or retried after a failure etc. There are some more suggestions hints on how to do it.

Six rules of a reliable system

  1. Isolation – when one process crashes, it should not crash others. Naturally leads to better fault-tolerance, scalability, reliability, testability and comprehensibility. It all also means much easier code upgrades.

  2. Concurrency – pretty obvious: you need more than one computer to make a non-stop system, and that automatically means they will operate concurrently and be distributed.

  3. Failure detection – you can’t fix it if you can’t detect it. It has to work across machine and process boundaries because the entire machine and process can’t fail. You can’t heal yourself when you have a heart attack, it has to be external force.

    It implies asynchronous communication and message-driven model.

    Interesting idea: supervision trees. Supervisors on higher levels of the tree, workers in leaves.

  4. Fault identification – when it fails, you also need to know why it failed.

  5. Live code upgrade – obvioius must have for zero downtime. Once you start the system, never stop it.

  6. Stable storage – store things forever in multiple copies, distributed across many machines and places etc.

    With proper stable storage you don’t need backups. Snapshots, yes, but not backups.

Others: Fail fast, fail early, let it crash. Don’t swallow errors, don’t continue unless you really know what you’re doing. Better crash and let the higher level process decide how to deal with illegal state.

Actor model in Erlang

We’re used to two notions of running things concurrently: processes and threads. The difference? Processes are isolated, live in different places in memory and one can’t screw the other. Threads can.

Answer from Erlang: Actors. They are isolated processes, but they’re not the heavy operating system processes. They all live in the Erlang VM, rely on it for scheduling etc. They’re very light and you can easily run thousands of them on a computer.


Much of this is very natural in functional programming. Perhaps that’s what makes functional programming so popular nowadays – that in this paradigm it’s so much easier to write reliable, fault-tolerant scalable, comprehensible systems.

“Cooking for Geeks” by Jeff Potter; O’Reilly Media

And now for something completely different… cooking!

Cooking for Geeks

I’ve always been intrigued by “Cooking for Geeks”. I came across it several times, and finally when I saw it in O’Reilly Blogger Reviews program I couldn’t resist.

This book is true to its title – it explains the principles of cooking in a slightly different way. It explains the tastes and some basic ideas of balancing and composition. It talks a lot about food consistency and “doneness” in terms of chemical reactions between various components and in response to temperature over time. It shows how the basic principles of cooking and baking work from such perspective. Finally, it has some great points on hardware and foot safety.

All this narrative is interleaved with recipes, placed right after related material. Sauces in taste composition, pizza, bread and cakes in chapters on baking, and so on. They are pretty exceptional – often discussing a few ways to approach a problem, or things to pay attention to, why they matter and what depends on them. Finally, there are some interviews with chefs, geeks and cooking passionates.

It’s very light language, though sometimes quite information-dense and a bit too abstract. I learned a ton from it. A lot of very basic stuff that every homegrown cook does intuitivelly, but you never know why it works this way. Some things may have been too advanced or too abstract, though. I wish there were more of those recipes, which really do a great job of explaining things on real examples. Sometimes I felt it’s a bit too abstract for me, and at the same time it was probably too basic for experts.

All in all, I think it was worth the read. Entertaining and enlightening at the same time, even if not all content is immediately interesting to everyone.

Paper edition could be a bit better. It looks beautiful, and you could keep it handy and even make annotations in it.