Category Archives: Programming

“Beautiful REST + JSON APIs” by Les Hazlewood (notes from the talk)

Here’s a summary of a few interesting technical details from a very good presentation on designing REST APIs by Les Hazlewood. Many interesting, elegant and not so obvious solutions here.

  • Keep resources coarse-grained, especially if you’re designing a public API and you don’t know the use cases.
  • Resources are either collections or single instances. They’re always an object, a noun. Don’t mix the URLs with behavior. /getAccount and /createDirectory can easily explode to /getAllAccounts, /searchAccounts, /createLdapDirectory…
  • Collection is at /accounts, instance at /accounts/a1b2c3. Create, read, update and delete with GET/POST/PUT/PATCH/DELETE… Everyone knows, but anyway.
  • Media types:
    • application/json – regular JSON.
    • application/foo+json – JSON of type foo. In my understanding, roughly corresponding to XML schema or DTD.
    • application/foo+json;application – JSON of type foo, where the response type (entity format?) is application.
    • application/json, text/plain – JSON prefered, text acceptable.
  • Versioning:
    • https://api.foo.com/v1
    • Media-Type application/json+foo;application&v=1
  • HREF:
    • Use instead of IDs
    • Sample use:
      GET /accounts/a11b223
      200 OK
      {
        "href": "https://api.foo.com/v1/accounts/a11b223",
        "givenName": "Tony",
        "surname": "Stark",
        "directory": {
          // Instance reference
          "href": "https://api.foo.com/v1/directories/ffb554"
        },
        "groups": {
          // Collection reference
          "href": "https://api.foo.com/v1/accounts/a11b223/groups"
        }
      }
      
    • Reference (link) expansion:
      GET /accounts/a11b223?expand=directory
      {
        "href": "https://api.foo.com/v1/accounts/a11b223",
        "givenName": "Tony",
        "surname": "Stark",
        "directory": {
          "href": "https://api.foo.com/v1/directories/ffb554",
          "name": "Avengers",
          "creationDate": "2013-08-08T14:55:12Z"
        }  
      }
      
    • Partial representations:
      GET /accounts/a11b223?fields=givenName,surname,directory(name)
      
    • Pagination:
      GET .../applications?offset=50&limit=25
      {
        "href": ".../applications",
        "offset": 50,
        "limit": 25,
        "first": { 
          "href": ".../applications?offset=0"
        },
        "prev": {
          "href": ".../applications?offset=25"
        },
        "items": [
          {"href": "..."},
          {"href": "..."}
        ]
      
  • Many-to-many – represent as another, special resource, e.g. /groupMembership. Not so obvious potential benefits: If you make it a resource, it’s easy to reference from either end of the association. If you want, you can easily add data to the association (e.g. creation date, responsible user, whatever).
  • Error handling – credit to Twilio.
    POST /directories
    409 Conflict
    {
      "status": 409,
      "code": 40924, // because HTTP has too few codes
      "property": "name",
      // Ready-to-use user-friendly message:
      "message": "A directory named Avengers already exists",
      // Dev-friendly message, can be stacktrace etc.
      "developerMessage": "A directory named Avengers already 
        exists. If you have a stale local cache, please expire
        it now.",
      "moreInfo": "http://foo.com/docs/api/errors/40924"
    }
    
  • Security – many interesting points here, but one particularly valuable. Authorize on content, not URL. URLs can change and may diverge from security configuration…

It’s too bad that the presenter doesn’t really leave the happy CRUD path. I’d really like to hear his recommendation or examples on more action-oriented use cases (validate or reset user password, approve order, what have we).

The entire presentation is a bit longer than that. I talks quite a lot about why you would want to use REST and JSON, as well as security, caching and other issues.

Clojure on Pedestal

Yesterday I gave a two-hour talk at Lambda Lounge Kraków on Pedestal (and some ClojureScript).

I talked only about the client side, and in that mostly about the dataflow engine and how it connects to rendering and services. We also had some unplanned discussion on testing.

Most of the talk was actually a tutorial, developing a basic monitoring application in a few steps. It probably doesn’t add much over the tutorial, but hopefully it was a faster introduction.

If you’re interested in resources:

Now that I’m on the Internet, I guess I have to explain myself. Yes, I am aware that there are better ways than that jQuery, that I could have created a library for time series, and that it would not run after advanced Closure compilation. It’s not even a perfect Pedestal app. But I think it did the job of explaining the concepts behind Pedestal without drowning the audience in details, using basic pieces that everyone was familiar with. Let’s say that I left ironing out the wrinkles as an exercise.

Despite my lack of experience from my point of view it went fairly well. Always an interesting experience. If you’ve been there, feel free to tell me what you think.

Strange Loop 2013

This year I had the pleasure to be at the Strange Loop conference in Saint Louis. Here’s a brief recap of what was happening there.

Preconf: Emerging Languages Camp

On the first day I sat in about a dozen presentations related to “emerging languages”. The good ones:

  • Daniel Gregoire – Gershwin: stack-based, concatenative Clojure. There were a few presentations of stack-based languages here, but in my opinion this one was the best. It’s been the first time I actually heard about such languages. Not very practical, but quite an interesting excercise none the less.
  • Bento Miso – Daimio: a language for sharing. An attempt at creating a language for creating web plugins, with sandboxing, clear and isolated execution spaces, interactions defined in in/out ports… A bit too hard to follow though..
  • Andreas Rumpf – Nimrod – very good presentation of quite interesting and well thought out language intended mainly for game development. Reportedly fast, very expressive, decent compiler with macros and some serious compile time optimizations… Worth a look.
  • Walter Wilson – Axiomatic Languages. Interesting, very theoretical approach to programming. Define programs in terms of axioms and logical reasoning. Want to sort a list? No problem, just define what it means for elements to be ordered and what a permutation is, then say that the desired output is a permutation that is ordered. Rather academic.
  • Tracy Harms – J. The joke says it’s been an emerging language for 23 years. I’ve never been able to get it (not that I tried much), and this presentation was a very good introduction. It included and decently explained a lot of the syntax as well as the underlying flow and concepts. Quite unusual and interesting.
  • Bodil Stokke – BODOL. This gem was one of the wildest things that I’ve seen not only on this conference, but also in my professional life. Ponies, pattern matching, logic programming, lambdas, monads, types, parsing, programming language theory, live coding… All running from Emacs.

I also attended two unsessions.

  • One was a live recording session of the Think Distributed podcast. 4 smart folks took questions from audience and shared their thoughts on testing, programming, handling failures etc. in distributed environment. Worthwhile.
  • The other un-session was largely based on the she++ documentary on bringing women to programming (and technology in general). Important problem that reappeared a few more times at the conference. It’s worth mentioning, though, that Strange Loop had an unusually high number of female speakers. They might have been preferred in some ways, but nonetheless delivered many awesome talks and the gender did not really matter.

Day One

  • Jenny Finkel – Machine Learning for Relevance and Serendipity. In this opening keynote we got to learn some of the challenges behind Prismatic. (If you haven’t seen Prismatic yet, do it now. It’s awesome.). Very good insights on the discovery problem, machine learning, collective intelligence, user behavior, statistics…
  • Jason Gilman – Visualization Driven Development. One more stab at the problem of using visual aids in programming. This time more like: Tired of waiting for the tools? Roll your own, very custom for the code you’re currently working with.
  • Philipp Haller – Scala Async. I don’t really do Scala, but I liked this presentation a lot. Solid, elegant presentation of upcoming features of Scala for async programming (inspired by F# and C#). Lots of good code, easy to follow examples and great explanation of the matter.
  • Craig Andera – Real-World Datomic. A little bit of a case study, much insight into design and usage of the database. We got to learn about designing and representing the domain model, all the index types and their use cases and some other internals of Datomic. Great talk.
  • Rich Hickey – Clojure core.async. Very, very good presentation not only on core.async (which I’ve never actually used before), but also on asynchronicity in general, CSP, performance, code expressiveness and cohesion… Not so much syntax and case studies, more of a solid explanation of reasons and usage of the library. Plus Rich is as brilliant a speaker as he is a creator.
  • Craig Muth – Xiki: GUI and Text Interfaces Are Converging. Demonstration of a nice… engine (?) for doing all kinds of things from plain text files or just a text editor: GUI prototyping, using git, databases, rails, file system navigation. Very impressive demos, but is it practical? Seems like it may need a good deal of Emacs-fu or a few dozen keyboard shortcuts to do effectively.
  • Joseph Wilk – Creative Machines. The presenter shared many different attempts to get the computer to create music (as well as some examples of work done by others on this topic). All this trying to answer whether machines can be creative, what it means to be creative, how does human creativity work and whether it can be reproduced by a machine. Passionate, well-delivered and thought-provoking, even though I do not fully agree with the conclusions.
  • Jen Myers – Making Software Development Make Sense to Everyone. Very passionate keynote on encouraging women to technology. Talking about iconic heroines of the past (Sally Ride, Nichelle Nichols), as well as something closer to our field and time. Jen presented the “little”, things that she is directly involved with at the moment (Dev Bootcamp, Girl Develop It), as well as shared her vision of a future where hopefully it no longer will be necessary.

At night there was a wild party in the City Museum. Warehouse turned to museum with lots and lots of industrial or fossilized attractions: High and twisted slides, caves, hanging bus, ball pit, crazy places to crawl and walk and lie and sit everywhere (ceilings, floors, going in all directions). Everything adult sized, and boy were those geeks happy as kids to do all this stuff. Last but not least, it was accompanied by very appropriate cave/jazz/dubstep band Moon Hooch. Words can’t really describe it, you had to experience it.

Day Two

  • Martin Odersky – The Trouble With Types. The first half of this keynote was a dense study of type systems in various languages, including the tradeoffs of dynamic vs. static and strong vs. weak. Martin did a great job explaining why Scala is looking the way it is, and why he thinks a strong, static type system is the way to go. The second half delved deep into Scala and to me was too abstract or needed too deep knowledge about Scala. Still, Odersky is a great speaker and thinker. Bonus points for the joke that Rich Hickey could take those slides and give the same talk and treating us to Bach.
  • Avi Bryant – Add all the things: Abstract algebra meets analytics. Very good presentation with very practical examples of how thinking in basic mathematical terms can make software development easier. It showed how you can apply the group theory to data aggregation and storage. It also had a few interesting tricks generally related to the area of data collection and analytics. Great presenter.
  • Crista Lopes – Exercises in Style. Take one problem (word count in text file) and solve it in one language in several different styles, every time deliberately taking some very specific constraints. Spaghetti script, procedural, functional, continuation passing style… Great exercises, worth a look (much bigger book coming out shortly). Unfortunately, was too slow in the beginning and had to cut a few examples later.
  • Brenton Ashworth – Web Apps in Clojure and ClojureScript with Pedestal. Good talk demonstrating and explaining Pedestal. Might have been a little bit too abstract, but I guess you can only fit so much in one hour. Included an impressive dashboard demo (great user experience and data visualizations).
  • David Pollak – Getting Pushy: Pushing data from server to browser. The creator of Lift, appropriately dressed in a lab coat, did a very good job talking about client-server (or server-client?) interaction in web apps. It involved communication in both directions, nicely encapsulated in one accessible piece that solves all the pain of possible failure modes in HTTP and network itself. Included very slick demos in AngularJS and Lift, as well as a little experiment in Clojure (Plugh) based on core.async and clang. Very good.
  • Josh Bloch, Bob Lee – Java Puzzlers, Strange Loop Edition. A handful of entertaining puzzles that often can suprise even veteran Java developers. Java specification seems to have infinite amounts of craze corners, apparently created only so Josh can have fun with the puzzlers.
  • Chris Granger – Finding a way out. The inventor of Light Table unveiled another interesting project supposed to make programming different and more accessible: Aurora. It made for some very slick and impressive demos, but it’s far from finished or usable as an end-to-end product. This time it looks like the target is not a developer of complex apps, but rather the “spreadsheet programmer”. Really nice, but not sure if very practical, and it would be great to actually see a finished, polished environment (Aurora or Light Table).
  • Douglas Hofstadter – What is a strange loop and what is it like to be one? Very good closing keynote, related to Hofstadter’s books. Impressive speech on life, perception, abstraction and consciousness.
  • David Stutz – Thrown for a Loop: A Carnival of Consciousness. Performance/theater very closely related to the preceding speech and Hofstadter’s works. Good deal of discussion of the material and immaterial, self-consciousness and imagination. Appropriately included variations of Goedel theorem, Escher graphics (plus a good deal of Emacs) and live play of Bach fugues by a brass quintet. Plus some very pleasant acrobatics. Very spectacular and unusual, I wish my English sucked less so I could get more of it.

Final Thoughts

I’m using the words “very good”, “great” and “impressive” all the time. It’s boring, but it doesn’t even do justice to the quality of talks and the whole conference. I had very high expectations for the whole thing, and I tend to be hard to please, yet the reality was simply a dream come true.

In some ways the conference was surprisingly minimalistic – I did not even get a note pad or printed program, no sponsor adverts, well nothing except for the ID badge, breakfast at the hotel and cold catered lunch. On the other hand, we got the wild party at the City Museum, the conference venue was very spectacular and comfortable, not to mention the speaker list. I do not mind the tradeoff at all! That’s the way to do it – cut the semi-important stuff to minimum and spend all the money on what really matters and becomes a truly unforgettable experience. Hats off to Alex Miller for putting it all together.

“Spring Data. Modern Data Access for Enterprise Java” (Book Review)

Spring Data is a relatively young set of tools that seems to be quickly gaining popularity. I got introduced to it by Oliver Gierke at 33rd Degree Conference and immediately got really interested (not to say fell in love).

Spring Data is an elegant combination of tools for almost uniform access to various data stores (JDBC, JPA, MongoDB, Neo4j, Redis and others). It has a very slick way to generate queries automatically (even from method names on repository interfaces), with support for paging and sorting as well as auto-generated CRUD.

It makes it very easy to expose repositories on the web via a REST API (and I mean REST, with HATEOAS in the box). In fact, sometimes all you need is to provide entity mapping (e.g. JPA, but there are ways to do similar things with NoSQL stores) and write the repository interface with the right set of annotations. That can be all you need to get a basic REST data service up and running!

Of course you can mix and match or bring more power in. Write your own queries when you need it. Use the repository as data access layer for a “thicker” business layer. Use whatever other Spring tools you like.

The book itself is really well written. It quickly explains the basic ideas behind the data access API. It starts on familiar ground and shows a new way to solve old annoying problems, demonstrating new, streamlined ways to deal with JPA and JDBC repositories. It also offers a new way to write typesafe queries with all the benefits of IDE support: Querydsl.

Then it shows how you can use most of the same API for NoSQL stores, including MongoDB, Neo4j and Redis. Each of those chapters starts with an introduction to the store itself (what it is, what it’s good for). Then it shows how you can use the Spring Data API for, pardon me, Object-NoSQL mapping and writing repositories and queries. As it progresses it gradually dives deeper in the technical details.

Next section is devoted to rapid application development. It features Spring Roo (which I’m not much interested in, not being a fan of codegen) and the REST repository explorer. The latter is a true gem and worth attention on its own. This chapter is also a very good demonstration of (introduction to?) a complete REST API with hyperlinks, CRUD, search, relationships between resources etc.

Towards the end of the book there are also 3 chapters on Hadoop. The final chapter is devoted to GemFire, a distributed data grid.

Spring Data is definitely worth a close look, and this book is a perfect resource to get started. Authors had a very good plan on what they wanted to say and executed it perfectly. The language is fairly light, clear and easy to follow. The examples are interesting, simple and concise at the same time. Very easy to understand, but not simplistic – they demonstrate the underlying power and do a good job of exposing all the gory details. You get to see a fairly wide and reasonably deep view of the framework.

What else could you possibly want from a technical book? Oh yes, and it has a squirrel on the cover!

Many thanks to Oliver for a copy of the book.

Systems that Run Forever Self-heal and Scale

I recently saw a great presentation by Joe Armstrong called “Systems that run forever self-heal and scale” . Joe Armstrong is the inventor of Erlang and he does mention Erlang quite a lot, but the principles are very much universal and applicable with other languages and tools.

The talk is well worth watching, but here’s a few quick notes for a busy reader or my future self.

General remarks

  • If you want to run forever, you have to have more than one instance of everything. If anything is unique, then as soon as that service or machine goes down your system goes down. This may be due to unplanned outage or routine software update. Obvious but still pretty hard.

  • There are two ways to design systems: scaling up or scaling down. If you want a system for 1,000 users, you can start with design for 10 users and expand it, or start with 1,000,000 users and scale it down. You will get different design for your 1,000 users depending on where you start.

  • The hardest part is distributing data in a consistent, durable manner. Don’t even try to do it yourself, use known algorithms, libraries and products.

    Data is sacred, pay attention to it. Web services and such frameworks? Whatever, anyone can write those.

  • Distributing computations is much easier. They can be performed anywhere, resumed or retried after a failure etc. There are some more suggestions hints on how to do it.

Six rules of a reliable system

  1. Isolation – when one process crashes, it should not crash others. Naturally leads to better fault-tolerance, scalability, reliability, testability and comprehensibility. It all also means much easier code upgrades.

  2. Concurrency – pretty obvious: you need more than one computer to make a non-stop system, and that automatically means they will operate concurrently and be distributed.

  3. Failure detection – you can’t fix it if you can’t detect it. It has to work across machine and process boundaries because the entire machine and process can’t fail. You can’t heal yourself when you have a heart attack, it has to be external force.

    It implies asynchronous communication and message-driven model.

    Interesting idea: supervision trees. Supervisors on higher levels of the tree, workers in leaves.

  4. Fault identification – when it fails, you also need to know why it failed.

  5. Live code upgrade – obvioius must have for zero downtime. Once you start the system, never stop it.

  6. Stable storage – store things forever in multiple copies, distributed across many machines and places etc.

    With proper stable storage you don’t need backups. Snapshots, yes, but not backups.

Others: Fail fast, fail early, let it crash. Don’t swallow errors, don’t continue unless you really know what you’re doing. Better crash and let the higher level process decide how to deal with illegal state.

Actor model in Erlang

We’re used to two notions of running things concurrently: processes and threads. The difference? Processes are isolated, live in different places in memory and one can’t screw the other. Threads can.

Answer from Erlang: Actors. They are isolated processes, but they’re not the heavy operating system processes. They all live in the Erlang VM, rely on it for scheduling etc. They’re very light and you can easily run thousands of them on a computer.

Conclusion

Much of this is very natural in functional programming. Perhaps that’s what makes functional programming so popular nowadays – that in this paradigm it’s so much easier to write reliable, fault-tolerant scalable, comprehensible systems.

Client-Side Routing with Pedestal

It’s fairly common for rich web applications to use some kind of client-side routing: Load the application once, then navigate with special URLs like http://example.com/#/subpage. Browser doesn’t perform a round trip to server for them, but instead the right chunk of JavaScript can rebuild part of the page locally, hide and show elements etc.

JavaScript tools

There is a number of JavaScript tools to do it. I find two examples particularly inspiring.

Flatiron Director lets you do something like:

var author = function () { console.log("author"); };
var books = function () { console.log("books"); };
 var routes = {
  '/author': author,
  '/books': books
};</p>

<p>var router = Router(routes);
router.init();

Then when you visit page ending with #/books, it will call the books function. Very flexible, but then you need to write a lot of code yourself.

Another good example with completely different philosophy is the Angular JS Router. In your page you can bless a special section as the ngView, and configure router with:

$routeProvider
    .when('/author', {templateUrl: '/t_author.html', controller: "authorController"})
    .when('/books', {templateUrl: '/t_books.html', controller: "booksController"})
    .otherwise({redirectTo: '/books'});

When you go to #/author, Angular will replace the contents of ngView with the template and install the selected controller on it. It fits well in the Angular philosophy and has some benefits.

Pedestal

Anyway, how can we use routing with Pedestal? Turns out there is no built-in or even recommended way to do it, but you can plug in whatever you want. After some research I decided to use goog.History from Closure which comes free of charge with ClojureScript.

All code for these examples is in my GitHub repository. Note that there are 3 tags corresponding to each of the solutions.

Attempt 1: Simple, hardcoded

The first thing I did was a rather ugly, hardcoded solution using the entire Pedestal data flow.

First I installed a listener on goog.History that extracts the location token and pushes it to the :route topic in Pedestal input queue:

(defn ^:private set-route [input-queue route]
  (p/put-message input-queue {msg/topic :route msg/type :set-route :value route}))

(defn configure-router 
  ([input-queue] (configure-router input-queue ""))
  ([input-queue default-route]
    (doto (goog.History.)
      (goog.events/listen (goog.object/getValues goog.history/EventType) 
                          (fn [e]
                            (let [token (.-token e)]
                              (if (= "" token)
                                (set-route input-queue default-route)
                                (set-route input-queue token)))))
    (.setEnabled true))))

I plugged in the default transform and emitter that basically push that message up to the application model:

(def count-app {:transform {:route {:init nil :fn #(:value %2)}}
                :emit {:router {:fn app/default-emitter-fn :input #{:route}}}})

(defn ^:private set-route [input-queue route]
  (p/put-message input-queue {msg/topic :route msg/type :set-route :value route}))

The last step on the pipeline is renderer. This one here is fairly uninteresting, it basically replaces contents of some specific div with different text:

(defn route-changed [_ [_ _ old-value new-value] input-queue]
  (.log js/console "Routing from" old-value "to" new-value)
  (let [container (dom/by-id "view-container")]
    (dom/destroy-children! container)
    (dom/append! container (str "<p>" new-value "</p>"))))

The main function:

(defn ^:export main []
  (let [app (app/build count-app)
        render-fn (push/renderer "content" [[:value [:route] route-changed]])]
    (render/consume-app-model app render-fn)
    (configure-router (:input app) "first")
    (app/begin app)))

This solution is hardly reusable, but it’s also very flexible. At any point we can decide to do something specific when switching the location – change something in data or application model, make a server request, etc.

I wouldn’t recommend it, it’s just the very first attempt at the problem.

Attempt 2: Generic

It’s quite obvious that the above solution can be generalized. One way to do it is to generalize the renderer. We can initialize it with some configuration telling it what action to execute for each path. For instance, something like this:

(defn render-route [msg]
  (let [container (dom/by-id "view-container")]
    (dom/destroy-children! container)
    (dom/append! container (str "<p>" msg "</p>"))))

(defn route-first []
  (render-route "This is the first route"))

(defn route-second []
  (render-route "This is the second route"))

(def router-config
  {:routes {"first" route-first
            "second" route-second}
   :default-route "first"
   :listener (fn [old-value new-value]
               (.log js/console "Routing from" old-value "to" new-value))})

There isn’t too much interesting stuff happening here, but it clearly is more reusable.

The renderer is now initialized with this config like:

(defn route-renderer [cfg]
  (fn [_ [_ _ old-value new-value] input-queue]
    (when-let [listener (:listener cfg)]
      (listener old-value new-value))
    (if-let [dispatcher (get-in cfg [:routes new-value])]
      (dispatcher)
      (.log js/console "Unknown route:" new-value))))

I’ll omit the history listener and main for clarity, but I hope the point is clear. Again, code is at Github under the generic tag.

All this solution does is bind a function call to each path. I could easily extract it to a tiny generic library. I could also make it more powerful – for instance, use higher order “constructor functions” that let each action access state or push to the input queue. I could use templating. And so on.

Attempt 3: Generic – light

As I was wrapping that up, I figured out one more way to do it. Remember, both solutions above use the entire Pedestal stack – the history listener pushes a message to input queue, and we need a transform and emitter to pass it up to renderer. Maybe I don’t need to involve the “lower layers” with navigation and rendering?

I realized I can just plug the rendering in the history listener itself:

(defn route-first [input-queue]
  (render-route "This is the first route"))

(defn route-second [input-queue]
  (render-route "This is the second route"))

(def router-config
  {:routes {"first" route-first
            "second" route-second}
   :default-route "first"
   :listener (fn [new-value]
               (.log js/console "Routing to" new-value))})

(defn route-changed [{:keys [input]} route-config]
  (fn [e]
    (let [token (.-token e)
          token (if (= "" token) (:default-route route-config) token)]
      (when-let [listener (:listener cfg)]
        (listener token))
      (if-let [dispatcher (get-in route-config [:routes token])]
        (dispatcher input)
        (.log js/console "Unknown route:" token)))))

(defn configure-router 
  ([app route-config]
    (doto (goog.History.)
      (goog.events/listen 
        (goog.object/getValues goog.history/EventType) 
        (route-changed app route-config))
      (.setEnabled true))))

(defn ^:export main []
  (let [app (app/build)]
    (configure-router app router-config)
    (app/begin app)))

That’s it. It doesn’t even need Pedestal. The router is a generic little thing based only on Google Closure. However, in this case each action also has access to the input queue and it’s pretty obvious how you could expose any other things from Pedestal. You can still push messages to Pedestal, install renderers etc. – but that’s no longer required for routing itself.

Wrapping Up

In the end, I’m fairly satisfied with the last solution. Well, in a way – it still requires me to do a ton of manual work! Not for the routing itself, but for DOM rendering. It feels a lot like jQuery, with tons and tons of tedious, manual DOM manipulation.

I guess I became a little spoilt by Angular, and I’m already experimenting with marrying the two.

All the time I’m facing friction though – if I let Angular do too much, I don’t really have any use for Pedestal. On the other hand, there is quite some impedance mismatch between the Angular “pull-oriented model” and Pedestal’s differential pushes. If I ever come to any sane conclusions on this front I’ll write that up. But that’s another story.

Getting Started with Pedestal on Client Side

I finally got to spend a longer while with Pedestal. It is a pure Clojure web framework, where by pure I mean that you’ll see Clojure on back- as well as front end. For starters I only focused on the front end and that’s what this post is about.

What Is Pedestal?

The documentation is good, but it only explains the fundamental concepts. I think it lacks practical examples and more exhaustive API docs (like inputs and expected outputs of each function, options for each configuration etc.). Or maybe it lacks a tutorial (if I win some time on a lottery, I might write one). There are two good sample applications in their repository: helloworld and chat. Unfortunately, helloworld is very simplistic, and chat is a little bit too complicated for beginners. It does a great job showing off all of the features, but it was a little bit too much for me.

So, after a good while of reading the documentation and reverse engineering the samples, I think I’m finally starting to understand what Pedestal is all about.

On the client side Pedestal is a purely functional, reactive, message-oriented framework. All the state is defined in two pieces: the data model and application model. On the surface each of those models is just a single Clojure map, effectively a tree.

Most of the time you don’t operate on the state directly. You define functions (transforms) that react to messages on specific topics and get current state of data model as argument. They return the new value of the state, and the framework will make it the new data model. Another essential kind of function is emitter: In response to certain messages it emits commands (deltas) to update the application model (like “enable transform”, “disable transform”, “create node”, “set value of node”). Finally, in the application level, you can define functions that change something in the behavior or in DOM in response to those deltas.

Everything goes through this pipeline. If an event happens in GUI, or anywhere in the application, it appends a message to the queue and kicks off all the transforms that bubble up from data model through application model to DOM.

This is a simplistic view that doesn’t do justice to all the features, but I think it’s a good starting point.

Better Sample App for Beginners

During this exploration I wrote another sample application. It’s yet another reincarnation of the TODO app that has a single text entry and a list of everything that has been entered so far. It’s much simpler than the chat demo, and it has plenty of comments explaining what’s going on. This is the kind of application that would help me tremendously during in the learning process, and I hope it saves someone some trouble. You can find the source at Github.

Verdict

My initial verdict of Pedestal? It’s definitely worth a close look! I haven’t built any real apps with it yet, but it looks very promising. It’s a breath of fresh air – I’ve never seen anything similar, especially on the client side with JavaScript. And it feels very appropriate: composed of clean, small, well-defined and isolated pieces with sane flow of information. It appears to lead to much cleaner codebase than JS frameworks like Knockout or Angular with their shared state and dangling functions and callbacks everywhere.

Two Ways to Access Properties in ClojureScript

There are two pairs of complementary functions to set properties on objects in ClojureScript. One is aset and aget, another is set! and .-propname:

(def scope (js-obj))
(aset scope "var1" "Value")
(aget scope "var1")
(def scope (js-obj))
(set! (.-var2 scope) "Value")
(.-var2 scope)

Are they equivalent? Is syntax the only difference?

For a demo, let’s say we set and print two fields on an object, like this:

(def scope (js-obj))
(aset scope "var1" "Value")
(set! (.-var2 scope) "Value")

(.log js/console (str "(aget scope \"var1\") ->" (aget scope "var1")))
(.log js/console (str "(.-var2 scope) ->" (.-var2 scope)))
(.log js/console (str "(.-var1 scope) ->" (.-var1 scope)))
(.log js/console (str "(aget scope \"var2\") ->" (aget scope "var2")))

The resulting “setting” instructions are deceptively similar:

mynamespace.scope = {};
mynamespace.scope["var1"] = "Value";
mynamespace.scope.var2 = "Value";

And of course all combinations print out correctly – I can see this on the console:

(aget scope "var1") ->Value
(.-var2 scope) ->Value
(.-var1 scope) ->Value
(aget scope "var2") ->Value

So far so good.

Now, if you’re at least a little bit serious about using ClojureScript, you have to consider advanced optimizations. What happens in this mode?

My code got compiled to:

var Xm;
Xm = {var1:"Value", lc:"Value"};

And on the console all I see is:

(aget scope "var1") ->Value
(.-var2 scope) ->Value
(.-var1 scope) ->
(aget scope "var2") ->

What happened to var2? Why is set! suddenly incompatible with aget and aset with .-field?

If you look at compiler output in both cases, you should see that in one case the value is bound by name and in the other by symbol. In my understanding, it has a few consequences:

  • Advanced compiler renames symbols, but leaves strings intact.
  • aset and aget operate on names. set! and .-var operate on symbols. If you set a variable using “symbolic reference”, the compiler will rename the symbol so you won’t be able to get it by name anymore. If you set it by name with aset, reading by symbol as .-var will not work either because the symbol is renamed.
  • If you deal only with ClojureScript, you should probably use the symbolic references. They look more idiomatic and will benefit from advanced compiler features such as minimization. On the other hand, in this case you often may want to leverage the idiomatic data bindings – vars, atoms and such.
  • As soon as you want your names to be accessible from outside, or you want to access names set outside, use only aset/aget. It obviously applies to interop with JavaScript libraries. Less obvious case (for me) was libraries that operate on DOM, like Angular or Knockout. If you want that {{user.name}} to work, you cannot use the symbolic version.

It all makes sense now, but I haven’t seen it documented. Well, that’s still true for most ClojureScript.

Angular Tutorial Rewritten to ClojureScript

Over the last few months I learned some more ClojureScript and I finally came back to Angular. First I followed their excellent tutorial. Then I decided to rewrite it to plain Clojure and ClojureScript, and it went pretty well.

I made one change on the go – rather than load JSON files directly from disk, it talks to a Ring-provided service.

Raw files are below:

All code is available at GitHub.

It’s almost a one-to-one rewrite from JavaScript. Compared to the original, it is pretty ugly – for two reasons. The first is that I wanted it to work with advanced Closure compiler, so I had to use explicit dependencies. The second is that there is a lot JavaScript interop.

Many of those issues can be mitigated with a glue layer. It is possible to write functions or macros that would automatically generate array syntax for functions with injected dependencies, create functions automatically converting arguments with clj->js, and provide a better replacement for $scope.property = function(...){...}.

I may do that later, but firstly I wanted to have a one-to-one replacement.

ClojureScript Does Not (Always) Need Painkiller

A few weeks ago I shared my confusion about writing object-oriented ClojureScript and a little library called cljs-painkiller. Thanks to the awesome Clojure / ClojureScript community I soon learned much better ways to do it.

Painkiller Example

I complained that I had to write ClojureScript that looks like this:

(defn Bag []
  (this-as this
           (set! (.-store this) (array))
           this))
 
(set! (.. Bag -prototype -add)
      (fn [val]
        (this-as this
                 (.push (.-store this) val))))
 
(set! (.. Bag -prototype -print)
      (fn []
        (this-as this
                 (.log js/console (.-store this)))))
 
(def mybag (Bag.))
(.add mybag 5)
(.add mybag 7)
(.print mybag)

Wrong! Soon after that article appeared on DZone, David Nolen (@swannodette) showed me a few snippets in plain ClojureScript that do the same thing:

(deftype Bag [store]
  Object
  (add [_ x] (.push store x))
  (print [_] (.log js/console store)))

(defn bag [arr] (Bag. arr))
(defn bag [store]
  (reify
    Object
    (add [this x] (.push store x))
    (print [this x] (.log js/console store))))

Much better, isn’t it? And it compiles to fairly idiomatic, interoperable JavaScript, not some higher-level magic.

I am in two minds about the need to expose store like this. One one hand, it makes all the mutable things explicit. On the other, it means you’re exposing much “private” stuff to the consumer, even requiring it from him. To deal with that, you can hide array creation in constructor function:

(defn bag [] (Bag. (array)))
(defn bag []
  (let [store (array)]
    (reify
      Object
      (add [this x] (.push store x))
      (print [this x] (.log js/console store)))))

Backbone Example Revisited

When I was just starting with ClojureScript, I shared an example with Backbone integration. Then I complained it was downright unusable with any less trivial Backbone code.

Here’s what my sample looked like:

(def MyModel
  (.extend Backbone.Model
    (js-obj
      "promptColor"
      (fn []
        (let [ css-color (js/prompt "Please enter a CSS color:")]
          (this-as this
                   (.set this (js-obj "color" css-color))))))))
 
(def my-model (MyModel.))

It turns out it can be rewritten to:

(def MyModel
  (.extend Backbone.Model 
    (reify Object
      (promptColor [this] 
        (let [ css-color (js/prompt "Please enter a CSS color:")]
          (.set this (js-obj "color" css-color)))))))

(def my-model (MyModel.))

Much noise gone. It seems that such reify call is the way to go in this case.

Saved?

I love being proven wrong by the community, and clearly there are better ways to do it than I thought initially. Actually, when I started my adventure with ClojureScript I was quarreling with the compiler – now I finally am beginning to know what I’m doing.

ClojureScript requires some ceremony around object creation, separating behavior from state and its initialization. In some contexts it is too restrictive, in some it’s just fine.

My last example is the Knockout spike where I had JavaScript like this:

function AppViewModel() {
    this.firstName = ko.observable("Bert");
    this.lastName = ko.observable("Bertington");
    this.fullName = ko.computed(function() {
        return this.firstName() + " " + this.lastName();
    }, this);
    this.capitalizeLastName = function() {
        var currentVal = this.lastName();
        this.lastName(currentVal.toUpperCase());
    };
 
}
 
ko.applyBindings(new AppViewModel());

To those unfamiliar with Knockout, firstName etc. are methods. Particularly interesting methods are fullName and capitalizeLastName. Here we have method created by call to ko.computed, wrapping a function that references other methods of this object. Not so bad in an OO language…

… but in ClojureScript apparently the best you can do is what I did back when I was getting started:

(def my-model
  (js-obj
    "firstName" (.observable js/ko "Bert")
    "lastName" (.observable js/ko "Bertington")
    "fullName" (this-as this (.computed js/ko 
                 (fn []  this (.firstName this)), this))))

(.applyBindings js/ko my-model)

I don’t like this at all. This is where I think macros are really necessary. Could be the painkiller, or some special macros just for Knockout integration.

ClojureScript does not always need painkiller, but as your code gets more interesting macroing your way out may be inevitable. I guess you don’t always need to write such OO code, but when you do – be ware.