Category Archives: Clojure

Careful with def in Clojure

Clojure, Programming

Let’s start with a puzzle. Let’s create a little Leiningen project called careful. Let’s set :main careful.core in project.clj and put this in careful/core.clj:

(ns careful.core
  (:gen-class))

(defn get-my-value []
  (println "Sleeping...")
  (Thread/sleep 5000)
  (println "Woke up")
  "Done")

(def my-def (get-my-value))

(defn -main [& args]
  (println "Hello, World!"))

Here’s the question: What happens when you compile this project?

And the answer is…

$ time lein2 compile
Compiling careful.core
Sleeping...
Woke up
Compilation succeeded.

real	0m7.098s
user	0m5.276s
sys	0m0.212s

Hey, my program actually printed something during compilation! And it took way too much time. I didn’t expect that.

Such an apparently innocuous def can get you in a lot of trouble. Sure, no-one puts a sleep like that, but what about:

  • Code that computes something, perhaps something taking time or space?
  • Code that loads something from the network?

I don’t yet understand why it is resolved at compile time.

But I can understand why using def in that way is not a good idea. It’s an old imperative habit. This may be a perfectly valid imperative program:

public static void main(String[] args) {
	Data data = loadDataFromInternet();
	ProcessedData proc = process(data);
	generateReport(proc);
}

You may be tempted to do it this way in Clojure:

(def data (load-data-from-internet))
(def proc (process data))
(defn -main [& args]
  (generate-report proc))

… but that’s still an imperative style and it feels wrong.

It also is real pain to test.

How about one of these equivalents?

(defn -main [& args]
  (let [data (load-data-from-internet)]
     (generate-report (process proc))))

(defn -main [& args]
   (generate-report (process (load-data-from-internet))))

In the end, I arrived at the following conclusion. You should only use def for constants, some global parameters, dynamic variables, definitions of higher order functions – that kind of static stuff. All logic and behavior belongs in functions.

Update

As djork pointed out at Reddit, it’s because def creates a var in the current namespace with specific value.

It makes some sense when you think of what defn looks like – it’s really a macro wrapping def (also pointed out by djork). And we do expect functions introduced by defn to be compiled, right. Even docs clearly state that defn is the “same as (def name (fn [params* ] exprs*))“.

I still find it very confusing, though. I wonder if I’m just abusing the language.

Second Update

I came back to it later, and I may have finally understood.

This is a perfectly valid statement:

(def my-def (get-my-value))

But what about these?

; Unexpected argument:
(def my-def (get-my-value))

; Type cast exception (vector to number)
(def my-def-2 (+ 2 []))

; Whatever invalid statement
(def my-def-2 (+ 2 +))

Should they throw a compile-time error? It makes sense, right?

Now, when you type this:

(def my-def (get-current-date))

At runtime, do you expect it to have state as of compile time, or as of run time? In other words, should it be date of compilation, or “now” at the time of execution? The latter, right?

I can see why both evaluations (at compile and run time) are needed. Depending on point of view, it’s either some sort of language fragility or developer abusing the language. Either way, the conclusion stays the same: Careful with that def, Eugene.

Discussion

Aside from this blog, there is an interesting discussion with more detail at Reddit. Thanks guys!

Spring & Velocity Tools (No XML)

Clojure, Programming, , , ,

A few months ago I wrote about integrating Spring, Velocity and Tiles. I discovered that one bit was missing from there: Velocity Tools. Two hours of yak shaving, frantic googling and source reading later, I figured out how to add support for Velocity Tools to such project with no XML configuration. Here’s how.

For starters, let’s say I want to use some tools in my Velocity and Tiles pages. Let’s add the LinkTool.

template.vm:

<html>
	<head><title>#tiles_insertAttribute({"name":"title"})#end</title></head>
	<body>
		#tiles_insertAttribute({"name":"body"})#end
		<p>Spring macros work in tiles template, too: #springUrl("/myUrl")</p>
		<p>Do Velocity tools work in template? $link.contextPath</p>
	</body>
</html>

body.vm:

<p>Here's a demonstration that Spring macros work with Tiles: #springUrl("/myUrl")</p>
<p>Do Velocity tools work in Tile? $link.contextPath</p>

When I render the code from previous post, I get this:

Here's a demonstration that Spring macros work with Tiles: /SpringVelocityTiles/myUrl

Do Velocity tools work in Tile? $link.contextPath

Spring macros work in tiles template, too: /SpringVelocityTiles/myUrl

Do Velocity tools work in template? $link.contextPath

Not good.

After some googling, I found a similar question on StackOverflow. It had two helpful answers – one from serg, delegating to this blog post, and another from Scott.

None of them worked out of the box, though. I’m tired of XML configs, and apparently it’s too easy to get weird exceptions related to some Struts tools. No wonder I get them, I don’t use Struts and don’t want any of its tools!

Apparently the issue is that Spring support for Velocity Tools is rubbish. One way out is to write your own ViewResolver or View, and that’s what I did in the end.

For starters, I’ll configure my ViewResolver to use a new view class:

@Bean
public ViewResolver viewResolver() {
	VelocityViewResolver resolver = new VelocityViewResolver();
	resolver.setViewClass(MyVelocityToolboxView.class);
	resolver.setSuffix(".vm");
	return resolver;
}

MyVelocityToolboxView is below. This time I’m pasting it with imports to avoid ambiguity on names like Context or VelocityView.

package pl.squirrel.svt;

import java.util.Map;
import java.util.Set;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.velocity.context.Context;
import org.apache.velocity.tools.Scope;
import org.apache.velocity.tools.ToolboxFactory;
import org.apache.velocity.tools.config.ConfigurationUtils;
import org.apache.velocity.tools.view.ViewToolContext;
import org.springframework.web.servlet.view.velocity.VelocityView;

public class MyVelocityToolboxView extends VelocityView {
	@Override
	protected Context createVelocityContext(Map<String, Object> model,
			HttpServletRequest request, HttpServletResponse response) {
		ViewToolContext context = new ViewToolContext(getVelocityEngine(),
				request, response, getServletContext());
		
		ToolboxFactory factory = new ToolboxFactory();
		factory.configure(ConfigurationUtils.getVelocityView());
		
		for (String scope : Scope.values()) {
			context.addToolbox(factory.createToolbox(scope));
		}

		if (model != null) {
			for (Map.Entry<String, Object> entry : (Set<Map.Entry<String, Object>>) model
					.entrySet()) {
				context.put(entry.getKey(), entry.getValue());
			}
		}
		return context;
	}
}

It’s important that we only use ConfigurationUtils.getVelocityView() – it includes generic tools and view tools, but not Struts tools.

That’s it, now we have a project which uses Tiles for high-level templating, Velocity for individual pages and details, with (hopefully) full support for Spring macros and Velocity tools in all areas. Even if you don’t like Tiles, it may still serve as a good example of how to integrate Spring and Velocity Tools.

I pushed updated code for the demo application to my GitHub repository.

In the irregular habit of posting a sermon on Friday… Less than a week ago I saw an excellent presentation on using Clojure for Spring views. Compared to all this mess and yak shaving here, that Clojure solution is infinitely simpler, more elegant and more powerful at the same time. Too bad it does not have the market share of Spring & Velocity yet.

Confitura 2012 & “Boring” Application of Clojure

Clojure, Programming,

This year I attended the Confitura conference for the first time. Many posts have been written on it, so I’ll focus on one (but not the only) presentation that I found particularly worthwhile.

Clojure…

I started my day with talk on Clojure as HTML Templating Language by Łukasz Baran. Despite all the advocacy for Clojure that I do, I was surprised to find this presentation in the agenda – specialized, concrete talk instead of a generic “Clojure is awesome! Go use it!”. Another surprise came from the fact that someone in Poland was serious about using it commercially. I suspect I wasn’t the only surprised person there, as there were a few dozen people in the room and only few of them used Clojure or even functional programming.

Łukasz showed how you can use Clojure for HTML templating in Java projects. He described a team which moved from Velocity to Clojure ca. 2009. They created a DSL for HTML similar to Hiccup (before Hiccup became popular), and went a step further implementing a component library and other automations. From the very beginning they assumed it would be called from Java, so they wrote a loosely coupled component that you can plug in to Spring. Finally, they’ve been using it in production for years and are very happy with it.

Why do this, and why this way? Compared to Velocity, Clojure is very fast, concise, powerful and productive. It has gentle learning curve (when narrowed down to DSL). It was much easier to introduce Clojure in a big enterprise Java shop this way than to write purely Clojure projects from scratch (this may be a good approach in general, by the way).

Personally, I really liked it that this presentation was limited to such a boring, but concrete area. Everyone knows something about writing view layers in Java webapps. Everyone knows pains of JSP (Velocity & other frameworks or not). This presentation showed that it can be done differently and the pain points can be mitigated. It also was a great proof that Clojure has its place and is no academic black magic. In other words, advocacy done right: Instead of showing off the new tool from an ivory tower, demonstrate how it can solve a concrete everyday problem.

… and the Rest

That was not the only good thing at this Confitura. In fact, I really enjoyed most of the talks I saw there.

Paweł Wrzeszcz gave a very good presentation on How to work remotely and don’t go crazy. He showed many good habits for teams and individuals that let you live a healthy life in a healthy project. Though my personal conclusion is that even if you do everything right as an individual, team culture can kill and sometimes the only way out is… out.

I saw two talks on testing, by Jacek Kiljański and Tomek Kaczanowski. Jacek seemed to be a young passionate who believes in everything he says, but also had a well prepared presentation on Clean Tests with clear message and good examples. The following talk by Tomek was quite different – felt more like a rational, sometimes skeptical veteran sharing war stories. It may be due to my tiredness or combination of high temperature and low oxygen, but I did not find this presentation as sharp and clear as the previuous one. Part of the story might be that Jacek stole Tomek’s thunder and there was much overlap.

After a break I went to Maciek Próchniak’s talk on Scala, CQRS and Event Sourcing, but I was rather disappointed. It was pretty chaotic and shallow. I suspect that if you had a slight idea of CQRS and ES, you could not learn much new – even though the problem at hand had some depth that could be discussed in more detail. On the the hand, it assumed too much of the listener to be suitable for a beginner.

Then I saw Sławek Sobótka’s presentation about Soft aspects for IT experts. It was centered around Dreyfuss skill acquisition model (again, and even Sławek admitted it’s something that appears a few times at each conference). Still, it managed to offer something new by only treating the model as a framework and an excuse to dive into many interesting aspects of psychology. Very professional, enjoyable and worthwhile.

Our day ended with Wojciech Seliga’s keynote titled How to be awesome at a Java developer job interview. Less of a talk, more of an emotional rant, but most of the time I really agreed with the presenter. I know way too many careless, ignorant people who consider themselves experts and neglect common tools and practices, stopped learning years ago or simply don’t know what they’re doing.

All in all, it was a very good conference. More technical and low-level than 33rd Degree. By no means “worse” or “better”, just different. Felt like a family get-together, rather than a big conference with big names talking about big stuff that put things in perspective or show some trends, but are somewhat detached from our daily work.

“Programming Concurrency on the JVM”

Clojure, Programming, , ,

A few years ago when I took concurrency classes pretty much everything I was told was that in java synchronized is key. That’s the way to go, whenever you have multithreading you have to do it this way, period. I also spent quite a while solving many classic and less classic concurrency problems using only this construct, reimplementing more fancy locks using only this construct, preventing deadlocks, starvation and everything.

Later in my career I learned that is not the only way to go, or at least there are those fancy java.util.concurrent classes that take care of some stuff for you. That was nice, but apparently I never took enough time to actually stop and think how those things work, what they solve and why.

The light dawned when I started reading Programming Concurrency on the JVM: Mastering Synchronization, STM, and Actors by Venkat Subramaniam.

The book starts with a brief introduction on why concurrency is important today with its powers and perils. It quickly moves on to a few examples of different problems: IO-intensive task like calculating size of a large directory, and computationally intensive task of calculating prime numbers. Once the ground is set, it introduces three approaches to concurrent programming.

The first way to do it is what I summed up in the first paragraph, and what Venkat calls the “synchronize and suffer” model. Been there, done that, we know how bad it can get. This approach is called shared mutability, where different threads mutate shared state concurrently. It may be tamed (and a few ways to do it are shown in the book), but is a lot harder than it seems.

Another approach is isolated mutability, where each mutable part of state is only accessed by one thread. Usually this is the actor based concurrency model. The third way is pure immutability where there simply is no mutable state. That is typical for functional programming.

In the following chapters the book explores each of those areas in depth. It briefly explains the Java memory model nad shows what options for dealing with shared mutability and coordinating threads exist in core Java. It clearly states why the features from Java 5 are superior to the classic “synchronize and suffer” and describes locks, concurrent collections, executors, atomic references etc. in more detail. This is what most of us typically deal with in our daily Java programming, and the book is a great introduction to those modern (if old, in a way) APIs.

That’s about one third of the book. The rest is devoted to much more interesting, intriguing and powerful tools: software transactional memory and actors.

Sometimes we have to deal with shared mutability, and very often we need to coordinate many threads accessing many variables. The classic synchronization tools don’t have proper support for it: Rolling back changes and preventing one thread from seeing uncommited changes of another is difficult, and most likely they lead to coarse-grained locks which basically lock everything while a thread is mutating something.

We know how relational databases deal with it with their ACID transactional model. Software transactional memory is just that but applied to memory, with proper atomicity, consistency and isolation of transactions. If one thread mutates a transactional reference in transaction, another will not see it until that transaction is committed. There is no need for any explicit locks as the libraries (like Akka or Clojure) monitor what variables you access and mutate in transaction and apply locking automatically. They even can rollback and retry the transaction for you.

Another approach is isolated mutability, a.k.a. actors, best demonstated on Akka. Each actor runs in a single thread and all it can do is receive or pass messages. This is probably closest to the original concept of object-oriented programming (recommended reading by Michael Feathers). You have isolated cells that pass messages to each other, and that’s it. When you have a task to execute, you spawn actors and dispatch it to them as immutable messages. When they’re done, they can call you back by passing another message (if the coordinator is also an actor), or if you’re not that pure you can wait for the result. Either way, eveything is neatly isolated in scope of a single thread.

Lengthy as this summary/review is, it really does not do justice to the book. The book itself is dense with valuable information and practical examples, which are as close to perfection as possible: There are a few recurring problems which are fairly simple and easy to grasp, solved over and over again with different techniques and different languages. There are many examples in Java, Scala, Groovy, Clojure and JRuby, dealing with libraries such as the core Java API, Clojure, Akka, GPars… In a few words, a ton of useful stuff.

Last but not the least, it’s excellently written. If anyone has seen Venkat in real life, this book is all like him – entertaining, but also thought-provoking, challenging and inspiring. It reads like a novel (if not better than some of them) and is very hard to put down until you’re done.

Highly recommended.

IDEs of the Future

Clojure, Life, Programming,

I suspect that by now everyone has seen Bret Victor’s “Inventing on Principle” talk. If you haven’t, here it is:

Bret Victor – Inventing on Principle from CUSEC on Vimeo.

I found it great and inspiring not only for the interactive features, but also for the moral parts. I’ve seen some previous inventions by Bret in the past around WorryDream.com. I wish I had been taught science like this. I try to teach like this, and this definitely is the way I am going to try to teach my children with.

Back to the point, though. Cool as they were, Bret’s interactive examples made me ask myself whether it could work with something more complicated (or less contrived?). I was not quite sure.

Today Prismatic served me Chris Ganger’s Light Table – similar concept, even inspired by Bret, applied to an IDE. Take a look at this video:

The video tells a lot, but if you prefer text you can read more at Chris’ blog.

Now, this is something that probably could work. Maybe not exactly like showed here, maybe not that minimalistic, but it has some ideas that I would really like to see in my IDE today.

  • Showing and searching documentation this way seems absolutely possible. Eclipse does not do it for me yet, though.
  • I would really love to see this kind of view where you focus on a single chunk of code (class, method or function) and can automatically see all the referenced bits (and only them) and navigate back and forth. I imagine a navigable graph that takes me between places, but it shows contents of more than one file at a time, and is not really based on files but “chunks” like functions or classes. Does not seem very far either, and could be as life-changing as multiple desktops or monitors (if not more).
  • Finally, the interactive debugging. Looks great and could work on functional language like Clojure and I can see how it would work there. It would be one hell of an effort to get it working for Java though, with all the encapsulation and baroque ceremony.

All in all, very inspiring ideas. I really hope my IDE does some of those things one day. Keep up the good work, guys!

Ring Handlers – Functional Decorator Pattern

Clojure, Programming,

During our last pairing session with Jacek Laskowski on Librarian there was a brief moment of confusion over Ring handlers. We struggled for a short while trying to figure out what order to put them in and what it really means to have code like:

(def app
  (-> routes
    ; sandbar
    (auth/with-security security-policy log-in)
	; compojure helper that includes a few Ring handlers
    site
	; sandbar again
    session/wrap-stateful-session))

It didn’t take us long to figure it out and the solution turns out to be a very elegant functional flavor of decorator.

It’s easy to dive too deep without proper understanding (and that’s what I admittedly did). Let’s start from the very beginning and see what these bits really mean. For starters, here’s a very basic app in plain Ring that simply returns the entire request:

(defn my-handler [request]
    {:body (str request)})

(def app my-handler)

When I hit http://localhost:3000/?my_param=54 in my browser, in return I get:

{:remote-addr "0:0:0:0:0:0:0:1", 
 :scheme :http, 
 :request-method :get, 
 :query-string "my_param=54", 
 :content-type nil, 
 :uri "/", 
 :server-name "localhost", 
 :headers {"cookie" "__utma=111872281.60059650.1328613066.1328726201.1328785442.5; __utmz=111872281.1328613066.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)", "connection" "keep-alive", "accept-encoding" "gzip, deflate", "accept-language" "pl,en-us;q=0.7,en;q=0.3", "accept" "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "user-agent" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20100101 Firefox/11.0", "host" "localhost:3000"}, 
 :content-length nil, 
 :server-port 3000, 
 :character-encoding nil, 
 :body #<Input org.mortbay.jetty.HttpParser$Input@7c04703c>}

Note that my_param made it to :query-string, but obviously it’s quite inconvenient at this point and that’s not what we really want to deal with.

What is app at this point? No magic here, it’s just a very simple and boring function.

Let’s move on and add one of the seemingly magic Ring handlers – ring.middleware.params/wrap-params:

(def app
  (wrap-params my-handler))

This time for the same URL I get the same map, with a few new entries:

{:remote-addr "0:0:0:0:0:0:0:1", 
 ; Trimmed for brevity
 :query-params {"my_param" "54"}, 
 :form-params {}, 
 :query-string "my_param=54", 
 :params {"my_param" "54"}}

I can see that the wrapper added a few new entries: :query-params, :form-params and :params. Great, just like it was supposed to.

Now, what is app at this point? Just like before, it’s a regular function of request. So what does wrap-params really do? Let’s take a peek at (parts of) its source:

(defn wrap-params
  [handler & [opts]]
  (fn [request]
    (let [request  (if (:query-params request)
                     request
                     (assoc-query-params request))]
      (handler request))))

assoc-query-params is no magic, it simply parses query params and merges it with the request map.

Now let’s take a close look at the last line and back at wrap-params signature. Here’s what’s really going on:

  1. wrap-params takes handler (which is a function of request) as argument. In our case, it’s the trivial function that returns request in body.
  2. It then performs some work, in this case rebinding request to a map with a few more entries.
  3. Eventually it calls the handler that it got as parameter with the augmented request map.

In other words, wrap-params takes a handler function, and returns a function that performs some extra work and invokes the original handler.

Does it look familiar? Yup, it’s the old good decorator pattern. Do some work, then pass control on to the next handler (which can also be a decorator and delegate it further). In this case, though, it’s astonishingly simple (compared to what it takes in Java).

Now let’s say I want to chain one more handler that relies on the previous one. Let’s say I dislike strings and want to map params by Clojure keywords. There’s a handler for it: ring.middleware.keyword-params/wrap-keyword-params.

No need to think too long, let’s jump in and use it:

(def app 
  (wrap-keyword-params (wrap-params my-handler)))

… and I get:

{; Trimmed for brevity
 :params {"my_param" "54"}}

Whoops, that’s not what I expected. wrap-keyword-params was supposed to create a map with keys as keywords, not strings. Why didn’t it work?

Naive intuition tells me to treat wrappers as function calls. I wrap my-handler in wrap-params and pass the result of this invocation to wrap-keyword-params, right? Wrong!

Take a look at a sample wrapper above (wrap-params) and think what we were trying to do. What I really created here is a reversed chain like:

  1. Given a request, map its :params into keywords (wrap-keyword-params).
  2. Then pass control to the next function in chain, wrap-params. It parses query string and adds :params map to request.
  3. Then pass control to my-handler which prints the whole thing to browser

Nothing happens in the first step, because :params does exist at this point – it’s only created by wrap-params in the second step.

If we reverse it, it works like expected:

(def app 
  (wrap-params (wrap-keyword-params my-handler)))

{; Trimmed for brevity
 :params {:my_param "54"}}

To recap, a few things to remember from this lesson:

  • In functional programming, the decorator pattern is elegantly represented as a higher order function. I find it much easier to grasp than the OO flavor – in Java I would need an interface and 3 implementing classes for the job, greatly limiting (re)usability and readability.
  • In case of Ring wrappers, typically we have “before” decorators that perform some preparations before calling the “real” business function. Since they are higher order functions and not direct function calls, they are applied in reversed order. If one depends on the other, the dependent one needs to be on the “inside”.

33rd Degree 2012

Clojure, Life, Programming

After great success of the first edition of 33rd Degree I did not really need anyone to convince me to participate this year. I can honestly say that I’ve been waiting for it for a year, and it definitely was worth it.

Day 1

The conference opened with three keynotes. The first speaker was Raffi Krikorian from Twitter who spent good deal of time explaining what Twitter really is: A gigantic event pump that consumes massive amounts of messages and broadcasts them as fast as it can. He also provided a very coarse-grained look over some pieces of their architecture, including heavy use of parallel scatter-gather algorithms even for fairly basic tasks like rendering a timeline. Another interesting bit was their own Finagle library for asynchronous networking with all the fancy stuff of failure detection and failover/retry, service discovery, load balancing etc. Worth noting is that Raffi did not say a bad word about Rails, but only explained why it did not scale to their level.

The second keynote was Ken Sipe talking about the complexity of complexity. I am not sure who was first, but I’m under the impression that this talk was in many ways similar to Rich Hickey’s Simple Made Easy, except for that it was a lot more fluffy. Rich had a very clear point to make and he did it perfectly, with witty and merciless criticism of some flavors of agile as a little cherry on top.

The third keynote was Venkat Subramaniam on Facts and Fallacies of Everyday Software Development. Addressed to pointy-haired bosses (including inside each one of us), it challenged some common opinions like trusting “proven technologies” (read: fossils or large vendors) and advocated polyglot programming (he can’t help but do it even when he talks about “pure Java”). Hardly anything new, but Venkat’s beloved style still made it an entertaining talk.

After those keynotes we got smaller presentations in parallel tracks with some hard decisions to make. I wish I went to Matthew McCullough’s git workshop. Instead I wasted some time looking at JFrog presentation by Frederic Simon (I expected some interesting stuff on continuous integration & delivery). Sadek Drobi’s talk on Play and non-blocking IO was quite OK, but I did not really buy the product. It may be because I’m not into Play, but I found the solution somewhat overengineered and much less appealing than Finagle.

The last technical talk of the day for me was Wojciech Seliga’s piece on tests around JIRA. Those guys are quality nuts, but I can honestly say I would feel really safe and comfortable in this environment. He explained all kinds of unit tests that they have: Regular “boring” unit tests, integration tests (including talking to other Atlassian products), functional tests and platform tests. Even though most of the functional tests run against an API (and only some on Selenium), testing the whole system takes many hours on a 70-worker CI cluster, and they still do it on each and every commit. That is mostly because of platform and performance tests, where they meticulously test against all kinds of platforms: combinations of OS, architecture, DB and whatnot.

Wojciech openly admitted that sometimes it takes them days to make a build green, but they are still very rigorous about it. He also shared some good tricks. Tests are very neatly organized in categories and you can drill down to see what exactly is failing and why. Failing tests go to quarantine in order to keep developers from assuming that there is a false negative that can be ignored while there can actually be other issues making the build fail. Once something goes into quarantine it’s either fixed or eradicated, there is zero tolerance for failure.

Day 2

I started the second day with Barry O’Reilly’s talk on agile & lean startups. I expected much fluff, but I actually learned a few interesting things. We all now the agile/lean cycle of (rougly speaking) iteratively inventing, creating, testing and evaluating, and we got to hear about it here as well. What was interesting and new to me though was how much you can do in the market research field. Two good examples include: Create very detailed portraits of your potential users, to the point where you can actually imagine this person as if you walked their shoes, feel their needs, think how they may want to use your product etc, eventually leading to detailed sketches and scenarios. Consider several distinct portraits and see what functionaliy and user experience emerges. The second idea that sounds crazy to my introvertic self is actually going to the street, cafes etc. and asking strangers about their needs and opinions. Reportedly they love it (“Building new Facebook? I wanna be a part of it!”).

Then I listened to Matthew McCullough’s presentation on economic games in software (and life, actually). Perfectly prepared, rich and fluent, easily one of the best speakers I saw in action (even when he’s coding, like last year on git and hadoop). Matthew touched a few topics: Money is not the only (and not the most important) part of the equation. It may be better to live in a nice place with your family and friends around you than to be paid $300/hour and work on the South Pole. It goes the other way round as well: If you hire people, make sure you motivate them in other ways – create a great environment, listen to their needs etc. Speaking of money, he also explained some tricks that are used on us every day – such as shaping restaurant menu or software license pallette so that they have some ridiculously overpriced things that no-one would buy only to make you buy other overpriced (but cheaper) products.

Enough soft talks, next one was Venkat Subramaniam talking about concurrency, reportedly in pure Java. Started off with a seemingly trivial synchronization issue that is very hard to do right in Java and easily generates a ton of boilerplate. Then he went on to resolve it in a few lines of Clojure (surprise, surprise), explained the concept of STM and went back to Java, resolving the original problem using this little concurrency library (yes, Clojure in Java). Very witty and entertaining, and even though I know some Clojure I wasn’t aware how easy it is to get it to work for you in Java. It really can be just an accessible concurrency library!

Next on my schedule was Sławek Sobótka’s talk on basic software engineering patterns. I enjoy most of the works that he publishes on his blog and the live presentations. This was no exception, even if most of the stuff wasn’t new to me. In short, it was a gentle introduction to some ideas behind DDD and modeling. From basic building blocks (value objects, entities, aggregates) up to operations (services) and customization (policies), all neatly organized in clear layers and levels. Professional, clear, fluent and to the point.

Then I saw Ken Sipe’s presentation on web security, which was basically a run through OWASP Top Ten threats. Unlike the keynote this one was little fluff much stuff and taught me something about common security issues. I knew most of the threats, but never really went through the OWASP list myself. Very good and thorough presentation, complemented by personal stories and interesting digressions.

Venkat Subramaniam showing Scala in a terribly overcrowded room was the last thing on my agenda for the day. I’ve never done anything in this language and I found this presentation a very gentle and interesting introduction. Looks like something between powerful but chaotic Groovy (that feels more like a scripting language to me) and old and baroque Java. I could actually give it a try, though I am yet to think of an applicable project.

Day 3

I kicked off the last day with Jacek Laskowski’s talk on Clojure. It clearly was not possible to show all the interesting corners of the language, but I think he did a good job at explaining the very basics as well as some concurrency concepts. It could have included more details or examples on why we would want to use those concurrency constructs, but still it managed to spark some interest (and not scare people off), and that alone is a big win.

Then I listened to Bartosz Majsak talking about Arquillian – a JBoss library for running higher level tests (demonstrated on Selenium-driven acceptance tests) using mock classes in a managed container. Quite interesting, I may give it a try when I get to play with JEE webapps.

The last strictly technical talk of the day was Nate Schutta on HTML5. Nate asked the audience which parts of the stack they are interested in, and after the introduction went over what we were most interested in: Canvas, client-side data storage and worker threads. Very good, fluent talk, explaining the foundations and history behind HTML5 and complemented by practical examples.

Finally we got three keynotes. The first one was Nate Schutta on code craft. Just like you may expect from keynote, it was a mile wide and an inch deep talk about code and work quality. Keep it simple, keep it short, write tests, don’t reinvent the wheel, get and respect feedback… you know it. Professional and interesting, but immediately forgettable. Things to take away: PMD, Clover – run them regularly, radiate information and encourage others to participate in improvement. Culture kills.

Second keynote was Jurgen Appelo telling us how to change the world. It was very well prepared, starting with a witty and masochistic self-introduction, but then it continued to explain a 4-tier model of influencing people:

  • Work iteratively and reflect on your progress, for example using the classic Plan-Do-Check-Act approach.
  • Understand and drive individuals. Make them aware to the idea or product. Make them desire it by making it challenging, interesting or in other ways appealing. Finally pass on knowledge and abilities.
  • Affect social interactions and culture. Understand that each successful adoption has several stages: Innovators and early adopters who are personally interested in getting your product or idea (crazy people who are ready to drive around the city looking for the last iPad). The majority or skeptics whose needs are largely different (more interested in the ability to watch a movie on the train than in having the latest and coolest hardware). Finally the laggards who can undo your efforts if you stop early.
  • Shape the environment to reinforce all of the above: Radiate information, create a strong cultural identity, use incentives, infrastructure and institutions that encourage or guard the right direction.

There was much more to it than this shallow summary, and there is a reason why it fits in a 19-page booklet (downloadable).

The last keynote was Uncle Bob Martin demanding professionalism. Let me sum it up. Started off with a weird off-topic discussion on why people see more colors than dogs and les than bees. Then came a good part: Creating software resembles engineering, but not the way we popularly believe. “Regular” engineers and architects create very detailed documents and are relatively cheap compared to the manufacturing and execution. It’s often said that programmers are on the “manufacturing” stage, while actually everything they do is on engineering. They actually produce very detailed documents (source code), and that’s a huge effort compared to execution (compilation and running). So the cost model is inverted.

Then Uncle Bob slowly went to the point. Our computers are many orders of magnitude more powerful than those of the distant past, yet we use very similar constructs. Execution on modern hardware is practically instant and free. Uncle Bob asked where it got us, and someone replied “Angry Birds”. To this point I very much agree: It’s extremely sad how we waste all the power on bloated frameworks or worthless entertainment. But the point that Uncle Bob tried to make was quite different: Since hardware and execution is so cheap, we need to keep the model inverted by making execution as fast as possible, and that is professionalism. Write tests, don’t test through GUI, avoid database.

Let me repeat: Hardware is blazingly fast, so our primary concern should be to keep the build fast so that most of the time is spent on coding, not compiling deploying. Am I the only one who finds it self-conflicting? The issue is not that we create useless junk. It’s not that we miss business requirements. It’s not that we suck at estimation. It’s to keep the cost model inverted. Actually, when Uncle Bob encouraged the audience to ask questions, someone asked: “How to estimate better?” At this point he laughed and left the stage.

One thing worth taking away: Good architecture is what makes decisions about infrastructure deferrable to the last stages of the project. Keep expanding the core and mocking if necessary, and think of database or frameworks last.

Final Word

All in all, I would say this edition was as good as the last one if not better. I paid more attention to technical talks and some of them paid off. One thing that was definitely worse: It was terribly overcrowdeed! It was next to impossible to walk the halls or even the large exhibition area, not to mention lunch or bathrooms. I don’t think making it larger in the same venue was a good idea.

Sequential ID Generation in Congomongo

Clojure, Programming,

By default MongoDB uses 12-byte BSON id for objects. For some reason I wanted to use an increasing sequence of integers.

The samples in documentation are in JSON and I was not sure how they translate to the Java driver. The JSON sample looks like:

function counter(name) {
    var ret = db.counters.findAndModify({query:{_id:name}, update:{$inc : {next:1}}, "new":true, upsert:true});
    // ret == { "_id" : "users", "next" : 1 }
    return ret.next;
}

db.users.insert({_id:counter("users"), name:"Sarah C."}) // _id : 1
db.users.insert({_id:counter("users"), name:"Bob D."}) // _id : 2

After some googling I found an implementation in Java. Just as I expected, it’s much longer and completely different.

public static String getNextId(DB db, String seq_name) {
    String sequence_collection = "seq"; // the name of the sequence collection
    String sequence_field = "seq"; // the name of the field which holds the sequence
 
    DBCollection seq = db.getCollection(sequence_collection); // get the collection (this will create it if needed)
 
    // this object represents your "query", its analogous to a WHERE clause in SQL
    DBObject query = new BasicDBObject();
    query.put("_id", seq_name); // where _id = the input sequence name
 
    // this object represents the "update" or the SET blah=blah in SQL
    DBObject change = new BasicDBObject(sequence_field, 1);
    DBObject update = new BasicDBObject("$inc", change); // the $inc here is a mongodb command for increment
 
    // Atomically updates the sequence field and returns the value for you
    DBObject res = seq.findAndModify(query, new BasicDBObject(), new BasicDBObject(), false, update, true, true);
    return res.get(sequence_field).toString();
}

Not much later I checked docs and source code for Congomongo and discovered fetch-and-modify. I rewrote the Java sample above to Clojure and later polished it using code from this commit by Krzysztof Magiera. In the end my sequence generator looks like this:

(defn next-seq [coll]
  (with-mongo db
    (:seq 
	  (fetch-and-modify :sequences {:_id coll} {:$inc {:seq 1}} :return-new? true :upsert? true))))
	
(with-mongo db 
  (insert! :books {:author "Adam Mickiewicz" :title "Dziady" :_id (next-seq :books)}))

The raw call to insert! could be wrapped in a function or macro to save some boilerplate if there are more collections. For instance:

(defn insert-with-id [coll el]
  (insert! coll (assoc el :_id (next-seq coll))))
  
(with-mongo db
  (insert-with-id :books {:author "Adam Mickiewicz" :title "Dziady"}))

In some circles this probably is common knowledge, but it took me a while to figure it all out.

Connection Management in MongoDB and CongoMongo

Clojure, Programming,

I decided to take the opportunity offered by Jacek Laskowski (in Polish) and take a closer look at interaction with MongoDB in Clojure. It has a nice, challenging learning curve as I haven’t done much practical work in Clojure and I’ve never actually dealt with Mongo before. Double win – learning two interesting things at a time.

The obvious choice for the integration is CongoMongo. It’s really easy to get it all set up and working. The official docs encourage you to simply do this:

(def conn (make-connection "mydb")

(set-connection! conn)

(insert! :robots {:name "robby"})

(fetch-one :robots)

; ... and so on

Easy. Too easy and comfortable. Coming from the old good and heavy JDBC/SQL I felt uneasy with the connection management. How does it work? Does it just open a connection and leave it dangling in the air the whole time? Might be good for a quick spike in REPL, but not for a real application which needs concurrency, is supposed to be running for days and weeks, and so on. How do you maintain it properly?

clojure.contrib.sql has with-connection. That opens the connection, runs something with it and then eventually closes it. CongoMongo has with-mongo, but all it does is bind the argument to *mongo-config* and execute body. Nothing is ever opened or closed.

That seemed insane and broken, until I took a step back and compared source of CongoMongo to documentation of underlying Java driver for MongoDB. The light dawned.

What make-connection really does is create an instance of Mongo and DB (if database name was provided). The result of this function is plain map: {:mongo #<Mongo>, :db #<DB>}.

Javadoc for Mongo say it’s a database connection with internal pooling. For most application, you should have 1 Mongo instance for the entire JVM. A page dedicated to Java Driver Concurrency explains it in more detail: The Mongo object maintains an internal pool of connections to the database. For every request to the DB (find, insert, etc) the java thread will obtain a connection from the pool, execute the operation, and release the connection..

At first I thought CongoMongo docs were misleading. The truth is, it’s just a wrapper for the Java driver. It’s fair for it to assume you know the basic principles of the underlying driver.

So what is called a “connection” here (the Mongo class) is in fact a much more sophisticated object. It maintains a connection pool and creates nice little DB objects for all the data handling, which in turn are smart enough to maintain the actual low-level connections for you. No ceremony, just gets out of the way as quickly as possible and lets you get the job done.

This is amazingly simple and elegant compared to JDBC / JEE / SQL. I guess I soon will be scratching my head over ACID, but at the moment I’m pleasantly surprised with the look of things.

Fill and Print an Array in Clojure

Clojure, Programming, ,

In a post in the “Extreme OO in Practice” series (in Polish), Koziołek used the following example: Fill a 3x3x3 matrix with subsequent natural numbers and print it to stdout.

The example in Java is 27 lines of code (if you remove comments). It is later refactored into 55 lines that are supposed to be more readable, but personally I can’t help but gnash my teeth over it.

Anyway, shortly after reading that example, I thought of what it would look like in Clojure. Here it is:

(def array (take 3 (partition 3 (partition 3 (iterate inc 1)))))
(doall (map #(doall (map println %)) array))

That’s it. I know it’s unfair to compare this to Java and say that it’s shorter than your class and main() declaration. But it is, and it makes the full-time Java programmer in me weep. Anyway, I have two observations.

As I like to point out, Java is awfully verbose and unexpressive. Part of the story is strong typing and the nature of imperative programming. But then again, sometimes imperative programming is very clumsy.

Secondly, this solution in Clojure presents a completely different approach to solving the problem. In Java you would declare an array and then iterate over it in nested loops, incrementing the “current value” in every innermost iteration. In Clojure, you start with an infinite sequence of lazy numbers, partition (group) it in blocks (that’s one row), then group those blocks into a 2D array, and take 3 of those 2D blocks as the final 3D matrix. Same thing with printing the array. You don’t iterate over individual cells, but naturally invoke a function on a sequence.

The difference is subtle, but very important. Note how the Java code needed 4 variables for the filling (“current value” and 3 indexes) and 3 indexes for printing. There was a very clear distinction between data and what was happening to it. In Clojure you don’t need to bother with such low-level details. Code is data. Higher-order functions naturally “serve as” data in the first line, and are used to iterate over this data in the second.