Squirrel's

Culture Kills (or Wins)

Konrad Garus — Fri, 18 May 2012 16:17:44 +0000

This post is about one of the things that everyone is aware of to some degree. It feels familiar, but the picture becomes a lot sharper once you put it in a proper perspective.

The Project

There is an older project created a few years ago, perhaps in 2000 or 2005. At the time it was running a single service on a single server for 100 users. The architecture and tools were adequate: One project, one interface, one process. Some shortcuts – that’s fine, you need to get something out to get the money in. Now-standard tools and techniques like TDD, IoC / DI, app servers etc. were nowhere to be seen – either because they were not needed at that time, or they did not even exist in a reasonable form (Spring, Guice, JEE in 2000 or 2005?).

Five or ten years later, the load went up by a few orders of magnitude. So did the number of features. The codebase has been growing and growing, and new features have been added all the time.

Now, let’s consider two situations.

Bad Becomes Worse

What if the architecture and process remained the same from the very beginning? Single project (now many megabytes of source code). Single process with core logic driven by eight-KLOC-long abominations. Many interesting twists related to “temporary” shortcuts from the past. No IoC. No DI. No TDD. Libraries from 2000 or 2005. No agile.

It could be for different reasons. The developers are poor, careless souls who have not improved themselves for so many years and they are not aware that different ways exist (or neglect them). Or they know there are better ways, but are drowned with feature requests.

There is little rotation in the team. A few persistent core members persist, able to navigate in this sea of spaghetti. They are still able to pump out new features. Slower and slower, but anyway. That’s probably the only reason keeping the project alive. They just neglect change, because if the thing still kind of works. Why fix something that is not broken? Why spend money on improving something that’s working?

Now we have a fairly fossilized team and project. I dare say in this shape it can only get worse. Even if the product itself was somewhat interesting and stable, would you like to change your job to join the team and work on it? Dealing with tons of legacy code, in a culture that fears change, with no modern tools at your disposal? With no way to learn anything?

Right.

Very good developers usually already have a job and there is no way they would ever quit for this. You will not be able to hire them, unless you pay insane amount of money. Even then, money is not as good a motivator as genuine passion and interest.

Who would do it? Only people with poor skill, lack of experience, desperates or those who don’t give a damn. They will take forever to get up to speed in this ever growing mudball. And because they’re not top class, chances are the project won’t get any better.

We get a nasty negative feedback loop. Bad code and culture encourages more bad code. Mediocre new hires make it even worse. And so the spiral continues.

Good Gets Better

In the second scenario, the team has some caring craftsmen. They constantly read, learn, think, explore and experiment. They observe their product and process and improve them as they recognize more adequate tools and techniques. Some time along the path they broke it down into modules and instead of a monolithic mudball got an extensible service-oriented architecture. They understood inversion of control and brought it in together with a DI container, refactoring the god classes before they grew out of control. They got higher test coverage. In short, they constantly evaluate what and how they’re doing, how they can make it better, and put that to practice.

Now this team can hire pretty much anyone they like. They may decide to hire inexperienced people with the right attitude and train them. But they also are able to attract passionates who are much above average and who will make it even better.

It creates a sweet positive feedback loop. Great culture never loses the edge and it attracts people who can only make it better.

Quality Matters

That’s why quality and refactoring matter. It’s OK to take a shortcut to get something out. It’s OK to use basic tools for a basic job. But if you never improve, the project will stagnate and rot away.

Sure, fulfilling business needs is important. Having few bugs is important. Avoiding damage (to people or the business) is important. But in the long run if you just keep cranking out features and never retrospect or pay down technical debt, it will become a nasty ever-slowing grind. If you’re lucky, it will just get slower and require some babysitting in production and emergency bug fixing. If you’re less lucky, it will become inoperable and completely unmaintainable if some of the persistent spaghetti wranglers leave or are hit by a truck.

Are We Doomed?

To end this sermon with a positive accent, let’s say that while the feedback loops are strong, they are not unbreakable. Culture change is hard in either direction, but possible. If the “bad” team or its management realizes the situation in time and starts improving, they may be able to shift to the positive loop. Introduce slack or retrospectives, start discussion and slowly, but regularly improve. And if for whatever reason you abandon good practices, letting leaders go or drowning them up to the neck with work, it will start the drift towards the negative loop.

Software for Use

Konrad Garus — Thu, 26 Apr 2012 16:27:31 +0000

Here’s confession of a full time software developer: I hate most software. With passion.

Why I Hate Software

Software developers and people around the process are often very self-centered and care more about having a good time than designing a useful product. They add a ton of cool but useless and bugged features. They create their own layers of frameworks and reinvent everything every time, because writing code is so much more fun than writing, reusing or improving it.

They don’t care about edge cases, bugs, rare conditions and so on. They don’t care about performance. They don’t care about usability. They don’t care about anything but themselves.

Examples? Firefox that has to be killed with task manager because it slows to a crawl during the day on most powerful hardware. Linux which never really cared or managed to solve the issues with drivers for end user hardware. Google maps showing me tons of hotel and restaurant names instead of street names, the exact opposite of what I want when planning a trip. Eclipse or its plugins that require me to kill the IDE from task manager, waste some more time, and eventually wipe out the entire workspace, recreate it and reconfigure.

All the applications with tons of forms, popups, dialogs and whatnot. Every error message that is a page long, has a stacktrace, cryptic code and whatever internal stuff in it. All the bugs and issues in open source software, which is made in free time for fun, rarely addressing edge cases or issues happening to a few percent users because they’re not fun.

It’s common among developers to hate and misunderstand the user. It’s common even at helpdesk, support and many people who actually deal with end users. In Polish there is this wordplay “użyszkodnik”, a marriage of “użytkownik” (user) and “szkodnik” (pest).

What Software Really Is About

Let me tell you a secret.

The only purpose of software is to serve. We don’t live in a vacuum, but are always paid by someone who has a problem to solve. We are only paid for two reasons: To save someone money, or to let them earn more money. All the stakeholders and users care about it is solving their problems.

I’ve spent quite a few years on one fairly large project that is critical for most operations of a corporation. They have a few thousand field workers and a few dozen managers above, and only a handful of people responsible for software powering all this. Important as it is, the development team is a tiny part of the entire company.

Whenever I design a form, a report, an email or whatever that the end user will ever see, the first and most important thing to do is: Get in their shoes. Understand what they really need and what problem they are trying to solve. See how we can provide it to the them so that it’s as simple, concise, self-explanatory and usable as possible. Only then we can start thinking about code and the entire backend, and even then the most important thing to keep in mind is the end user.

We’re not writing software for ourselves. Most of the time we’re not writing it for educated and exceptionally intelligent geeks either. We write it for housewives, grandmas, unqualified workers, accountants, ladies at bookshops or insurance companies, all kinds of business people.

We write it for people who don’t care about software at all and do not have a thorough understanding of it. Nor do they care care how good a time you were having while creating it. They just want to have the job done.

You’re Doing It Wrong

If someone has to ask or even think about how something works, it’s your failure. If they perform some crazy ritual like rebooting the computer or piece of software, or wipe out a work directory, that’s your fault. If they have to go through five dialogs for a job that could be done with two clicks, or are forced to switch between windows when there is a better way, it’s your failure. When they go fetch some coffee while a report that they run 5 times a day is running, it’s your fault. If there is a sequence of actions or form entries that can blow everything up, a little “don’t touch this” red button, it’s your fault. Not the end user’s.

It’s not uncommon to see a sign in Polish offices that reads (sadly, literally): “Due to introduction of a computer system, our operations are much slower. We are sorry for the inconvenience.” Now, that’s a huge, epic failure.

Better Ways

That’s quite abstract, so let me bring up a few examples.

IKEA. I know furniture does not seem as complicated as software, but it’s not that trivial either. It takes some effort to package a cabinet or a chest of drawers in a cardboard box that can be assembled by the end user. They could deliver you some wood and a picture of cabinet, and blame you for not knowing how to turn one into another. They could deliver a bunch of needlessly complicated parts without a manual, and blame the user again.

They know they need to sell and have returning customers, not just feel good themselves and blame others.

What they do is carefully design every single part and deliver a manual with large, clear pictures and not a single line of text. And it’s completely fool-proof and obvious, so that even such a carpentry ignorant as you can assemble it.

LEGO. Some sets have thousands of pieces and are pretty complex. So complex that it would be extremely difficult even for you, craftsman proficient in building stuff, to reproduce.

Again, they could deliver 5,000 pieces and a single picture to you and put the blame on you for being unable to figure it out. Again, that’s not what what they do. They want to sell and they want you to return. So they deliver a 200-page-long manual full of pictures, so detailed and fool-proof that even a child can do it.

There are good examples in software world as well. StackOverflow is nice, but only for certain kind of users. It’s great for the Internet geeks who get the concept of upvotes, gamification, focusing on tiny narrow parts and not wider discussion etc. Much less for all kinds of scientists and, you know, regular people, who seem to be the intended audience of StackExchange.

Google search and maps (for address search, intuitiveness and performance), DuckDuckGo are pretty good. Wolfram Alpha. Skyscanner and Himpunk. Much of the fool-proof Apple hardware and software.

In other words, when you know what it does and how to use it the first time you see it, and it Just Works, it’s great.

Conclusion

Successful startups know it. They want to sell and if they make people think or overly complicate something, people will just walk on by. I guess many startups fail because they don’t realize it. Many established brands try to do it and learn from startups, simplifying and streamlining their UIs (Amazon, MS Office, Ebay…). It’s high time we applied it to all kinds of software, including the internal corporate stuff and open source.

After all, we’re only here to serve and solve problems of real people.

That’s the way you do it.

“Programming Concurrency on the JVM”

Konrad Garus — Sat, 21 Apr 2012 12:35:09 +0000

A few years ago when I took concurrency classes pretty much everything I was told was that in java synchronized is key. That’s the way to go, whenever you have multithreading you have to do it this way, period. I also spent quite a while solving many classic and less classic concurrency problems using only this construct, reimplementing more fancy locks using only this construct, preventing deadlocks, starvation and everything.

Later in my career I learned that is not the only way to go, or at least there are those fancy java.util.concurrent classes that take care of some stuff for you. That was nice, but apparently I never took enough time to actually stop and think how those things work, what they solve and why.

The light dawned when I started reading Programming Concurrency on the JVM: Mastering Synchronization, STM, and Actors by Venkat Subramaniam.

The book starts with a brief introduction on why concurrency is important today with its powers and perils. It quickly moves on to a few examples of different problems: IO-intensive task like calculating size of a large directory, and computationally intensive task of calculating prime numbers. Once the ground is set, it introduces three approaches to concurrent programming.

The first way to do it is what I summed up in the first paragraph, and what Venkat calls the “synchronize and suffer” model. Been there, done that, we know how bad it can get. This approach is called shared mutability, where different threads mutate shared state concurrently. It may be tamed (and a few ways to do it are shown in the book), but is a lot harder than it seems.

Another approach is isolated mutability, where each mutable part of state is only accessed by one thread. Usually this is the actor based concurrency model. The third way is pure immutability where there simply is no mutable state. That is typical for functional programming.

In the following chapters the book explores each of those areas in depth. It briefly explains the Java memory model nad shows what options for dealing with shared mutability and coordinating threads exist in core Java. It clearly states why the features from Java 5 are superior to the classic “synchronize and suffer” and describes locks, concurrent collections, executors, atomic references etc. in more detail. This is what most of us typically deal with in our daily Java programming, and the book is a great introduction to those modern (if old, in a way) APIs.

That’s about one third of the book. The rest is devoted to much more interesting, intriguing and powerful tools: software transactional memory and actors.

Sometimes we have to deal with shared mutability, and very often we need to coordinate many threads accessing many variables. The classic synchronization tools don’t have proper support for it: Rolling back changes and preventing one thread from seeing uncommited changes of another is difficult, and most likely they lead to coarse-grained locks which basically lock everything while a thread is mutating something.

We know how relational databases deal with it with their ACID transactional model. Software transactional memory is just that but applied to memory, with proper atomicity, consistency and isolation of transactions. If one thread mutates a transactional reference in transaction, another will not see it until that transaction is committed. There is no need for any explicit locks as the libraries (like Akka or Clojure) monitor what variables you access and mutate in transaction and apply locking automatically. They even can rollback and retry the transaction for you.

Another approach is isolated mutability, a.k.a. actors, best demonstated on Akka. Each actor runs in a single thread and all it can do is receive or pass messages. This is probably closest to the original concept of object-oriented programming (recommended reading by Michael Feathers). You have isolated cells that pass messages to each other, and that’s it. When you have a task to execute, you spawn actors and dispatch it to them as immutable messages. When they’re done, they can call you back by passing another message (if the coordinator is also an actor), or if you’re not that pure you can wait for the result. Either way, eveything is neatly isolated in scope of a single thread.

Lengthy as this summary/review is, it really does not do justice to the book. The book itself is dense with valuable information and practical examples, which are as close to perfection as possible: There are a few recurring problems which are fairly simple and easy to grasp, solved over and over again with different techniques and different languages. There are many examples in Java, Scala, Groovy, Clojure and JRuby, dealing with libraries such as the core Java API, Clojure, Akka, GPars… In a few words, a ton of useful stuff.

Last but not the least, it’s excellently written. If anyone has seen Venkat in real life, this book is all like him – entertaining, but also thought-provoking, challenging and inspiring. It reads like a novel (if not better than some of them) and is very hard to put down until you’re done.

Highly recommended.

IDEs of the Future

Konrad Garus — Sat, 14 Apr 2012 06:12:42 +0000

I suspect that by now everyone has seen Bret Victor’s “Inventing on Principle” talk. If you haven’t, here it is:

Bret Victor – Inventing on Principle from CUSEC on Vimeo.

I found it great and inspiring not only for the interactive features, but also for the moral parts. I’ve seen some previous inventions by Bret in the past around WorryDream.com. I wish I had been taught science like this. I try to teach like this, and this definitely is the way I am going to try to teach my children with.

Back to the point, though. Cool as they were, Bret’s interactive examples made me ask myself whether it could work with something more complicated (or less contrived?). I was not quite sure.

Today Prismatic served me Chris Ganger’s Light Table – similar concept, even inspired by Bret, applied to an IDE. Take a look at this video:

The video tells a lot, but if you prefer text you can read more at Chris’ blog.

Now, this is something that probably could work. Maybe not exactly like showed here, maybe not that minimalistic, but it has some ideas that I would really like to see in my IDE today.

Showing and searching documentation this way seems absolutely possible. Eclipse does not do it for me yet, though.
I would really love to see this kind of view where you focus on a single chunk of code (class, method or function) and can automatically see all the referenced bits (and only them) and navigate back and forth. I imagine a navigable graph that takes me between places, but it shows contents of more than one file at a time, and is not really based on files but “chunks” like functions or classes. Does not seem very far either, and could be as life-changing as multiple desktops or monitors (if not more).
Finally, the interactive debugging. Looks great and could work on functional language like Clojure and I can see how it would work there. It would be one hell of an effort to get it working for Java though, with all the encapsulation and baroque ceremony.

All in all, very inspiring ideas. I really hope my IDE does some of those things one day. Keep up the good work, guys!

Ring Handlers – Functional Decorator Pattern

Konrad Garus — Tue, 10 Apr 2012 21:28:10 +0000

During our last pairing session with Jacek Laskowski on Librarian there was a brief moment of confusion over Ring handlers. We struggled for a short while trying to figure out what order to put them in and what it really means to have code like:

(def app
  (-> routes
    ; sandbar
    (auth/with-security security-policy log-in)
	; compojure helper that includes a few Ring handlers
    site
	; sandbar again
    session/wrap-stateful-session))

It didn’t take us long to figure it out and the solution turns out to be a very elegant functional flavor of decorator.

It’s easy to dive too deep without proper understanding (and that’s what I admittedly did). Let’s start from the very beginning and see what these bits really mean. For starters, here’s a very basic app in plain Ring that simply returns the entire request:

(defn my-handler [request]
    {:body (str request)})

(def app my-handler)

When I hit http://localhost:3000/?my_param=54 in my browser, in return I get:

{:remote-addr "0:0:0:0:0:0:0:1",
 :scheme :http,
 :request-method :get,
 :query-string "my_param=54",
 :content-type nil,
 :uri "/",
 :server-name "localhost",
 :headers {"cookie" "__utma=111872281.60059650.1328613066.1328726201.1328785442.5; __utmz=111872281.1328613066.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)", "connection" "keep-alive", "accept-encoding" "gzip, deflate", "accept-language" "pl,en-us;q=0.7,en;q=0.3", "accept" "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "user-agent" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20100101 Firefox/11.0", "host" "localhost:3000"},
 :content-length nil,
 :server-port 3000,
 :character-encoding nil,
 :body #}

Note that my_param made it to :query-string, but obviously it’s quite inconvenient at this point and that’s not what we really want to deal with.

What is app at this point? No magic here, it’s just a very simple and boring function.

Let’s move on and add one of the seemingly magic Ring handlers – ring.middleware.params/wrap-params:

(def app
  (wrap-params my-handler))

This time for the same URL I get the same map, with a few new entries:

{:remote-addr "0:0:0:0:0:0:0:1",
 ; Trimmed for brevity
 :query-params {"my_param" "54"},
 :form-params {},
 :query-string "my_param=54",
 :params {"my_param" "54"}}

I can see that the wrapper added a few new entries: :query-params, :form-params and :params. Great, just like it was supposed to.

Now, what is app at this point? Just like before, it’s a regular function of request. So what does wrap-params really do? Let’s take a peek at (parts of) its source:

(defn wrap-params
  [handler & [opts]]
  (fn [request]
    (let [request  (if (:query-params request)
                     request
                     (assoc-query-params request))]
      (handler request))))

assoc-query-params is no magic, it simply parses query params and merges it with the request map.

Now let’s take a close look at the last line and back at wrap-params signature. Here’s what’s really going on:

wrap-params takes handler (which is a function of request) as argument. In our case, it’s the trivial function that returns request in body.
It then performs some work, in this case rebinding request to a map with a few more entries.
Eventually it calls the handler that it got as parameter with the augmented request map.

In other words, wrap-params takes a handler function, and returns a function that performs some extra work and invokes the original handler.

Does it look familiar? Yup, it’s the old good decorator pattern. Do some work, then pass control on to the next handler (which can also be a decorator and delegate it further). In this case, though, it’s astonishingly simple (compared to what it takes in Java).

Now let’s say I want to chain one more handler that relies on the previous one. Let’s say I dislike strings and want to map params by Clojure keywords. There’s a handler for it: ring.middleware.keyword-params/wrap-keyword-params.

No need to think too long, let’s jump in and use it:

(def app
  (wrap-keyword-params (wrap-params my-handler)))

… and I get:

{; Trimmed for brevity
 :params {"my_param" "54"}}

Whoops, that’s not what I expected. wrap-keyword-params was supposed to create a map with keys as keywords, not strings. Why didn’t it work?

Naive intuition tells me to treat wrappers as function calls. I wrap my-handler in wrap-params and pass the result of this invocation to wrap-keyword-params, right? Wrong!

Take a look at a sample wrapper above (wrap-params) and think what we were trying to do. What I really created here is a reversed chain like:

Given a request, map its :params into keywords (wrap-keyword-params).
Then pass control to the next function in chain, wrap-params. It parses query string and adds :params map to request.
Then pass control to my-handler which prints the whole thing to browser

Nothing happens in the first step, because :params does exist at this point – it’s only created by wrap-params in the second step.

If we reverse it, it works like expected:

(def app
  (wrap-params (wrap-keyword-params my-handler)))

{; Trimmed for brevity
 :params {:my_param "54"}}

To recap, a few things to remember from this lesson:

In functional programming, the decorator pattern is elegantly represented as a higher order function. I find it much easier to grasp than the OO flavor – in Java I would need an interface and 3 implementing classes for the job, greatly limiting (re)usability and readability.
In case of Ring wrappers, typically we have “before” decorators that perform some preparations before calling the “real” business function. Since they are higher order functions and not direct function calls, they are applied in reversed order. If one depends on the other, the dependent one needs to be on the “inside”.

Java Puzzlers: Number.MIN_VALUE

Konrad Garus — Tue, 03 Apr 2012 20:57:25 +0000

If I asked you what the largest possible Integer in Java is, you would easily answer 2^31-1, or Integer.MAX_VALUE. Smallest possible Integer? Integer.MIN_VALUE, -2^31.

How about a Long? It’s also a Number, and each Number has MIN_VALUE and MAX_VALUE, and you would be right to refer to them.

Obvious and boring, right?

How about a Double then?

It’s also a Number, and so it has Double.MAX_VALUE, and yes that is the maximum. Minimum? Let me go already, obviously Double.MIN_VALUE!

Nope.

I only learned it today even though I’ve been programming Java for years. Double.MIN_VALUE is actually a positive number. Quick look at docs says it all: It is the smallest positive nonzero value of type double. If you’re really looking for the smallest value of Double, you need to refer to Double.MIN_NORMAL.

Yes, docs are very clear about it. Yes, it is a static constant, so LSP does not really apply (as if Java didn’t violate it all the time). Yet I still think it looks like a duck, it quacks like a duck, but needs batteries. It has different semantics for no good reason, does not follow the principle of least surprise and is plain unintuitive. I add it to my shelf of quirks along with half of Swing, everything around Dates and Calendars, equals and… what am I still doing here by the way?

33rd Degree 2012

Konrad Garus — Fri, 23 Mar 2012 18:06:22 +0000

After great success of the first edition of 33rd Degree I did not really need anyone to convince me to participate this year. I can honestly say that I’ve been waiting for it for a year, and it definitely was worth it.

Day 1

The conference opened with three keynotes. The first speaker was Raffi Krikorian from Twitter who spent good deal of time explaining what Twitter really is: A gigantic event pump that consumes massive amounts of messages and broadcasts them as fast as it can. He also provided a very coarse-grained look over some pieces of their architecture, including heavy use of parallel scatter-gather algorithms even for fairly basic tasks like rendering a timeline. Another interesting bit was their own Finagle library for asynchronous networking with all the fancy stuff of failure detection and failover/retry, service discovery, load balancing etc. Worth noting is that Raffi did not say a bad word about Rails, but only explained why it did not scale to their level.

The second keynote was Ken Sipe talking about the complexity of complexity. I am not sure who was first, but I’m under the impression that this talk was in many ways similar to Rich Hickey’s Simple Made Easy, except for that it was a lot more fluffy. Rich had a very clear point to make and he did it perfectly, with witty and merciless criticism of some flavors of agile as a little cherry on top.

The third keynote was Venkat Subramaniam on Facts and Fallacies of Everyday Software Development. Addressed to pointy-haired bosses (including inside each one of us), it challenged some common opinions like trusting “proven technologies” (read: fossils or large vendors) and advocated polyglot programming (he can’t help but do it even when he talks about “pure Java”). Hardly anything new, but Venkat’s beloved style still made it an entertaining talk.

After those keynotes we got smaller presentations in parallel tracks with some hard decisions to make. I wish I went to Matthew McCullough’s git workshop. Instead I wasted some time looking at JFrog presentation by Frederic Simon (I expected some interesting stuff on continuous integration & delivery). Sadek Drobi’s talk on Play and non-blocking IO was quite OK, but I did not really buy the product. It may be because I’m not into Play, but I found the solution somewhat overengineered and much less appealing than Finagle.

The last technical talk of the day for me was Wojciech Seliga’s piece on tests around JIRA. Those guys are quality nuts, but I can honestly say I would feel really safe and comfortable in this environment. He explained all kinds of unit tests that they have: Regular “boring” unit tests, integration tests (including talking to other Atlassian products), functional tests and platform tests. Even though most of the functional tests run against an API (and only some on Selenium), testing the whole system takes many hours on a 70-worker CI cluster, and they still do it on each and every commit. That is mostly because of platform and performance tests, where they meticulously test against all kinds of platforms: combinations of OS, architecture, DB and whatnot.

Wojciech openly admitted that sometimes it takes them days to make a build green, but they are still very rigorous about it. He also shared some good tricks. Tests are very neatly organized in categories and you can drill down to see what exactly is failing and why. Failing tests go to quarantine in order to keep developers from assuming that there is a false negative that can be ignored while there can actually be other issues making the build fail. Once something goes into quarantine it’s either fixed or eradicated, there is zero tolerance for failure.

Day 2

I started the second day with Barry O’Reilly’s talk on agile & lean startups. I expected much fluff, but I actually learned a few interesting things. We all now the agile/lean cycle of (rougly speaking) iteratively inventing, creating, testing and evaluating, and we got to hear about it here as well. What was interesting and new to me though was how much you can do in the market research field. Two good examples include: Create very detailed portraits of your potential users, to the point where you can actually imagine this person as if you walked their shoes, feel their needs, think how they may want to use your product etc, eventually leading to detailed sketches and scenarios. Consider several distinct portraits and see what functionaliy and user experience emerges. The second idea that sounds crazy to my introvertic self is actually going to the street, cafes etc. and asking strangers about their needs and opinions. Reportedly they love it (“Building new Facebook? I wanna be a part of it!”).

Then I listened to Matthew McCullough’s presentation on economic games in software (and life, actually). Perfectly prepared, rich and fluent, easily one of the best speakers I saw in action (even when he’s coding, like last year on git and hadoop). Matthew touched a few topics: Money is not the only (and not the most important) part of the equation. It may be better to live in a nice place with your family and friends around you than to be paid $300/hour and work on the South Pole. It goes the other way round as well: If you hire people, make sure you motivate them in other ways – create a great environment, listen to their needs etc. Speaking of money, he also explained some tricks that are used on us every day – such as shaping restaurant menu or software license pallette so that they have some ridiculously overpriced things that no-one would buy only to make you buy other overpriced (but cheaper) products.

Enough soft talks, next one was Venkat Subramaniam talking about concurrency, reportedly in pure Java. Started off with a seemingly trivial synchronization issue that is very hard to do right in Java and easily generates a ton of boilerplate. Then he went on to resolve it in a few lines of Clojure (surprise, surprise), explained the concept of STM and went back to Java, resolving the original problem using this little concurrency library (yes, Clojure in Java). Very witty and entertaining, and even though I know some Clojure I wasn’t aware how easy it is to get it to work for you in Java. It really can be just an accessible concurrency library!

Next on my schedule was Sławek Sobótka’s talk on basic software engineering patterns. I enjoy most of the works that he publishes on his blog and the live presentations. This was no exception, even if most of the stuff wasn’t new to me. In short, it was a gentle introduction to some ideas behind DDD and modeling. From basic building blocks (value objects, entities, aggregates) up to operations (services) and customization (policies), all neatly organized in clear layers and levels. Professional, clear, fluent and to the point.

Then I saw Ken Sipe’s presentation on web security, which was basically a run through OWASP Top Ten threats. Unlike the keynote this one was little fluff much stuff and taught me something about common security issues. I knew most of the threats, but never really went through the OWASP list myself. Very good and thorough presentation, complemented by personal stories and interesting digressions.

Venkat Subramaniam showing Scala in a terribly overcrowded room was the last thing on my agenda for the day. I’ve never done anything in this language and I found this presentation a very gentle and interesting introduction. Looks like something between powerful but chaotic Groovy (that feels more like a scripting language to me) and old and baroque Java. I could actually give it a try, though I am yet to think of an applicable project.

Day 3

I kicked off the last day with Jacek Laskowski’s talk on Clojure. It clearly was not possible to show all the interesting corners of the language, but I think he did a good job at explaining the very basics as well as some concurrency concepts. It could have included more details or examples on why we would want to use those concurrency constructs, but still it managed to spark some interest (and not scare people off), and that alone is a big win.

Then I listened to Bartosz Majsak talking about Arquillian – a JBoss library for running higher level tests (demonstrated on Selenium-driven acceptance tests) using mock classes in a managed container. Quite interesting, I may give it a try when I get to play with JEE webapps.

The last strictly technical talk of the day was Nate Schutta on HTML5. Nate asked the audience which parts of the stack they are interested in, and after the introduction went over what we were most interested in: Canvas, client-side data storage and worker threads. Very good, fluent talk, explaining the foundations and history behind HTML5 and complemented by practical examples.

Finally we got three keynotes. The first one was Nate Schutta on code craft. Just like you may expect from keynote, it was a mile wide and an inch deep talk about code and work quality. Keep it simple, keep it short, write tests, don’t reinvent the wheel, get and respect feedback… you know it. Professional and interesting, but immediately forgettable. Things to take away: PMD, Clover – run them regularly, radiate information and encourage others to participate in improvement. Culture kills.

Second keynote was Jurgen Appelo telling us how to change the world. It was very well prepared, starting with a witty and masochistic self-introduction, but then it continued to explain a 4-tier model of influencing people:

Work iteratively and reflect on your progress, for example using the classic Plan-Do-Check-Act approach.
Understand and drive individuals. Make them aware to the idea or product. Make them desire it by making it challenging, interesting or in other ways appealing. Finally pass on knowledge and abilities.
Affect social interactions and culture. Understand that each successful adoption has several stages: Innovators and early adopters who are personally interested in getting your product or idea (crazy people who are ready to drive around the city looking for the last iPad). The majority or skeptics whose needs are largely different (more interested in the ability to watch a movie on the train than in having the latest and coolest hardware). Finally the laggards who can undo your efforts if you stop early.
Shape the environment to reinforce all of the above: Radiate information, create a strong cultural identity, use incentives, infrastructure and institutions that encourage or guard the right direction.

There was much more to it than this shallow summary, and there is a reason why it fits in a 19-page booklet (downloadable).

The last keynote was Uncle Bob Martin demanding professionalism. Let me sum it up. Started off with a weird off-topic discussion on why people see more colors than dogs and les than bees. Then came a good part: Creating software resembles engineering, but not the way we popularly believe. “Regular” engineers and architects create very detailed documents and are relatively cheap compared to the manufacturing and execution. It’s often said that programmers are on the “manufacturing” stage, while actually everything they do is on engineering. They actually produce very detailed documents (source code), and that’s a huge effort compared to execution (compilation and running). So the cost model is inverted.

Then Uncle Bob slowly went to the point. Our computers are many orders of magnitude more powerful than those of the distant past, yet we use very similar constructs. Execution on modern hardware is practically instant and free. Uncle Bob asked where it got us, and someone replied “Angry Birds”. To this point I very much agree: It’s extremely sad how we waste all the power on bloated frameworks or worthless entertainment. But the point that Uncle Bob tried to make was quite different: Since hardware and execution is so cheap, we need to keep the model inverted by making execution as fast as possible, and that is professionalism. Write tests, don’t test through GUI, avoid database.

Let me repeat: Hardware is blazingly fast, so our primary concern should be to keep the build fast so that most of the time is spent on coding, not compiling deploying. Am I the only one who finds it self-conflicting? The issue is not that we create useless junk. It’s not that we miss business requirements. It’s not that we suck at estimation. It’s to keep the cost model inverted. Actually, when Uncle Bob encouraged the audience to ask questions, someone asked: “How to estimate better?” At this point he laughed and left the stage.

One thing worth taking away: Good architecture is what makes decisions about infrastructure deferrable to the last stages of the project. Keep expanding the core and mocking if necessary, and think of database or frameworks last.

Final Word

All in all, I would say this edition was as good as the last one if not better. I paid more attention to technical talks and some of them paid off. One thing that was definitely worse: It was terribly overcrowdeed! It was next to impossible to walk the halls or even the large exhibition area, not to mention lunch or bathrooms. I don’t think making it larger in the same venue was a good idea.

“Enterprise Integration Patterns”

Konrad Garus — Tue, 06 Mar 2012 09:18:25 +0000

Today messaging is a common tool, but for some reason it also does not seem to be getting much attention. It seems that blogs and schools are busy with OO (how to do it (badly) & let me show you how to it Right), languages, libraries, and other such low-level stuff. On the other hand, messaging has a very strong position on the market and potential employers are more likely to ask you about it than talk about all the 50 libraries in your CV.

Before plunging headfirst into coding I was looking for some solid semi-theoretical introduction that would explain what exactly messaging is, what problems it tries to solve and what exactly you can do with it. I found Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregory Hohpe and Bobby Woolf and decided to give it a try.

The book starts with a gentle introduction to enterprise integration and its challenges and compares four solutions: shared database, file exchange, RPC and messaging. It explains why messaging is sometimes superior to the alternatives (but is fair about the cost).

The next part presents a high-level overview of the messaging concepts: channels, messages, pipes & filters, routers, translators and endpoints. It also shows a non-trivial (but very clear and interesting) example application which demonstrates where each of the pieces fits.

Finally, it goes through each of the concepts in detail, presenting a handful of patterns for each. This third part appears to be a very rich and complete catalog of common patterns, ranging from basic channel design (point-to-point vs. publish-subscribe, datatype channels, guaranteed delivery etc.), through message construction (command vs. document vs. event, request-reply etc.), routing (splitting and aggregation, content-based routing, dynamic routing etc.), and so on.

Each pattern is presented in a similar way. Authors sketch a problem, discuss possible solutions with their pros and cons, and finally present the pattern as the proposed common way to address it. After the discussion many patterns also have code examples in JMS, MSMQ in .net or TIBCO. Between the chapters there also are a few coding “interludes” with a larger application using several patterns together.

I loved this book. It was well thought over. It’s clear from the first to the last page that the authors knew what they want to convey and masterfully designed a book that probably reads very well on all levels, from absolute beginner to someone who has already worked with messaging. It’s not intimidating to beginners, nor is it a shallow entry-level piece that will make everyone else yawn.

It’s not only a “dry” catalog of patterns. Everything is provided in broader context and it really shows how those pieces work together. It’s a very comprehensive introduction to messaging that you can actually use to learn the subject (nothing like trying to learn OO from reading the Gang of Four). On the other hand, the catalog of patterns is very rich and I am quite sure it satisfies most or even all of the needs of real-world applications.

I haven’t read too many books on messaging, but this one seems so good that if you are even slightly interested in the topic, it is a must read.

Sequential ID Generation in Congomongo

Konrad Garus — Wed, 29 Feb 2012 07:47:29 +0000

By default MongoDB uses 12-byte BSON id for objects. For some reason I wanted to use an increasing sequence of integers.

The samples in documentation are in JSON and I was not sure how they translate to the Java driver. The JSON sample looks like:

function counter(name) {
    var ret = db.counters.findAndModify({query:{_id:name}, update:{$inc : {next:1}}, "new":true, upsert:true});
    // ret == { "_id" : "users", "next" : 1 }
    return ret.next;
}

db.users.insert({_id:counter("users"), name:"Sarah C."}) // _id : 1
db.users.insert({_id:counter("users"), name:"Bob D."}) // _id : 2

After some googling I found an implementation in Java. Just as I expected, it’s much longer and completely different.

public static String getNextId(DB db, String seq_name) {
    String sequence_collection = "seq"; // the name of the sequence collection
    String sequence_field = "seq"; // the name of the field which holds the sequence

    DBCollection seq = db.getCollection(sequence_collection); // get the collection (this will create it if needed)

    // this object represents your "query", its analogous to a WHERE clause in SQL
    DBObject query = new BasicDBObject();
    query.put("_id", seq_name); // where _id = the input sequence name

    // this object represents the "update" or the SET blah=blah in SQL
    DBObject change = new BasicDBObject(sequence_field, 1);
    DBObject update = new BasicDBObject("$inc", change); // the $inc here is a mongodb command for increment

    // Atomically updates the sequence field and returns the value for you
    DBObject res = seq.findAndModify(query, new BasicDBObject(), new BasicDBObject(), false, update, true, true);
    return res.get(sequence_field).toString();
}

Not much later I checked docs and source code for Congomongo and discovered fetch-and-modify. I rewrote the Java sample above to Clojure and later polished it using code from this commit by Krzysztof Magiera. In the end my sequence generator looks like this:

(defn next-seq [coll]
  (with-mongo db
    (:seq
	  (fetch-and-modify :sequences {:_id coll} {:$inc {:seq 1}} :return-new? true :upsert? true))))

(with-mongo db
  (insert! :books {:author "Adam Mickiewicz" :title "Dziady" :_id (next-seq :books)}))

The raw call to insert! could be wrapped in a function or macro to save some boilerplate if there are more collections. For instance:

(defn insert-with-id [coll el]
  (insert! coll (assoc el :_id (next-seq coll))))

(with-mongo db
  (insert-with-id :books {:author "Adam Mickiewicz" :title "Dziady"}))

In some circles this probably is common knowledge, but it took me a while to figure it all out.

Integrating Spring, Velocity and Tiles

Konrad Garus — Mon, 20 Feb 2012 12:10:28 +0000

I like Tiles and I heard a lot about Velocity. They seem to serve different purpose and reportedly are easily married together, so I decided to give it a shot and use them both in Spring web app. The integration actually took many hours and was a real rollercoaster during which I learned a lot about all three technologies. Hopefully this post can spare someone this fun and let them focus on business.

Goals

When I use tiles, I don’t like the default (?) approach of tiles.xml. I don’t want to put imports of JS & CSS, page title, navigation, body etc. each in its own file like in the snippet below, because that makes me switch between editor windows.

Obviously I don’t want to put too much detail in tiles.xml, either.

What I really like is having one file per page, assembling the template in one place, such as this piece of JSP: