Connection Management in MongoDB and CongoMongo

February 4th, 2012

I decided to take the opportunity offered by Jacek Laskowski (in Polish) and take a closer look at interaction with MongoDB in Clojure. It has a nice, challenging learning curve as I haven’t done much practical work in Clojure and I’ve never actually dealt with Mongo before. Double win – learning two interesting things at a time.

The obvious choice for the integration is CongoMongo. It’s really easy to get it all set up and working. The official docs encourage you to simply do this:

(def conn (make-connection "mydb")
 
(set-connection! conn)
 
(insert! :robots {:name "robby"})
 
(fetch-one :robots)
 
; ... and so on

Easy. Too easy and comfortable. Coming from the old good and heavy JDBC/SQL I felt uneasy with the connection management. How does it work? Does it just open a connection and leave it dangling in the air the whole time? Might be good for a quick spike in REPL, but not for a real application which needs concurrency, is supposed to be running for days and weeks, and so on. How do you maintain it properly?

clojure.contrib.sql has with-connection. That opens the connection, runs something with it and then eventually closes it. CongoMongo has with-mongo, but all it does is bind the argument to *mongo-config* and execute body. Nothing is ever opened or closed.

That seemed insane and broken, until I took a step back and compared source of CongoMongo to documentation of underlying Java driver for MongoDB. The light dawned.

What make-connection really does is create an instance of Mongo and DB (if database name was provided). The result of this function is plain map: {:mongo #<Mongo>, :db #<DB>}.

Javadoc for Mongo say it’s a database connection with internal pooling. For most application, you should have 1 Mongo instance for the entire JVM. A page dedicated to Java Driver Concurrency explains it in more detail: The Mongo object maintains an internal pool of connections to the database. For every request to the DB (find, insert, etc) the java thread will obtain a connection from the pool, execute the operation, and release the connection..

At first I thought CongoMongo docs were misleading. The truth is, it’s just a wrapper for the Java driver. It’s fair for it to assume you know the basic principles of the underlying driver.

So what is called a “connection” here (the Mongo class) is in fact a much more sophisticated object. It maintains a connection pool and creates nice little DB objects for all the data handling, which in turn are smart enough to maintain the actual low-level connections for you. No ceremony, just gets out of the way as quickly as possible and lets you get the job done.

This is amazingly simple and elegant compared to JDBC / JEE / SQL. I guess I soon will be scratching my head over ACID, but at the moment I’m pleasantly surprised with the look of things.

Tags: ,
Posted by Konrad Garus in Programming | No Comments »

Two Stories of Ignaz Semmelweis

January 4th, 2012

Here’s two versions of the story of Ignaz Semmelweis. You may have heard it as it’s quite popular in the programming & craftsmanship community. I heard the two versions about a year apart and together they’re even more interesting.

First Story

Ignaz Semmelweis was a Hungarian physician. He believed that by washing their hands consistently and well enough, doctors can significantly reduce the infection and mortality rate among mothers and their newborn children.

Unfortunately, those findings were in conflict with what most doctors believed at the time. They were prejudiced and had their old, established point of view and would not listen to the poor man.

Semmelweis did not manage to convince his peers to his findings. He died a miserable death in an asylum.

Second Story

Ignaz Semmelweis was a Hungarian physician. He believed that by washing their hands consistently and well enough, doctors can significantly reduce the infection and mortality rate among mothers and their newborn children.

Unfortunately, in the process of achieving his superior results, he managed to alienate or offend his entire profession. Things eventually got so bad that his colleagues started to deliberately avoid washing their hands just to defy him.

Semmelweis was a genius but he was also a lunatic, and that made him a failed genius. He died a miserable death in an asylum.

Which is True?

I heard the first story for the first time on the last year’s 33rd Degree. The way my inner wannabe-craftsman understood it at that time was: poor guy ahead of his time was tormented to death by his prejudiced, petrified community. He was right, and his stubborn peers made his life miserable.

I read the second version in Apprenticeship Patterns by Dave Hoover and Adewale Oshineye. Here the lesson is completely different. There is no way you can get your point across if you alienate or offend your peers. You have to be very patient and careful, especially if your ideas do not go hand in hand with what the community believes at the point.

Even if your peers are willing to listen, and you have convincing evidence or arguments, you can never play the one and only enlightened man in the world. It goes the other way round, too. Learn to listen, even if some ideas don’t exactly go hand in hand with your point of view. See if there is any value you can take from them. And even if you disagree, have some respect and don’t play the “we know better, be gone ye stupid lunatic” role.

I suspect both versions of the story are true. People are full of judgement and prejudice. Such is human nature. It’s rarely a wall that you can take down with a ram. More often you find yourself sitting on a giant elephant and trying to convince it to go in a specific direction. The only way you can succeed is preparing and cutting the way through the jungle and pointing the elephant into the opening.

This elephant rider metaphor comes from an awesome talk by Venkat Subramaniam. Heck, I would say his presence alone makes 33rd Degree 2012 a must-see.

Tags: ,
Posted by Konrad Garus in Life, Programming | No Comments »

Apprenticeship Patterns

December 17th, 2011

Apprenticeship Patterns. Guidance for the Aspiring Software Craftsman by Dave Hoover and Adewale Oshineye has been lying on my bookshelf for quite a while. I expected a largely repetitive, buzz-driven soft read. It turned out to be quite a surprise.

Yes, Apprenticeship Patterns is a book about software craftsmanship. However, it’s very far from what wannabe-craftsmen say: “We’re all artists, why don’t people admire us like we deserve it.” Actually, it’s the exact opposite.

Learning is a very long road. Accept your ignorance as you begin, seek sources to learn from, kindred spirits or mentors. Get your hands dirty and learn by practice. Work with people, code breakable toys, read constantly, and practice, practice, practice. It’s only your responsibility to learn and improve your skills, diversify, and deepen your knowledge. Don’t repeat the buzz, but get your hands dirty and get to work. Finally, share what you learned with those behind you on the path and create communities where people can motivate each other and learn together. And don’t lose your motivation and goals along the road.

Now, the previous paragraph does look pretty fluffy. That’s just a birds-eye view, and those tend out to lack detail. The book itself is very down-to-earth, “dirty” and concrete. It’s an inspiring collection of thoughts, ideas and tricks on self-improvement.

I’ve seen quite a few talks and read a few articles on craftsmanship, and none of them was anywhere near as concrete and complete as this book. It doesn’t only apply to software. In fact, I believe it largely applies to any other area that involves learning in our life.

Highly recommended.

Tags:
Posted by Konrad Garus in Life, Programming | No Comments »

Careful with that PrintWriter

December 10th, 2011

Let’s assume we have a nice abstraction over email messages simply called Email. Something that can write itself to Writer and provide some other functionality (e.g. operating on the inbox).

Now, take this piece of code:

public void saveEmail(Email email) {
	FileOutputStream fos = null;
	try {
		fos = new FileOutputStream("file.txt");
		PrintWriter pw = new PrintWriter(fos);
		email.writeContentTo(pw);
		pw.flush();
		email.deleteMessageFromInbox();
	} catch (IOException e) {
		log.error("Saving email", e);	
	} finally {
		IOUtils.closeQuietly(fos);
	}
}

Question: What happens if for whatever reason the PrintWriter cannot write to disk? Say, your drive just filled up and you can’t squeeze a byte in there?

To my surprise, nothing got logged, and the messages were removed from inbox.

Unfortunately, I learned it the hard way. This code is only a piece of a larger subsystem and it took me quite a while to figure out. Used to dealing mostly with streams, I skipped this area and checked everything else first. If PrintWriter was like a stream, it would throw an exception, skipping deleteMessageFromInbox.

All streams around files work like that, why would this one be different? Well, PrintWriter is completely different. Eventually I checked its Javadoc and saw this:

“Methods in this class never throw I/O exceptions, although some of its constructors may. The client may inquire as to whether any errors have occurred by invoking checkError().”

At first I was furious about finding that’s all due to yet another inconsistency in core Java. Now I’m only puzzled and a bit angry. Most things in Java work like this – they throw exceptions on every occasion and force you to explicitly deal with them. And it’s not unusual to see 3 or 4 of them in a signature.

One could say, it’s all right there in the Javadoc. Correct. But still, it can be confusing to have two fundamentally different approaches to error handling in one corner of the library. If one gets used to dealing with one side of it, he may expect the same from the other. Instead, it turns out you always have to carefully look at signatures and docs, because even in core Java there is no single convention.

Related: try/catch/throw is an antipattern – interesting read, even if a bit too zealous.

Tags: , ,
Posted by Konrad Garus in Programming | 1 Comment »

TDD on Existing Code

December 1st, 2011

I recently read two blog posts about writing tests for existing (often “legacy”) code. Jacek Laskowski describes (in Polish) the moment when he realized that TDD actually means development driven by tests, starting with tests, and as such cannot be applied to existing codebase.

I can think of at least several cases where you can do TDD on existing, even not-so-pretty legacy code.

Introducing Changes

When you’re about to add a feature to existing code, start with a test for that single feature. When you see a bug report, write a test for it. Sure, it probably won’t cover much more than this feature or bug, but it can drive your development and guard against regression in future.

Understanding / Documenting Code

When you learn a library, you can try spikes in form of unit tests. It may be much more exhaustive and beneficial in future than a “breakable toy” that you throw away after use. You can use it as documentation or ready-to-use examples in future, and it may even protect you against bugs or incompatibilities introduced in new versions of the library.

You can also try the same trick on legacy code. As Tomek Kaczanowski points out, those tests will often be high-level, integration or end-to-end tests. That’s better than nothing and can be a good starting point for refactoring.

Is that TDD?

One could say that this is not test driven development. I would argue that the whole point of TDD is not a fanatic red-green-blue cycle. It is introducing small, fast, focused (as much as possible…), automated tests that become “live” specification and documentation, and protect you from regressions.

Yes, there is focus shift. There is no “red” phase. Moreover, in a way you write tests after code, even though the benefits left after the process are the same.

I’ve spent a few years on a fairly large project full of legacy code. And I mean legacy, sometimes in the facepalm way.

Whenever I start a piece of work, be it bug or feature, I try to think about tests. If it’s a rusty, legacy area, I may spend a while understanding the codebase. As I do it, I may leave tests for existing code as breadcrumbs. Very often they reveal weaknesses and beg for refactoring. Sometimes I may do the refactoring in place (if it’s very important, or easy), other times leave it for the future.

Sometimes I do know the area that I need to deal with, but it is pretty convoluted. Again, I may spend quite a while trying to write the first test for my new piece of work. But as I design this test, I carefully lay out and understand all collaborators and think about flow. Think of it: designing a test alone can help you understand codebase and the task at hand in much more depth, and feel much more safe about what you’re trying to do.

Once the first test is in, new tests are usually much easier and we’re back in the fancy red-green-blue groove.

There is much more depth to it. TDD is not limited to designing new code in green grass projects. Tests can also help you understand all the dependencies and conditions in existing environment. In other words, think carefully before hacking.

I would say it’s equally, if not even more beneficial, than on the green grass.

Tags:
Posted by Konrad Garus in Programming | 1 Comment »

Java Web Start Bugs: Offline Edition

November 24th, 2011

Java Web Start is full of bugs that never get resolved and make it next to unusable, and very costly, in business environment. If you have thousands of users who tend to cancel update prompts, or whose connections get interrupted, or who are simply unlucky, you are actually scared of releases. And if you try to release often? Yup, you guessed it.

Rant aside.

Recently I was trying to move away from web-based JWS to one that delivers jars and jnlp to the client and installs locally. I found several interesting bugs or quirks that I think are worth posting here.

Shortcut for all users

My first attempt was installing the application for all users from the Internet. To silently install shortcut on desktop for all users, you can add <shortcut> to your jnlp and execute:


javaws -import -system -silent http://example.com/myapp.jnlp

It does place shortcut on desktop for all users. But it doesn’t work!

Unfortunately, by default Java 6 on my Windows 7 has web start cache in C:\Users\Konrad\.... When you try to launch the app from John’s desktop, you get an ugly error because John cannot access that folder. It doesn’t matter that I run the command with -system switch from Administrator’s console. It lands in user-specific, protected directory.

Workaround: Force JWS to use another, accessible to everyone folder for cache.

Place this in C:\Windows\Sun\Java\Deployment\deployment.config:


deployment.system.config=file\:C\:/WINDOWS/Sun/Java/Deployment/deployment.properties
deployment.system.config.mandatory=false

And in C:\Windows\Sun\Java\Deployment\deployment.properties:


deployment.system.cachedir=C\:\\JavaWebStart
deployment.system.cachedir.locked=true

Offline installation and updates

Specification changed a bit and I researched purely “offline” solution. Place all files in a directory and install from there. Push updates with the same mechanism. In other words, use local file system for codebase and never ask JWS to look online. Official documentation advertises it as “CD installation”.

My JNLP starts with:


<jnlp codebase="file:/c:/myapp/" href="my.jnlp">

Files reside in C:\myapp and I install the app with:


javaws -import -system -silent file:/c:/myapp/my.jnlp

What could possibly go wrong?

Bug 1: You cannot change code if application is running

With web-based JWS, you can update your codebase anytime. Web Start keeps copies of the files in local cache and safely launches the application off that copy. Obviously.

Unfortunately, with local codebase it tries to be smart and does not copy to cache. Now, if you try to update by overwriting contents of C:\myapp while the application is running, it will die with a flood of ClassNotFoundExceptions in log. Oops.

Solution: None, sadly. Don’t update if the application is running. Period.

Bug 2: Curious “do you want to go online” message

OK, so I decided to update only when the application is not running.

Suppose John is running version 3.0.1. He closes the application. Now your smart updater overwrites contents of C:\myapp with 3.0.2. When John launches the application again, he may see a message requesting going online for an update because JNLP has changed.

This will happen if your <shortcut> is set for online="false".

That is plain wrong. My codebase is local, every single thing is local, and the user is bothered with a popup about going online. Then if I accept to “go online”, it innocently informs that it’s downloading update from file:/c:/...

Solution: Use <shortcut online="true">. Then it will always assume the application is “online” and therefore skip asking for permission to go online.

Bug 3: Update reinstalls app and removes shortcut

Remember, I am trying to maintain shortcut for all users at all times.

Now, if you update the application by simply overwriting contents of codebase directory, you’re in for another curiosity. On launch JWS will remove app from system cache and remove shortcuts from all desktops, and reinstall the application into this user’s cache. Others lose the shortcut for good.

Workaround: Reinstall the application into system cache yourself.

  1. javaws -uninstall file:/c:/myapp/my.jnlp
  2. Place new files in C:\myapp
  3. javaws -import -system -silent file:/c:/myapp/my.jnlp

Final rant

Every single time I deal with web start, it bites and makes me weep and gnash my teeth. I spent hours figuring all this out and fighting JWS who fought me back. As I deal with it I find some comfort researching the web and finding other desperate developers coping with similar issues. I hope this post saves someone some pain.

Tags: ,
Posted by Konrad Garus in Programming | No Comments »

Fill and Print an Array in Clojure

October 27th, 2011

In a post in the “Extreme OO in Practice” series (in Polish), Koziołek used the following example: Fill a 3x3x3 matrix with subsequent natural numbers and print it to stdout.

The example in Java is 27 lines of code (if you remove comments). It is later refactored into 55 lines that are supposed to be more readable, but personally I can’t help but gnash my teeth over it.

Anyway, shortly after reading that example, I thought of what it would look like in Clojure. Here it is:

(def array (take 3 (partition 3 (partition 3 (iterate inc 1)))))
(doall (map #(doall (map println %)) array))

That’s it. I know it’s unfair to compare this to Java and say that it’s shorter than your class and main() declaration. But it is, and it makes the full-time Java programmer in me weep. Anyway, I have two observations.

As I like to point out, Java is awfully verbose and unexpressive. Part of the story is strong typing and the nature of imperative programming. But then again, sometimes imperative programming is very clumsy.

Secondly, this solution in Clojure presents a completely different approach to solving the problem. In Java you would declare an array and then iterate over it in nested loops, incrementing the “current value” in every innermost iteration. In Clojure, you start with an infinite sequence of lazy numbers, partition (group) it in blocks (that’s one row), then group those blocks into a 2D array, and take 3 of those 2D blocks as the final 3D matrix. Same thing with printing the array. You don’t iterate over individual cells, but naturally invoke a function on a sequence.

The difference is subtle, but very important. Note how the Java code needed 4 variables for the filling (“current value” and 3 indexes) and 3 indexes for printing. There was a very clear distinction between data and what was happening to it. In Clojure you don’t need to bother with such low-level details. Code is data. Higher-order functions naturally “serve as” data in the first line, and are used to iterate over this data in the second.

Tags: , ,
Posted by Konrad Garus in Programming | No Comments »

Java Does Not Need Closures. Not at All.

October 17th, 2011

I hope that most Java developers are aware of the Swing threading policy. It makes sense, except for that it’s so insanely hard to follow. And if you do follow it, I dare say there is no place with more ridiculous boilerplate.

Here’s two examples that recently got me scratching my head, gnashing my teeth, and eventually weep in disdain.

Case One: Display Download Progress

class Downloader {
    download() {
        progress.startDownload();
        while(hasMoreChunks()) {
            downloadChunk();
            progress.downloadedBytes(n);
        }
        progress.finishDownload();
    }
}
 
class ProgressDisplay extends JPanel {
    JLabel label;
    startDownload() {
        SwingUtilities.invokeLater(new Runnable() {
            public void run() {
                label.setText("Download started");
            }
        });
    }
    downloadedBytes(int n) {
        SwingUtilities.invokeLater(new Runnable() {
            public void run() {
                label.setText("Downloaded bytes: " + n);
            }
        });
    }
    finishDownload() {
        SwingUtilities.invokeLater(new Runnable() {
            public void run() {
                label.setText("Download finished");
            }
        });
    }
}

Solution? Easy! Just write your own JLabel wrapper to hide this mess. Then a lengthy JTable utility. Then a JProgressBar wrapper. Then…

Case Two: Persist Component State

Task: Save JTable state to disk. You’ve got to read table state in EDT, but writing to disk from EDT is not the the best idea.

Solution:

ExecutorService executor = Executors.newSingleThreadExecutor();
 
void saveState(final Table table) {
    SwingUtilities.invokeLater(new Runnable() {
        public void run() {
            TableColumns state = dumpState(table);
            executor.execute(new Runnable() {
                public void run() {
                    saveToDisk(table.getTableKey(), state);
                }
            });
        }
    });
}

Two inner classes, inheritance, all those publics and voids and parentheses. And there is no way to avoid that!

In a sane language, it could look like:

(defn save-state [table]
  (do-swing
    (let [state (dump-state table)]
      (future (save-to-disk table state)))))

All semantics. Little ceremony, almost no syntax. If you counted characters in both solutions, I guess the entire second snippet is as long as the first one somewhere to the invocation of invokeLater. Before you get to the actual logic. No strings attached, do-swing and future are provided by the language or “official” utils.

Bottom Line

I recently saw two presentations by Venkat Subramaniam on polyglot programming where he mercilessly made fun of Java. In his (vaguely remembered) words, when you write such code for the whole day, you come back home and weep. You can’t sleep because you’re tormented by the thought of the abomination you created, and that you will have to deal with tomorrow. If your children asked you what do you for living, would like them to see this?

Tags: , , ,
Posted by Konrad Garus in Programming | No Comments »

Yaclot 0.1.2: Extended Date Conversions

October 8th, 2011

I released a new version of my Clojure conversion and map transformation library to Clojars. It includes two new features:

1. Added natural conversion between Long and Date.

2. You can pass a collection of formats for converting String to Date. It attempts parsing with each of the formats and returns first result which didn’t throw ParseException.

(convert
  "2/12/11"
  (using-format ["yyyy-MM-dd" "M/dd/yy"]
    (to-type java.util.Date)))
; => #<Date Sat Feb 12 00:00:00 CET 2011>

Enjoy!

Tags: ,
Posted by Konrad Garus in Programming | No Comments »

Hibernate Cache Is Fundamentally Broken

August 24th, 2011

Simple as the title says, I dare say Hibernate second level (L2) cache is fundamentally broken. At least with clustered cache, which appears to be supported from the official docs.

There are two strong reasons for clustering, that is spreading your load over multiple servers: Scaling out and availability.

What does availability mean? Among other things, zero downtime on updates. You take one server down, update, start it up and then continue with the next. With reasonable load balancer, proxy, middleware or what have you, the clients won’t notice a thing and be seamlessly redirected to whichever server is up at the moment.

Now, what if during an update you change definition of an entity? Add or remove a field? You have old servers with old definition, updated servers with new definition, all sharing the same clustered cache. Java has serialVersionUID for it. If you serialize an object, and then deserialize it on another node with a different version, it will fail with an exception.

Since Hibernate openly advertises clustered caching, one would expect it to work just fine with this case. Unfortunately, this is not so.

What Hibernate puts in cache is a plain old array of values of individual fields. That is submitted to clustered cache. Then a node with different version loads this array from cache and tries to populate entity with different definition with it, simply copying field by field by their numeric index.

When that happens, you’re screwed.

Suppose you have this entity definition:

class User {
  int id;
  String password;
  String email;
  String login;
}

… and you’re updating to this:

class User {
  int id;
  String password;
  Timestamp passwordExpires;
  String email;
  String login;
}

The best thing that can happen is an outage. For instance, the 3rd field used to be a String. You added a Timestamp field before it and in the new definition the 3rd field is a Timestamp. Hibernate on new nodes fails on load() with ClassCastException from String to Timestamp, because cache still has the old definition.

The much worse case is data corruption. Suppose you need a User for the following transaction:

User user = session.load(User.class, 4);
user.setPasswordExpires(...);
session.update(user);

Let’s say that email was null. load() does not yield a ClassCastException, because null is a perfectly valid Timestamp. But when Hibernate loads such entity from cache, the cached entry only has 4 fields. Login is not restored and remains null. When you update(), you’re doomed. In this made up example here this would hopefully fail on a DB not-null constraint. In real life, though, it can silently save corrupted data in database and guarantee hours of very interesting debugging and restoring from backups, if not physical damage caused by your application’s misbehavior.

There’s this old piece of music called “Careful with that Axe, Eugene”. If you don’t know it, don’t bother googling. Don’t even get me started about YouTube, it needs proper sound setup and dynamics. So, here’s how it goes. Apparently boring, monotonic bass softly playing “bing, bang, bing, bang” (or D, D, D, D as Wikipedia says). Nothing happens for a few minutes, except for just as soft ambientish keyboard tones. And so on for one minute, another, then another. Then, out of the blue, an air-shattering scream. For the first time in my life I heard it in Australians’ concert, and it literally made me jump with a shot of adrenaline and panic.

That’s the experience with clustered cache and Hibernate. It’s very robust and stable. Boring. Unnoticeable. Until one day it makes you scream hard and tear your hair out.

Handle with care. Be ware. Be prepared.

Little piece of disclaimer: I don’t know if this exact example here reproduces the problem. It’s merely a made up illustration. The fields might be ordered by name, and Hibernate may refuse to restore from an array that has fewer fields than the current definition. But I have witnessed both issues in real life, and they caused much pain and cost time and money.

Tags: ,
Posted by Konrad Garus in Programming | 2 Comments »

« Older Entries