Monthly Archives: February 2012

Sequential ID Generation in Congomongo

By default MongoDB uses 12-byte BSON id for objects. For some reason I wanted to use an increasing sequence of integers.

The samples in documentation are in JSON and I was not sure how they translate to the Java driver. The JSON sample looks like:

function counter(name) {
    var ret = db.counters.findAndModify({query:{_id:name}, update:{$inc : {next:1}}, "new":true, upsert:true});
    // ret == { "_id" : "users", "next" : 1 }
    return ret.next;
}

db.users.insert({_id:counter("users"), name:"Sarah C."}) // _id : 1
db.users.insert({_id:counter("users"), name:"Bob D."}) // _id : 2

After some googling I found an implementation in Java. Just as I expected, it’s much longer and completely different.

public static String getNextId(DB db, String seq_name) {
    String sequence_collection = "seq"; // the name of the sequence collection
    String sequence_field = "seq"; // the name of the field which holds the sequence
 
    DBCollection seq = db.getCollection(sequence_collection); // get the collection (this will create it if needed)
 
    // this object represents your "query", its analogous to a WHERE clause in SQL
    DBObject query = new BasicDBObject();
    query.put("_id", seq_name); // where _id = the input sequence name
 
    // this object represents the "update" or the SET blah=blah in SQL
    DBObject change = new BasicDBObject(sequence_field, 1);
    DBObject update = new BasicDBObject("$inc", change); // the $inc here is a mongodb command for increment
 
    // Atomically updates the sequence field and returns the value for you
    DBObject res = seq.findAndModify(query, new BasicDBObject(), new BasicDBObject(), false, update, true, true);
    return res.get(sequence_field).toString();
}

Not much later I checked docs and source code for Congomongo and discovered fetch-and-modify. I rewrote the Java sample above to Clojure and later polished it using code from this commit by Krzysztof Magiera. In the end my sequence generator looks like this:

(defn next-seq [coll]
  (with-mongo db
    (:seq 
	  (fetch-and-modify :sequences {:_id coll} {:$inc {:seq 1}} :return-new? true :upsert? true))))
	
(with-mongo db 
  (insert! :books {:author "Adam Mickiewicz" :title "Dziady" :_id (next-seq :books)}))

The raw call to insert! could be wrapped in a function or macro to save some boilerplate if there are more collections. For instance:

(defn insert-with-id [coll el]
  (insert! coll (assoc el :_id (next-seq coll))))
  
(with-mongo db
  (insert-with-id :books {:author "Adam Mickiewicz" :title "Dziady"}))

In some circles this probably is common knowledge, but it took me a while to figure it all out.

Integrating Spring, Velocity and Tiles

I like Tiles and I heard a lot about Velocity. They seem to serve different purpose and reportedly are easily married together, so I decided to give it a shot and use them both in Spring web app. The integration actually took many hours and was a real rollercoaster during which I learned a lot about all three technologies. Hopefully this post can spare someone this fun and let them focus on business.

Goals

When I use tiles, I don’t like the default (?) approach of tiles.xml. I don’t want to put imports of JS & CSS, page title, navigation, body etc. each in its own file like in the snippet below, because that makes me switch between editor windows.

<definition name="hello" template="/WEB-INF/templates/main.jsp">
	<put-attribute name="title" value="Hello" type="string" />
	<put-attribute name="head" value="/WEB-INF/templates/hello-js-and-css.jsp" />
	<put-attribute name="nav" value="/WEB-INF/templates/hello-nav.jsp" />
	<put-attribute name="body" value="/WEB-INF/templates/hello.jsp" />
</definition>

Obviously I don’t want to put too much detail in tiles.xml, either.

What I really like is having one file per page, assembling the template in one place, such as this piece of JSP:

<tiles:insertTemplate template="template.jsp">
	<tiles:putAttribute name="title" value="Hello" />
	<tiles:putAttribute name="head">
		<script type="text/javascript" src="/js/jQuery.js" />
		<script type="text/javascript" src="/js/hello.js" />
	</tiles:putAttribute>
	<tiles:putAttribute name="body">
		<div>Hello, world!</div>
	</tiles:putAttribute>
</tiles:insertTemplate>

In Velocity, it’s supposed to look like this:

#tiles_insertTemplate({"template": "template.vm"})
	#tiles_putAttribute({"name":"title", "value": "Hello"})#end
	#tiles_putAttribute({"name":"head"})
		<script type="text/javascript" src="/js/jQuery.js" />
		<script type="text/javascript" src="/js/hello.js" />	
	#end
	#tiles_putAttribute({"name":"body"})
		<div>Hello, world!</div>
	#end
#end

However, the docs on integration are really intended for adding some Velocity support to your Tiles-based app, while I wanted quite the opposite: Use Tiles in my rich Velocity app, with full support for spring context, macros etc.

Solution

In short, what we’re going to do is:

  1. Use VelocityViewResolver to resolve and render pages
  2. Add support for Tiles macros to this Velocity rendering engine
  3. Extend the Tiles renderer with full support for Velocity, including Spring context, macros etc. Ultimately we’re going to make it use the original Velocity engine created by Spring.

Full source code in form of a minimal, complete web app are at github. For details see below.

Spring & Velocity -> Tiles

For the first step, we define viewResolver and velocityConfig like this:

@Bean
public VelocityConfig velocityConfig() {
	VelocityConfigurer cfg = new VelocityConfigurer();
	cfg.setResourceLoaderPath("/WEB-INF/velocity/");
	cfg.setConfigLocation(context
			.getResource("/WEB-INF/velocity.properties"));
	return cfg;
}

@Bean
public ViewResolver viewResolver() {
	VelocityViewResolver resolver = new VelocityViewResolver();
	resolver.setViewClass(VelocityToolboxView.class);
	resolver.setSuffix(".vm");
	return resolver;
}

It’s important that we use VelocityToolboxView there, otherwise the tiles directives won’t work.

We also need to put the following in velocity.properties:

userdirective=org.apache.tiles.velocity.template.AddAttributeDirective,\
  org.apache.tiles.velocity.template.AddListAttributeDirective,\
  org.apache.tiles.velocity.template.DefinitionDirective,\
  org.apache.tiles.velocity.template.GetAsStringDirective,\
  org.apache.tiles.velocity.template.ImportAttributeDirective,\
  org.apache.tiles.velocity.template.InsertAttributeDirective,\
  org.apache.tiles.velocity.template.InsertDefinitionDirective,\
  org.apache.tiles.velocity.template.InsertTemplateDirective,\
  org.apache.tiles.velocity.template.PutAttributeDirective,\
  org.apache.tiles.velocity.template.PutListAttributeDirective

This adds basic support for Tiles directives to Velocity, but it’s still useless because once Velocity hands rendering over to Tiles, Tiles is unable to render Velocity and would simply ignore it (rendering syntax of #directives to browser.

Tiles -> Velocity

We need to teach Tiles to use Velocity. For this we’re going to need a custom TilesInitializer:

@Bean
public TilesConfigurer tilesConfigurer() {
	TilesConfigurer cfg = new TilesConfigurer();
	cfg.setTilesInitializer(new VelocityTilesInitializer(velocityConfig()));
	return cfg;
}	
public class VelocityTilesInitializer extends DefaultTilesInitializer {
	private VelocityConfig velocityConfig;

	public VelocityTilesInitializer(VelocityConfig velocityConfig) {
		this.velocityConfig = velocityConfig;
	}

	@Override
	protected AbstractTilesContainerFactory createContainerFactory(
			TilesApplicationContext context) {
		return new BasicTilesContainerFactory() {

			@Override
			protected List<TilesRequestContextFactory> getTilesRequestContextFactoriesToBeChained(
					ChainedTilesRequestContextFactory parent) {
				List<TilesRequestContextFactory> factories = super
						.getTilesRequestContextFactoriesToBeChained(parent);
				registerRequestContextFactory(
						VelocityTilesRequestContextFactory.class.getName(),
						factories, parent);
				return factories;
			}

			@Override
			protected AttributeRenderer createTemplateAttributeRenderer(
					BasicRendererFactory rendererFactory,
					TilesApplicationContext applicationContext,
					TilesRequestContextFactory contextFactory,
					TilesContainer container,
					AttributeEvaluatorFactory attributeEvaluatorFactory) {
				ContextPassingVelocityAttributeRenderer var = new ContextPassingVelocityAttributeRenderer(
						velocityConfig.getVelocityEngine());
				var.setApplicationContext(applicationContext);
				var.setRequestContextFactory(contextFactory);
				var.setAttributeEvaluatorFactory(attributeEvaluatorFactory);
				var.commit();
				return var;
			}
		};
	}
}

We’re almost there, but here’s a tricky bit. Normally in lines 31-32 you would put velocityAttributeRenderer. However, this renderer completely ignores the Spring-augmented Velocity context & engine that Tiles received from Velocity. It creates its own VelocityEngine and lets it do the rendering, throwing away all the Spring and tiles directives and context objects.

There is no way to change this behavior in Tiles (which otherwise seems to be an interesting study in design patterns and extensibility). I even created two JIRA issues for it: 541 for forwarding context and 542 for injecting VelocityEngine.

Meanwhile, we have to make do with this workaround (see github for full source):

public class ContextPassingVelocityAttributeRenderer extends
		AbstractTypeDetectingAttributeRenderer {
	// ...

	private VelocityEngine engine;

	public ContextPassingVelocityAttributeRenderer(VelocityEngine engine) {
		this.engine = engine;
	}
	
	// ...

	public void commit() {
		velocityView = new VelocityView(new TilesApplicationContextJeeConfig());
		velocityView.setVelocityEngine(engine);
	}

	@Override
	public void write(Object value, Attribute attribute,
			TilesRequestContext request) throws IOException {
		if (value != null) {
			if (value instanceof String) {
				InternalContextAdapter adapter = (InternalContextAdapter) ((VelocityTilesRequestContext) request)
						.getRequestObjects()[0];
				Context context = adapter.getInternalUserContext();
				Template template = velocityView.getTemplate((String) value);
				velocityView.merge(template, context, request.getWriter());
			} else {
				throw new InvalidTemplateException(
						"Cannot render a template that is not a string: "
								+ value.toString());
			}
		} else {
			throw new InvalidTemplateException("Cannot render a null template");
		}
	}
	
	// ...

It works arounds both of the JIRA issues and lets us accomplish the ultimate goal:

  1. The VelocityEngine injected to VelocityView here is the original VelocityEngine from Spring. Among other things, it supports Spring directives and context-dependent tools.
  2. The TilesRequestContext in write method still contains the original Velocity context created from Spring scaffolding. Standard implementation of VelocityAttributeRenderer simply throws it away. This workaround above extracts the original context and uses it for rendering.

Conclusion

This journey took much more time than I thought. Documentation on such cases is nonexistent, so I spent hours debugging, reading source code, experimenting, praying and cursing. It was even more fun as I had close to zero knowledge of internals of Spring view resolution and rendering engine, as well as Tiles and Velocity.

It’s quite satisfying since I learned a ton about all those technologies and eventually was able to resolve it in a fairly elegant way. But it was also a frustrating and time-consuming riddle and I hope this post spares someone some trouble.

Update – Velocity Tools

A while later I discovered this solution does not support Velocity Tools property. Here’s how to add it: Spring & Velocity Tools.

Connection Management in MongoDB and CongoMongo

I decided to take the opportunity offered by Jacek Laskowski (in Polish) and take a closer look at interaction with MongoDB in Clojure. It has a nice, challenging learning curve as I haven’t done much practical work in Clojure and I’ve never actually dealt with Mongo before. Double win – learning two interesting things at a time.

The obvious choice for the integration is CongoMongo. It’s really easy to get it all set up and working. The official docs encourage you to simply do this:

(def conn (make-connection "mydb")

(set-connection! conn)

(insert! :robots {:name "robby"})

(fetch-one :robots)

; ... and so on

Easy. Too easy and comfortable. Coming from the old good and heavy JDBC/SQL I felt uneasy with the connection management. How does it work? Does it just open a connection and leave it dangling in the air the whole time? Might be good for a quick spike in REPL, but not for a real application which needs concurrency, is supposed to be running for days and weeks, and so on. How do you maintain it properly?

clojure.contrib.sql has with-connection. That opens the connection, runs something with it and then eventually closes it. CongoMongo has with-mongo, but all it does is bind the argument to *mongo-config* and execute body. Nothing is ever opened or closed.

That seemed insane and broken, until I took a step back and compared source of CongoMongo to documentation of underlying Java driver for MongoDB. The light dawned.

What make-connection really does is create an instance of Mongo and DB (if database name was provided). The result of this function is plain map: {:mongo #<Mongo>, :db #<DB>}.

Javadoc for Mongo say it’s a database connection with internal pooling. For most application, you should have 1 Mongo instance for the entire JVM. A page dedicated to Java Driver Concurrency explains it in more detail: The Mongo object maintains an internal pool of connections to the database. For every request to the DB (find, insert, etc) the java thread will obtain a connection from the pool, execute the operation, and release the connection..

At first I thought CongoMongo docs were misleading. The truth is, it’s just a wrapper for the Java driver. It’s fair for it to assume you know the basic principles of the underlying driver.

So what is called a “connection” here (the Mongo class) is in fact a much more sophisticated object. It maintains a connection pool and creates nice little DB objects for all the data handling, which in turn are smart enough to maintain the actual low-level connections for you. No ceremony, just gets out of the way as quickly as possible and lets you get the job done.

This is amazingly simple and elegant compared to JDBC / JEE / SQL. I guess I soon will be scratching my head over ACID, but at the moment I’m pleasantly surprised with the look of things.