# “Processing in Clojure” Made Functional

Invited to nitpicking by Jacek Laskowski in his latest post titled Processing in Clojure, I decided to write my own version of his flower-drawing application. I find it an interesting challenge and illustration for my previous post about functional thinking and the use of language and decomposition.

Full source code is below, but let’s start with what I did not like and decided to rewrite.

I find Jacek’s code rather structural and hard to comprehend. Everything happens in one big function and it takes quite a while to understand what it does and how. It’s hard to see why `angle` is an angle, and variables called `a` and `value` don’t make it any easier.

Thus my first two improvements are: Break it down into smaller pieces and use more informative names. For instance, `value` is in fact `scale`. I extracted drawing flower to a separate function with clear distinction to drawing petals and the central piece. I also paid more attention to variable names and hid the more complex concepts behind simple, well-named functions.

The second type of improvements is use of functional features. After a while of careful inspection I observed that `angle` is a simple linear sequence, and `scale` is result of a simple function on this sequence. I decided to replace it with a dedicated infinite sequence.

In the end, this code is longer and has more levels of abstraction, but I think it’s more functional and comprehensible. Functions are much shorter. Details no longer obscure the view, but are hidden behind more descriptive names. Now you can actually see that the applet’s `draw` draws a flower which consists of petals and the central piece. The central piece is a simple circle, while petals are several circles of random size and color around the center. Then only if you want you can delve into details of how their color and size are generated.

Like I mentioned in the first paragraph, I find it a nice illustration of what Paul Graham describes as building a language for the solution.

# Yaclot 0.1.0 Released (Clojure Conversion and Record Transformation Library)

I released Yaclot 0.1.0 to Clojars. This version can be used to convert single values as well as records.

Converting individual values looks like:

```(convert "2011-02-12" (to-type java.util.Date))
; => #<Date Sat Feb 12 00:00:00 CET 2011>

(convert "2/12/11"
(using-format "M/dd/yy" (to-type java.util.Date)))
; => #<Date Sat Feb 12 00:00:00 CET 2011>

(convert 5000.42 (to-type String (using-format "%,.2f")))
; => "5,000.42"
```

Order of `using-format` and `to-type` doesn’t matter. `using-format` is optional (default for dates is `yyyy-MM-dd`).

You can also use it to convert records in a single operation:

```(map-convert
{:dt "2011-02-12" :int 42 :label "Label"}    ; Record
{:dt  (to-type java.util.Date)               ; Desired types
:int (to-type String)})
; => {:dt #<Date Sat Feb 12 00:00:00 CET 2011>, :int "42", :label "Label"}
```

The project originated from a pet web application, where I needed to deal with parsing and formatting numbers and dates between the “presentation”, “business logic” and “database” layers.

To learn more about motivation, features and syntax, take a look at the draft in my previous post.

This version supports conversions between String, Date and numeric types. Next on the road map is exception-free error handling with local bindings and validation.

# Designing Yaclot: Generic Clojure Conversion Library

In real Clojure applications you often cannot avoid explicit type conversions, even though the language does not require you to explicitly specify types all the time. Another great feature of Clojure is its universal data structure: most of the time you don’t need anything more than a simple map or record. However, these records often need to be a bit different in different areas of the applications.

## Rationale

Examples? You may have a `java.sql.Date` in database and backend, but your web front end provides you with a `String`. Or your backend uses rational numbers, but you need to format them into nice `Strings` for presentation.

Sometimes you operate on whole records. From database you may pull the following:

```(def db-sample
{:date    2011-02-01 ; java.sql.Date
:balance 2042.00    ; java.math.BigDecimal
:credit  1000.00    ; java.math.BigDecimal
:roi     2.13 })    ; java.math.BigDecimal
```

… for display you would like to use:

```(def presentation-sample
{:date    "Feb 1, 2011" ; String
:balance "\$2,042.00"   ; String
:credit  "\$1,000.00"   ; String
:roi     "2.13%" })    ; String
```

… and you need to support input from a web form as:

```(def form-sample
{:date    "2011-02-01"  ; String
:balance "2042"        ; String
:credit  "1000"        ; String
:roi     nil })        ; Rational calculated
; by back-end from other fields
```

All three have very similar structure, but implementing all these transformations can be pain.

## Planned API for generic converter

I am implementing a library that will do it for you, with API similar to the following:

```(def db-fmt
{:date    (to-type java.sql.Date)
:balance (to-type java.math.BigDecimal)
:credit  (to-type java.math.BigDecimal)
:roi     (to-type java.math.BigDecimal) })

(def presentation-fmt
{:date    (using-format "MMM d, yyyy" (to-type String))
:balance (using-format "\$%.2f"       (to-type String))
:credit  (using-format "\$%.2f"       (to-type String))
:roi     (using-format "%.2f%%"      (to-type String)) })

(def form-fmt
{:date    (using-format "yyyy-M-d" (to-type java.sql.Date))
:balance                          (to-type java.math.BigDecimal)
:credit                           (to-type java.math.BigDecimal) })

(map-convert db-sample presentation-fmt)
=> ; (similar to presentation-sample)

(map-convert form-sample db-fmt)
=> ; (similar to db-sample)
```

Another feature is generic conversion function for individual values:

```(convert "2011-02-12" (to-type java.util.Date))
; => #&lt;Date Sat Feb 12 00:00:00 CET 2011&gt;

(convert (java.util.Date. 111 1 12) (to-type String))
; => "2011-02-12"

(convert 42 (to-type String))
; => "42"

(convert "2/12/11" (using-format "M/dd/yy" (to-type java.util.Date)))
; => #&lt;Date Sat Feb 12 00:00:00 CET 2011&gt;
```

## Ideas for the Future

In the future, this library could support pre- and post-conversion validation. For instance, check that `balance` is not empty and `date` is in the correct format before conversion, and validate that `balance` and `credit` are positive once they are numbers.

Another idea is using it as a base for yet another HTML form manipulation library. I found existing libraries somewhat disappointing as they imposed to many restrictions on me. I would like to have the ability to manipulate and lay out forms as I please, and only use the bits of the library that I need right now.

## Current Status

Currently much of the API and features are designed, but only the above presented part is implemented (not even supporting formats for numbers, only for dates). Once I implement conversions between all popular/basic types I will mark it 0.1 and push to Clojars. Error handling and validation are next on the road map.

The code is available at github.

Feel free to share any comments or ideas.

# TDD in Clojure: Mocking & Stubbing

A few minutes into my first real TDD trip in Clojure I discovered there is two reasonable ways I can do tests: Mocking or black-box testing. I decided to go for mocking, and so I discovered `clojure.contrib.mock`. I found the docs fairly confusing, but finally understood it with a little help of this article and research.

## Verify Calls to a Function

Assuming we have a function to compute square of a number, we want to write another function for square of sum.

```(defn square [x] (* x x))
```

Our test can look like this:

```(ns squirrel.test.core
(:use [[clojure.test]])
(:use [[clojure.contrib.mock]]))

(deftest test-square-of-sum
(expect [square (has-args [3])]
(square-of-sum 2 1)))
```

This use of `expect` asserts that when we execute the inner form `(square-of-sum 2 1)`, it calls `square` with argument equal 3. However, it does not execute `square` itself. The only thing that this test checks is whether `square` got called. In particular, it does not check what `(square-of-sum 2 1)` returns. We’ll get back to stubbing in a moment.

## Stubbing

Let’s modify our test to also assert the final result:

```(deftest test-square-of-sum
(expect [square (has-args [3])]
(is (= 9 (square-of-sum 2 1))))
```

When the test runs, it fails because square-of-sum returns `nil`. The reason is that `expect` replaces `square` with a stub which by default doesn’t return anything.

To have stub return a concrete value, we can use the `returns` function:

```(deftest test-square-of-sum
(expect [square (returns 9 (has-args [3]))]
(is (= 9 (square-of-sum 2 1))))
```

Voila. Now the test passes.

To recap, what this two-line test does is:

• Create a stub for `square` which returns 9 for argument of 3.
• Assert this stub is called with argument 3.
• Assert that `square-of-sum` calls this stub with argument 3.
• Assert that `square-of-sum` eventually returns the correct value.

Quite a lot for such a tiny test.

## Expectation Hash

You may be wondering what exactly the second argument in each binding pair is. Clojure docs call it expectation hash.

Each function that operates on an expectation hash, such as `has-args` or `returns`, has two overloaded versions. One of them only takes a value or predicate and returns a new expectation hash. Examples include `(returns 9)` or `(has-args [3])`.

The other version takes an expectation hash as the second argument. These versions are used to pass the expectation hash through a chain of decorators. Order of decoration does not matter, so `(returns 9 (has-args [3]))` is effectively the same as `(has-args [3] (returns 9))`.

# Learn-Clojure.com: Clojure basics distilled

Kyle Cordes has put up a new site dedicated to Clojure: learn-clojure.com. As you can read in it’s FAQ, it is an attempt to consolidate just the most useful introductory material in one place, presented in a way that aims to help new or prospective Clojure developers.

At this moment that means basic information on what Clojure is, as well as many useful references to tutorials, videos and books. Clearly it’s not meant to compete with clojure.org or clojuredocs.org. Nor is it another tutorial or a garbled list of everything and nothing. It’s merely a clean, concise, nicely-organized gateway to the Clojure world.

This is very important, because while Clojure is rapidly gaining popularity (some are dubbing it “the next big thing” after OOP), it may be pretty discouraging at the beginning. In this world of enthusiastic magicians, crude immature frameworks and sparse documentation, by no means has finding my way been easy. Let alone answering the fundamental questions of “why?”, “what?” and “how?” That was an empty niche, and it’s great that Kyle made an effort and organized this together.

Unfortunately, apart from the Community section (not much going on in Poland) it does not provide much information to people who already are familiar with the basics. I’d love to see such a nice list on more concrete subjects like: “What are my options for building a web app?”, “How can I access the database?”, “How do I integrate it with JEE?” As well as a showcase of successful, inspiring (and mature) applications like Incanter or Ring. But that is yet to come, or may even be out of scope of this site.

# The Joy of Clojure: Learning “The Clojure Way”

Very often the journey with Clojure starts like this. You notice its growing popularity or want to attach a new tool to your belt. So you go through some quick introduction, learn the basics, get a few programs running… And eventually ask yourself: Fine, but what’s next? What’s the difference? How does one really work with Clojure?

That’s where The Joy of Clojure by Michael Fogus and Chris Houser enters the play. The goal of this book is to provide answers to these fundamental questions and teach you “The Clojure Way”.

It starts with an explanation of what Clojure is and the problems that it solves. Then it provides an overview of the language: data types, functions, collections, destructuring, composite data types and lazy evaluation. Finally it explores more advanced concepts: functional programming, metaprogramming, performance, mutation, concurrency and parallel programming.

However, by no means is it an average reference / language overview book. What it really does is explain the philosophy of LISP, functional programming and last but not least Clojure, and only then discuss all the technical stuff.

Just because the primary focus is not syntax and low-level technical details does not mean they are not explained in depth. Nothing is missing: syntax, performance, intricate details of data types and other constructs – they’re all here.

That’s where the book really shines. It’s easy to write a manual that describes a technology, but fails to describe its purpose. “The Joy of Clojure” managed to explain what Clojure really is, with all its fundamentals and idioms.

The major downside of the book is that some of the advanced low level technical / syntactical aspects are explained in a rather steep, if not discouraging way. Take macros for example. The concept is clear, but the examples could be less complicated and it falls short of explaining the basic syntax.

If you’re looking for an easy basic introduction to Clojure, or a practical guidebook on libraries, developing web apps etc. then probably this is not the best choice. But if you are up for a challenge or know some Clojure and are asking yourself the fundamental questions from the first paragraph of this review, you are going to love it. Sometimes it is steep and demanding, it may require you to reread some of the chapters later, but it definitely is worth it.

Note: This review is based on an early access edition provided by Manning Publications Co. The estimated publishing date is December 2010, but you can get a pre-release PDF already from Manning Publications.

# Corecursion in Clojure

Another concept which would be very hard to implement in Java is corecursion. It can be hard to describe in plain English (as evidenced in the somewhat hilarious Wikipedia article), so let’s have a look at an example.

Everyone knows the Fibonacci sequence and its linear time implementations. To keep the story short, here’s how you could do it in Clojure:

```(def fib (lazy-cat [0 1] (map + fib (rest fib))))
```

This snippet is a linear time, memoized, lazily-loaded infinite sequence of Fibonacci numbers. Although it’s very short, it can take quite a while to understand if you’re unprepared.

It generates a sequence which starts with numbers 0 and 1, and then a function which adds the sequence to itself, shifted by one:

```index              0        1      2
fib                0        1      f
(rest fib)         1        f
(+ fib (rest fib)) f
```

One way to understand how it works is this iterative sequence:

```index               0        1        2       3
fib                 0        1       [fib(2)  ...]
(rest fib)          1       [fib(2)   ...]
(+ fib (rest fib)) [fib(2)   ...]
```

The bottom row is a sum of the two rows above it. If you were doing it on a sheet of paper, you would add two elements in the bottom row (fib(2) = 0 + 1) and update the two rows above it. Then use the computed value for next element in the bottom row, and so on. Ellipsis symbolizes “next element”. Each time the top row is updated, all elements from the bottom row are copied – that is the newly completed value and another ellipsis. So the next step would be:

```index               0        1        2          3
fib                 0        1       [1          fib(3)  ...]
(rest fib)          1       [1        fib(3)     ...]
(+ fib (rest fib)) [1        fib(3)   ...]
```

That’s a very naive, but beginner-friendly explanation. Of course at all times there is only one sequence (pictured in square brackets). What’s not obvious is that thanks to smart caching of such lazy sequences the addition of each pair only occurs once. So even though fib(2) is computed from (+ [0 1] [1]) and fib(2) from (+ [0 1 1] [1 1]), 0 is only added to 1 one time.

### Explanation

The core idea is: Having a data structure, define a computation on that structure. Execute it, then repeat on result of this computation, and so on. It is similar to recursion, but it produces a sequence of data, with calculations taking place on demand when this data is dereferenced. It’s oriented on data, not the operations.

The Fibonacci sequence example is very popular, but corecursion is applicable in many more areas. Inspired by the before mentioned article, I decided to implement a breadth-first tree traversal (BFS).

Let’s define a tree as:

```(defstruct tree :val :left :right)

(def my-tree
(struct tree 1
(struct tree 2)
(struct tree 3
(struct tree 4
(struct tree 66))
(struct tree 5))))

;     1
;    / \
;   2   3
;      / \
;     4   5
;    /
;   66
```

Not yet being a master, it took me quite a while to get this right. Whatever I tried was nowhere near the elegant solution I got from ataggart at stackoverflow:

```(defn bftrav [& trees]
(when trees
(lazy-cat trees
(->> trees
(mapcat #(vector (:left %) (:right%)))
(filter identity)
(apply bftrav)))))

(bftrav my-tree)
```

It returns a lazy sequence of all subtrees in breadth-first order. It starts with the tree, then (lazily) appends its left and right subtree (unless they’re empty), then subtrees of the trees from previous step…

### Why do corecursion?

Once you understand corecursion, it’s not harder than recursion. It has several advantages which make it a better approach in some situations. Sometimes it’s shorter and faster (thanks for heavy use of memoization). The biggest advantage is that it produces a stream of data which you can easily iterate and do whatever you need without a callback. You define the algorithm once, and then reuse it. Whether you want to search the graph for an element or serialize a graph of objects, you use the exact same function. However, since it’s callback-free, the traversal algorithm is clearer and more cohesive.

# Software Transactional Memory in Clojure (alter vs. commute)

One of the selling points of Clojure is its excellent support for multithreading. Among other features, it has a very intuitive yet powerful implementation of Software Transactional Memory. There’s a nice introduction on it in R. Mark Volkmann’s famous article.

One notable feature of STM is a `ref` – shared binding which can only be modified inside transactions. Transactions are wrapped in `(dosync)` and behave similarly to database transactions. Value of a `ref` can be set in 3 ways: `ref-set` (obvious setter), `commute` and `alter`.

One thing that wasn’t obvious to me after reading the article was the difference between `alter` and `commute`. They both take a `ref` and a function, set the `ref` to result of running the function on previous value of the `ref`, and return the new value.

I decided to test its behavior with this simple experiment. Define an integer counter. Start 50 threads that increase this counter with `alter` and print the new value. After all threads finish, print the counter. Then repeat with `commute` instead of alter.

Here’s the sample with `alter`:

```(def my_num (ref 0))

(defn inc_alter []
(dosync
(try
(println "Alter: " (alter my_num inc))
(catch Throwable t
(do
(println "Caught " (.getClass t))
(throw t))))))

(defn test_alter []
(do
(println "---- Alter ----")
(println "---- After loop: ----")
(inc_alter))))

(test_alter)
```

The output can look like:

```---- Alter ----
Alter:  1
Alter:  2
Alter:  3
...
Alter:  47
Caught  clojure.lang.LockingTransaction\$RetryEx
Alter:  48
Caught  clojure.lang.LockingTransaction\$RetryEx
Alter:  49
Caught  clojure.lang.LockingTransaction\$RetryEx
Caught  clojure.lang.LockingTransaction\$RetryEx
Alter:  50
---- After loop: ----
Alter:  51
```

Explanation: when `alter` detects that the `ref` has been changed in another concurrent transaction, the whole transaction is rolled back. Then it is automatically retried. That’s fairly intuitive and straightforward.

When we replace `alter` with commute, it’s quite a different story. The output can look like:

```---- Commute ----
Commute:  1
Commute:  2
Commute:  3
...
Commute:  43
Commute:  45
Commute:  45
Commute:  45
Commute:  45
---- After loop: ----
Commute:  51
```

The modification is commutative. No transaction is ever rolled back. But it also means that inside the transaction a stale value can be used, what is the reason for getting “45” several times.

It doesn’t work exactly like I described earlier (get value, commute, set value, return). In fact, when `commute` is called it instantly returns result of running the function on the `ref`. At the very end of transaction it performs the calculation again, this time synchronously (like `alter`) updating the `ref`. That’s why eventually the counter value is 51 even though the last thread printed 45.

By the way, have a look at the sample code and think about how much it would take to implement it your Java or C#. Not only is it multithreaded, but also synchronized, transactional (with ACID or at least ACI), serializable with auto-retry, and so on. So much going on in just 20 lines.

# Clojure vs. Java: The Beauty of Conciseness

I decided to take up something completely new: Functional programming in Clojure. It’s far from being the most popular tech on the market, but on the other hand it’s a chance to experience something completely new that might come in handy in future.

One of the first striking observations is how concise it is compared to the old, heavy and static languages. Here’s an excersise: Write a function that flattens a list of objects.

Here’s what it looks like in Java (24 lines, 543 characters):

```import static java.util.Arrays.asList;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class Hello {
private static List<Object> myFlatten(Object in) {
if (in instanceof List<?>) {
List<Object> out = new ArrayList<Object>();
for (Object obj : (List<?>) in) {
}
return out;
} else {
return Arrays.asList(in);
}
}

public static void main(String[] args) {
System.out.println(myFlatten(asList(1, asList(2, 3, asList(3,
asList(asList(3)))))));
}
}
```

Now compare that to Clojure (6 lines, 141 characters):

```(defn my-flatten [v]
(if(sequential? v)
(vec (apply concat (map my-flatten v)))
[v]))

(println (my-flatten [1 [2 3 [3 [[3]]]]]))
```

In this case Java is awfully verbose. Some of it is related to static typing. That’s fine, we know how useful it becomes as the amount of code grows.

However, even in this trivial example the functional approach and Clojure show their strengths. All the interesting logic resides in this single line:

```(apply concat (map my-flatten v))
```

That has little to do with static typing or richness of Clojure’s standard library. It owes to the functional approach and closures

How simple and readable is that?