TDD, Emergent Design, and Punctuated Equilibrium

In my experience good TDD designs tend to have evolved (and hopefully continue to evolve) in a sort of punctuated equilibrium. There are long periods of small incremental changes to a code base, punctuated by infrequent bursts of large design-level refactorings. While the TDD mantra is "Red, Green, Refactor", I think a more accurate version would be "Red, Green, Red, Green, Red, Green, Refactor". Less catchy, I know.

Letting tests drive your design is great, but if you're not careful you will end up with a design which is the result of a simple accreation of functionality. This of course is the reason for the "Refactor" part of RGR. However, in my experience the refactoring part does not occur in a steady flow. Rather there will be periods of time where only small implementation-level refactorings take place during the RGR cycle as new functionality is implemented. Then at some point something will give. Someone may have a design breakthrough, realizing a more expressive/elegant/clean way to express some functionality. Alternatively some niggling annoyance in a subsystem's design will pass the team's tolerance threshold such that they decide they need to clean up the design to remove the annoyance. Both events will lead to a brief flurry of larger design-level refactorings.

A somewhat minor example of the 'niggling annoyance' case would be when someone is adding a 4th optional parameter to a method call and finally decides to bite the bullet and pull all the optional parameters out into a single options hash. An example of the 'design breakthrough' case might be the third time someone adds a subclass in order to specialize some aspect of a base classes functionality. They might realize that really what's going on here is that each subclass represents a Strategy, which can be plugged into the base class. Now both of these examples are rather small-scale, but hopefully they illustrate the concept. It's hard to come up with examples of larger refactorings which are easy to succinctly describe without a bunch of background information.

A key observation here is that principles like YAGNI and 'do the simplest thing that will work' will not work well in isolation. Rather they must live alongside principles like DRY, seperation of concerns, do one thing and do it well, etc. If your sole focus is on doing the minimum amount to get your tests passing you will not end up with a supple code base which is amenable to change. That requires dilligent attention on your part.

Does this mean that TDD leads to bad design? No. TDD does not lead to good design on its own, but it does set the stage for good design to emerge. A team with the safety net of a strong test suite has the courage to follow through and realize these design-level refactorings. A team without that safety net may know exactly what needs to be done but will lack the courage to do so, because they know that any changes will lead to unknown breakages. Thus, the team will be reluctant to undertake these design improvements, as much as they would like to do so.

Finally, this idea of punctuated design equilibrium may sound familiar to folks who've read Eric Evan's Domain Driven Design. He talks about how good teams who are really working to understand a domain will occasionally find themselves up against a breakthrough which requires them to stop forward momentum on features in order to realize a more elegant expression of their core domain. That small section of his really excellent book was certainly in my mind as I started writing this post.

Introducing Noguchi

Generating HTML tables in ruby has always seemed like more of a hassle than it should be. Most of the times that I've had to do it I've been a little frustrated with how, well, ugly the view code looks. So, I decided I'd try and create a nicer way to build tables. The end result is a small table creation library called Noguchi. Given a set of data and some optional configuration it will render a table as HTML or CSV.

Show me!

Here are some quick examples of the kind of things you can do with Noguchi. There are more features than I will show here; for more details check out the README in the library's github repo

Creating a simple table from ActiveRecord model instances

table = Noguchi.table_for(users)
table.render

generates

Name	Age	Sex
Jenny	24	F
Dave	32	M
Hank	27	M

Rendering CSV

table = Noguchi.table_for(users)
table.render_as_csv

generates

name,age,sex
Jenny,24,F
Dave,32,M
Hank,27,M

Rendering collections of hashes

fruits = [
  { :name => 'banana', :color => 'yellow' },
  { :name => 'apple', :color => 'green' },
  { :name => 'orange', :color => 'orange' } 
]
table = Noguchi.table_for(fruits)

generates

name	color
banana	yellow
apple	green
orange	orange

Adding arbitrary custom-rendered columns

table = Noguchi.table_for(users)
table.add_field(:edit)
table.to_render_body_cell_for(:edit) do |context,cell|
  cell.raw_content = link_to( "Edit this user", edit_user_path(context.datum) )
end

generates

Name	Age	Sex	Edit
Jenny	24	F	Edit this user
Dave	32	M	Edit this user
Hank	27	M	Edit this user

The approach

I decided that since the goal of this library was to make table creation code look less ugly I would let the design of the API drive the process. So I started off by thinking about how I'd like table creation code to look like, basically just writing some rough ruby code that used the as-yet-unwritten library to render a basic table. The idea here was to get a simple core API which would render a basic table with very few lines of code, but to make the table rendering customizable by adding various tweaks to that basic table. Once I was happy with how the core API looked I started writing more complex examples, letting that drive out more advanced features in the API. These features were optional configurations which aimed to handle the most common kinds of customization I seem to end up doing to a table. Things like customizing a column header, adding an extra column to contain an edit link, things like that.

Once I had a rough idea of the API in my head, I started writing tests that described the functionality, implementing just enough of the library at each step to make the tests pass. I was plesantly surprised at how well this worked; the entire implementation was driven out by these tests. While I do tend to TDD most of the time, I've seldom done so exclusively. I tend to find myself spending some small amount of time outside of the Red-Green-Refactor loop, usually while exploring a new design approach or a brand new subsystem. I'm quite pleased with how this exclusively TDD approach played out in this case, although I wonder whether it would work quite as well in different circumstances.

Stormcloud will watch over the unicorns.

Hipster geeks crack me up. A nice ancillary benefit of being in this industry is I get to read blog posts about interesting technical stuff which also include phrases such as:

...we developed a new monitoring script, called Stormcloud, to kill Unicorns when they ran out of control...

...Monit would still monitor the master Unicorn process, but Stormcloud would watch over the Unicorns...

...child death during request processing ... would cause that request and all requests queued in the mongrel to send 500 "robot" errors until the mongrel had been restarted...

No need to fear, children. Stormcloud will watch over the unicorns. And if all else fails, a mongrel will send 500 robot errors.

In case you were wondering, the blog post these come from is about Twitter's app server infrastructure.