Where unit testing fails

Note: I wrote this post before watching Uncle Bob’s excellent NDC 2011 talk called Transformation Priority Premise, which talks exactly about the subject below, however I am still not convinced. I will write a follow-up to this post.

I remembered a story from a few years ago about “some guy” who tried to build a Sudoku solver using TDD (Test Driven Development) five times, failing and giving up, while “another guy” simply wrote a solver that worked.

A search led me to a thread on reddit, leading to this post by Ravi Mohan. As it happens, “some guy” who tried the TDD approach was Ron Jeffries, one of the fathers of Extreme Programming (XP), while the “other guy” was an algorithm and AI expert Peter Norvig. I highly recommend reading the original post and the updates, it makes for very thought provoking read (the original links to Ron Jeffries’ articles are broken, but you can find them by searching for sudoku on his site).

So what happened? How is it that the very people who preach TDD fail to implement it themselves? I believe the problem lies with grasping one of the most fundamental principles of unit testing, and that is – what not to test.

In one of the Software Craftsmanship Israel group meetings, our programming task for the evening was to implement a program that detects whether a given number was a Lychrel number. On the projector screen a summary of what a Lychrel number was, and some samples. We decided to split into pairs of people who knew how to write code using TDD (myself included), and people who were less experienced in writing tests. And so we began…

A Lychrel number is a number that does not form a palindrome when digits are reversed and then added together iteratively. For example: 56 + 65 = 121, 125 + 521 = 646, therefore both not Lychrel numbers. The smallest suspected Lychrel number is 196, failing to form a palindrome after more than 2 million iterations. This is also known as 196-algorithm.

Some 30 minutes passed before I realized that something didn’t feel right. We had written almost 10 test cases, all of which just asserted against known inputs and outputs we used from the projector screen. Several of us started arguing whether or not we should test the method IsPalindrome, since it was an implementation detail, and not a part of the interface.

I then remembered the story about the Sudoku solver and it made sense to me, just as it made sense to Vlad Levin – TDD is not an algorithm generator! The solution to the Lychrel number had just one solution – implementing the algorithm. There was no design that was needed, therefore TDD did not help in this case. All it had given us was a convenient way to input values to a function and getting a result.

At Typemock where I work I am sometimes asked via support how to test the DataSet or the DataReader, and whether our product can be used to mock the database. Yes, Typemock Isolator can be used to mock just about any object, the question is asking the wrong thing. You could create a fake DataReader filled with test data, but that would be too much work and very hard to maintain, and in the end will give you no benefit over using a real database. If you want to test whether your data access code works correctly – make it an integration test instead.

When I write unit tests today I try abstract my problem space. I no longer start at the very bottom, but rather I focus on the business requirement. A unit in unit test is not necessarily a method or even a class – it could be several classes collaborating together to achieve a business requirement. If your specification says “given a country, get an alphabetized list of cities”, don’t write tests for the sorting algorithm – it’s an implementation detail that you shouldn’t bother testing. Your assert should verify is that the list was indeed alphabetized, not which algorithm was used to sort it.

Many people very new to unit testing sometimes give up. They find it hard to know what to test, and end up spending many hours testing the wrong thing. When it later breaks, they find it too much of a hassle to maintain both the code and the tests, so they give up the later. The road to successful unit testing begins with understanding what unit testing is, but most importantly – what it isn’t.

  • John Hilts

    I remember that thread from proggit.

    I think Ron Jeffries forgot his own rule (well, xp rule) to make a spike first so he could first figure out how to solve the problem, then “throw the spike away” and write the TDD’d code with the newly gained knowledge.

    TDD won’t necessarily help generate the algo, but it will help you write an API you would consume to utilize the algo and integrate it into an app organically.

  • http://zsoldosp.blogspot.com Peter Zsoldos

    Would be great to see what Uncle Bob’s take would be on this problem now  - given his Transformation Priority Premise posts earlier (http://cleancoder.posterous.com/the-transformation-priority-premise) and his NDC 2011 talk about it (the later I haven’t seen yet), I wonder whether that would be applicable here

    • http://www.hmemcpy.com/blog/ Igal Tabachnik

      You know, it’s so strange, I’ve seen Uncle Bob’s talk immediately after posting this! I felt kinda silly during the talk, but towards the end I was still confident in what I wrote! While I agree with Mr. Martin on many things he teaches, I still don’t know what to make of this.

      I believe that reinventing the wheel is bad – same is for reinventing known algorithms using empirical methods (tests). In my opinion, one of the reasons so many jobs require a bachelor’s degree (or equivalent) in Computer Science is that programmers are expected to know this stuff! I wouldn’t want people on my team to “derive” a sorting algorithm – it’s a solved problem! Moreover, it’s a solved problem with a mathematical, not empirical proof.

      I will write a follow-up to the post, once I have a chance to ask Uncle Bob his opinion.

      • http://zsoldosp.blogspot.com Peter Zsoldos

        I think the TPP’s point is not that you should derive known algorithms, but rather illustrating (maybe proving would be too a strong word) the TPP concept with known, familiar examples. I think the TPP’s goal is to provide you with tools so when you are working on a new problem (business, algorithmic, whatever), it helps you choose your next step/test, or to see why you might be stuck implementing a test, i.e.: to help you realize you’ve made too big of a step at once. IMHO :)

        Reinventing the wheel actually has its places – it’s a great tool for learning (pointed out to me by http://twitter.com/felhobacsi), and implementing a basic testrunner is great way to get familiar with a new language :)

  • X v


  • http://about.me/fernandogrd Fernando

    That’s right. But it can still help you to solve something you didn’t grok yet. I used unit tests the first time I implemented an AVL Tree. And it was very helpful, cause every case I wrote, I broke the others. Ok, I was learning and the tests help me to not only finish the problem, but also that I was probably solving it the wrong order.

  • Miraj Mohamed

    This post is unnecessarily criticising the TDD approach.
    TDD doesn’t help you derive a solution or an algorithm. If you do not
    know the solution or have no clue about the algorithm to use, TDD is not
    going to help you. TDD only promises a better design. See the wiki for
    TDD: “designs can be cleaner and clearer than is often achieved by other methods”http://en.wikipedia.org/wiki/Test-driven_development

  • Miraj Mohamed

     TDD doesn’t help you derive a solution or an algorithm. If you do not
    know the solution or have no clue about the algorithm to use, TDD is not
    going to help you. TDD only promises a better design. See the wiki for
    TDD: “designs can be cleaner and clearer than is often achieved by other methods”http://en.wikipedia.org/wiki/Test-driven_development