2k Unit Tests in One Year – Lessons Learned

by Stefan

I was in the lucky position to work on a green field project during the last two years. At the time the project started someone pointed me to the clean code books by Robert C. Martin. The proposed approach to professional software development seemed to be refreshingly different to the approach I was used to before. So the direction was clear: most of the code that has been created during the last two years was written test driven. I found myself writing an amount of tests in a way I never did before. I actually wrote something between 1k and 2k tests in the first year which is more than 5 tests per day. Today, the whole test suite is more than twice the size and provides a code coverage of more than 90 percent. (footnote: the UI code is actually split using an MVC like approach. The view code is not tested at all, as it doesn’t make sense IMHO to test the composition and labeling of widgets with unit tests.)

The Debugger was my best friend

When I started developing software in my first job, the debugger was my best friend. I used it to understand existing code, and even more important, to verify that my logics work. I was really proud that I was able to watch and inspect several variables simultaneously. Debugging code was one of the great insights I gained and I considered it to be the key to mastering software complexity.

This totally changed when I started using TDD. When I had spent half of my time with debugging in the past, now I used it only once a month or so. Relying on a debugger for code verification is not very profressional. It only reveals a small part of the system’s state at a single point in time. A comprehensive suite of tests covers a much larger part of the system’s behavior and allows verifying the requirements at any time with a single mouse click. Nowadays I rarely use the debugger. Exceptions are examining concurrency issues, e.g. the access to a shared resource by two different threads. This is easy to reproduce using breakpoints but can become very cumbersome or even impossible to reproduce with Unit tests (feel free to comment if you disagree).

I have always spent a lot of time with refactoring existing code, like splitting modules for better separation or merging code for better reusage. When starting to go with TDD I was afraid that the tests would double the amount of time I needed for refactoring, because I had to modify all tests, too. Fortunately, this did not become true for the following reasons:

  • Most tests are very small and only cover a single aspect. Changing functionality often means to change only few tests.
  • The test code is written following the same design principles as productive code. Some people argue that these are “only tests” to justify copying code and spaghetti code in general. I saw this leading to a test suite that is hard to maintain. And selling a couple of days to your product owner to refactor your test suite is not fun. So I tried my best to keep the coding standards for tests as high as for production code.
  • When writing tests that deal with complex subsystems or need a lot of setup, I started writing helper classes (fixtures) to keep tests simple. Although that might feel strange one should not hesitate to write tests for larger fixture classes, too.
  • I wrote integration tests to ensure that the wiring and communication between classes and modules worked as expected. These tests are needed to verify basic functionality after larger refactorings, e.g. splitting a large module into smaller ones or replacing the persistence provider implementation.

Following these guidelines I never had the feeling that refactoring tests decreased the development speed. In fact, I felt much more safe when doing major changes, even when the release date was in sight.

To test or not to test

I very often was in doubt which aspects of code I want to cover by tests and which not. Do I need test for the embedded workflow engine? Makes sense, if you want to verify that it works as expected in your scenarios. This also becomes very handy when updating to newer versions. I can safely perform the update when my workflow test suite passes. But what about more runtime like components as OSGi or even the JDK?

Java annotations are very popular for many purposes. It is easy and useful to test JPA annotations like @NotNull in integration tests. But what about annotations of your DI container, like @Inject and alike? I left those for my system test suite that ensures that the whole application is bootstrapped correctly. This also applies to other configurational things.

Some words about test execution time. It’s important to make the tests run as fast as possible. However, they simply need their time if you have several thousands of them. I saw teams that started holding a static state for all tests to safe time for test setup. However, this leads to instabilities as the tests are not really separated from each other. With this approach OMMIW becomes a popular term as running a single test uses a different (“clean”) setup as when all tests are run consecutively.

Another bad practice are super test classes, mostly named BaseTest, AbstractTest or alike. Following the delegation over inheritance principle all my tests don’t inherit from another class but might have similar setups and teardowns to handle common preparation tasks. This is much more flexible and improves the overall maintainability of the test suite. Actually, the best advice is to keep the number of integration tests low and the number of pure unit tests high. And of course make sure that all devs and the CI system is using the best hardware money can buy.

A last thing I learned is to make sure that I was able to run tests anywhere and anytime. No matter, if it’s a single test method, a test class or all tests within a package, project or the workspace. I used Eclipse and JUnit 4. I didn’t maintained JUnit TestSuites or special Eclipse run configurations. Using the run as… command had to work on any level, and if it didn’t, there was something wrong with the test itself. During CI, Maven picked up all tests, so no TestSuites necessary there, too.

Fantastic, amazing, gorgeous, isn’t it?


One Comment to “2k Unit Tests in One Year – Lessons Learned”

  1. I’m just about to dive back into my codebase today, and I’m faced again, as I am dozens of times a day, with the question of whether to begin by writing tests. I’m a TDD man, I keep the faith, but just sometimes I start to wonder if I’m getting the returns that I want out of my unit testing efforts. So thanks for your post; it encouraged me to keep going, to keep my tests small and fast, and to reflect on the horrors of the alternative.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: