Making integration tests run faster

The problem - slow tests

Our integration tests, using both Junit and Cucumber, are at least two orders of magnitude slower than our unit tests, which is to be expected. But we noticed that over time, our integration tests were just getting slower. 

Circumventing for the moment the debate of if and when integration tests are appropriate (some actually call any integration testing "a scam"), the simple fact of the matter is: we have them, many of them. Whether it's BDD we'd like to support, end-to-end tests, strict integrations with other frameworks (web-services, message-queues, persistence layers etc.) and those murky integration tests put in place to circumvent very non-test-friendly legacy code - they all need to be supported.

So, what can be done?

Profiling our test suits we found - unsurprisingly - that loading the Spring context was the number one hot-spot, both when running Cucumber and using Junit. And it was the growing size of the context that made single-test runs slower over time - starting up the context simply took more and more time. For our context loading, we found Spring took about 20 seconds to package-scan our annotated beans and another 5 minutes to actually load the beans, doing the needed wiring and initializations.
Start-up time isn't a big issue for our Jenkins builds - they reuse the same context between tests so load is done only once - but it was a big issue for developers. Waiting over five minutes just for the test to start meant developers were simply not running them locally.

Making Spring context load faster

The first step we took was to try and load the Spring beans lazily.

The easiest way to do this is declaratively, via the XML files (an attribute of either the <bean> or <beans> tags). Alas, for us this approach was not sufficient, as some of these XML's are being imported from other jars, and so we had difficultly controlling them: the thing with lazy initialization is that it needs to be done "all the way down" - if some bean isn't lazy, it will force all the beans it depends on to be loaded eagerly as well. So misbehaving beans imported from other jars "hampered with our cause". 

What eventually worked for us was to implement our own custom context loader for tests. Specifically, our own version of Spring's SmartContextLoader. This loader of ours changes the bean definitions to lazy during context start. That makes sure all beans are indeed defined as lazy (code below).

Two caveats are called for though:

1. In production, we usually want to have our context loaded eagerly (that's Spring's default), because we want to fail-fast if it's broken. Having it load lazily in tests means your load sequence is different than production. If your beans do any non-trivial stuff inside their initialization (they really shouldn't - but we can't always have it our way) be aware of this difference.
2. It seems there are certain types of beans that simply can't be set as lazy without breaking the context loading, so these need to either be filtered out from the context loading (if possible) or kept unchanged.

Another important lesson we've learned was to become cautious of Spring batch. Spring batch jobs are nice to have, but they come with a price: they put a very large burden on the context, creating lots of AOP proxy beans, and these can't be loaded lazily. If you defined such jobs - you need to take extra care that they are defined in separate XML's, added to the context only when really needed.

Componentization is key

The more profound steps we've embarked upon were to improve our application's componentization, at two levels: 1) breaking our main applications (AKA mother-ship / monolith) into a set of smaller services; and 2) better defining the internal component structure of our main applications.

Well defined components allow for:

1. Smaller, isolated and self-sufficient contexts that load quickly. Tests can then load only the minimal contexts needed to run the tests.
2. Stabler code: Spring contexts with many direct and transitive dependencies easily break due to "far-away" changes made by distant team-members working on some seemingly completely unrelated feature.
3. Most importantly: clear and well defined components help one understand what their code is doing.

Neglecting to pay attention to ones higher levels of componentization tends to lead over time to applications where everything is connected to everything else - a situation also known as a big ball of mud. Unfortunately, improving an application's level of componentization is difficult - it requires much more skill - and work - than refactoring single classes. In Uncle Bob's excellent Object Oriented Principles one can find six different principles to adhere to in order to reach this super important goal. One of the nice "side-effects" of this effort is, well, faster integration tests.

Our custom context loader: