Executable Jar for Integration Test

Recently, I wrote an integration test infrastructure at work.
Since integration test is different from simple unittests, we made our integration tests to be an executable jar.
In this blog post, I'd like to write about why we made executable jar and some of the considerations to achieve it.

Integration Test

First, the word "integration test" is very vague.

To clarify, in my case, integration test was:

  • run group of tests against externally running application

So, it is a black box test.

Integration tests as an executable jar

Instead of running tests from maven command such like mvn test or mvn verify, we generate an executable jar from integration test project.
There are multiple reasons, but first I want to describe our build infrastructure.

We use Jenkins and Artifactory for our build.


Our Jenkins jobs do followings:

  1. compile and run unittests
  2. deploy the artifacts to Artifactory as a build
  3. provision a new VM
  4. run the latest app resolved from Artifactory by specifying the build id
  5. run integration tests against the latest app


The benefits of using Artifactory for binary management are:

  • Don't need to checkout entire project to run integration test
  • Don't need to compile project to run integration test
  • Easy to provision VM with specific version of app using the build ID
  • Accurately reproduce environment(application version and integration test version) by retrieving binaries by build ID

By making integration test as an executable jar, we can retrieve binary from Artifactory and simply execute it in Jenkins Job.

To run tests

We use JUnit4 to run integration tests. In the main method of the executable jar, it simply calls JUnitCore to programmatically invoke all tests.

However, just invoking tests is not enough for integration test.
We have to consider followings:

  • generate test reports
  • code coverage
  • parallel test run

Generate test reports

To show up the test results in Jenkins, running integration tests from executable jar needs to produce XML reports.

If maven, gradle, ants are used to run tests, they by default or configuration generate XML reports. However, since we run tests from executable jar, we programmatically run tests using JUnitCore. Unfortunately, junit4 doesn't come with XML reporting feature for Jenkins to pick up.

I found XMLJUnitResultFormatter class in Apache Ant, however the class cannot directly used by JUnitCore due to the difference of interfaces.
I wrote a blog post "Generate JUnit XML report from JUnitCore". There, you can find how to integrate XMLJUnitResultFormatter to JUnitCore in detail.

Once executable jar starts producing XML reports, Jenkins can accurately show the result report same as unittests triggered by maven/gradle/ant.

Code coverage report

We use JaCoCo to generate code coverage for tests.
Our web application is also an executable jar. So, simply attach JaCoCo agent to JVM when we start app for integration test.
The generated report files can be seen in Jenkins using JaCoCo Jenkins Plugin.

> java -javaagent:jacocoagent.jar=destfile=integration-test.exec our-application.jar

Parallel test run

Usually integration test is a long running test. Thus, parallel run is necessary to shorten total run time.

Still POC but I was able to parallel run the tests by providing N number of environments, as well as configuring JUnitCore to concurrently run tests.
I have summarized the setup in my blog post: "Programmatically run JUnit in Parallel"

Summary

So far, we are happy with current integration test setup.

I had to deep dive into JUnit4 a little bit for test report generation and parallel run. Hopefully in JUnit5 (JUnit-Lambda), setup becomes straightforward, configurable, and easily achievable.


Written on October 16, 2015