I recently rewrote the end-to-end tests for a service. This brought up the question of what we should test in the end-to-end (e2e) tests vs our Dockerized tests.
This article is the first in a series about system testing:
- Dockerized testing vs end-to-end testing
- How to setup Dockerized testing
- Readable Java system tests with good old JUnit
e2e tests vs dockerized tests
Since the original e2e tests were written, we (my team) have also introduced another type of tests. We call them “dockerized system tests”. These are similar to the e2e tests. They also test the connections between services in the system, and how the system works as a whole. One key difference between e2e tests and dockerized tests is that the dockerized tests do not test the system when it is deployed the way we deploy it in staging and production environments. Our dockerized tests bring up all services in a single machine, and we usually don’t set up redundancy, clustering, load balancers and such things.
A benefit of the dockerized tests is that they are easy to spin up and can run either on your local machine or on your CI server (like Jenkins) and you don’t need to deploy the services to dedicated machines. This means you can run them more often, and earlier than e2e tests. If you work with branches, you can run them in the branch builds. Normally, we only deploy the master branch to e2e test environments.
The introduction of dockerized tests have reduced the need of e2e tests, but there are still things that won’t be tested in dockerized tests. As a general principle, you should always try to push your tests down the test pyramid.
What remains to verify in e2e tests?
So, what remains to be tested by e2e tests? Well, everything that isn’t already well covered by a test on a lower level. This includes verification of the provisioning itself. Maybe you use Ansible or Terraform to create infrastructure like server instances, load balancers and configure firewall settings. This is not done on your dockerized environment and therefore cannot be tested there. You don’t want that to be untested, do you? Let’s verify that our provisioning scripts are good by running some tests on the e2e environment!
Is your service coming up healthy? Have you setup your clustering correctly? Can you reach your service through the load balancer? These are things that should be tested on your e2e environment.
You could also run the same system test suite that you run on your dockerized test environment on your deployed e2e environment. They should pass there too, right?
I asked a few colleagues at Crisp what they thought should be tested in a e2e test. In summary the general idea was: “A few happy-path smoke tests.” The discussion also drifted into testing in production and production monitoring, but that’s the topic of another article.
Why just happy path tests? Well, your lower level tests should really cover the error cases. You just want to make sure everything is hooked up correctly. Your services come up healthy and can talk to each other.
Why just a few tests? Well, remember the test pyramid? You should really try to push your tests down the pyramid. There shouldn’t be a lot of tests needed in the e2e suite. This might not always be true for large legacy systems with poor test coverage. Then, maybe you want to have quite a few e2e tests in order to dare making the system more testable on lower levels.
Summary
What we test in which stage has changed with the introduction of Dockerized test setups. With Dockerized test setups we can test earlier and faster and have quicker feedback. But Dockerized test setups don’t completely eliminate the need for testing on a deployed e2e environment.
Read on about how to setup a Dockerized test environment!
