In this article, we will discuss the testing pyramid - what it is and what are some problems with that.
We will also discuss a different approach: The testing diamond.
Testing Pyramid
The testing pyramid is a testing strategy that emphasizes a test suite's composition and its coverage across different levels of the application. The pyramid shape implies that there should be fewer high-level end-to-end (E2E) tests and more low-level unit tests, with a moderate number of integration tests in the middle.
Unit Tests: These are small, fast, and isolated tests that verify the behavior of a single unit of code in isolation, typically a single function or method. Unit tests ensure that each individual unit of code behaves correctly.
Integration Tests: These are medium-level tests that verify that the interactions between different units of code work correctly. Integration tests are usually slower and more complex than unit tests and may require more setup and teardown.
End-to-End Tests: These high-level tests verify that the entire application works as expected. E2E tests simulate user behavior and interactions and are typically slower and more brittle than unit or integration tests.
So the biggest foundation is unit tests. And if you are following patterns like TDD (Test-driven development) then this is no news to you. But there are some problems with unit tests:
- False Sense of Security: Unit tests only test individual units of code in isolation and may not capture bugs that arise when units are integrated. Over-reliance on unit tests may give a false sense of security.
- Limited Coverage: Unit tests may not cover all possible scenarios or edge cases, leading to insufficient test coverage. Integration and E2E tests are needed to ensure the entire application works correctly.
- Slow Feedback Loop: Although unit tests are fast, the feedback loop may be slow when codebase changes require many tests. This can slow down development and impede iteration.
Especially the last point is something I have seen multiple times in the past - a solid test suite with thousands over thousands of tests. Of course, one could argue that those tests were not perfect, so we suffered from this. And this is totally valid, but I am also not the only one encountering those problems.
Another issue is that, especially with microservice-oriented architecture or bigger Web-API projects, you are not really in the position of your client. Your client, in the form of your front-end, is using your API or services. To enable that, we can shift from a big foundation of unit tests to a big foundation of integration tests:
The advantage of this is that you can create tests that cover a whole business requirement in the vertical stack! So you can name your tests accordingly (Hey, BDD!). You are also testing (de)serialization, communication, and your DI-Container. This is especially helpful in a micro-service-oriented architecture (or Web-API's). The best thing is that you write fewer tests overall with some quality outcomes. Please don't get me wrong, there is still plenty of space for unit tests. Imagine scenarios like caching, where it is quite hard to make a useful and stable setup from an API point of view.
Now you might ask yourself if I have to spin up a real DB and backend every time, isn't that slow? Well, let's have a look at Entity Framework - they have over 3300 tests and connect to a cosmos database (locally emulated via Azure Cosmos DB emulator) for the majority of tests where a real database would be needed. The overall runtime is something from 4 to 7 minutes. Quite fast. And for the server?
ASP.NET Core Web API
For ASP.NET Core Web API you can utilize the WebApplicationFactory. Oversimplified, it is an in-memory server that you can spin up in your integration tests in a matter of milliseconds. I will not go into too much detail here, because I already wrote a whole article about this: "Introduction to WebApplicationFactory".
Conclusion
With a shift towards diamond-shaped testing, you can write less, but more meaningful tests that are really a living documentation.
