Trust but verify. Using Pact for contract testing

Over a million small businesses, and their advisors are looking for the best cloud apps that integrate with Xero. Partner with us, and we’ll make sure they find yours.

Photo credit Quinn Dombrowski

TL;DR: This article introduces the basic concepts and motivations behind contract integration testing and demonstrates with a sample project how it can be implemented using Pact. The full source is on Github. Let’s get down to it.

When setting up a continuous deployment pipeline for any project, having a purposeful testing plan is of utmost importance.

Effective test suites with short feedback loop

An effective test suite includes a combination of testing strategies that leads to high test coverage, that in turn drives confidence. These strategies are typically in the form of unit, component, integration and acceptance tests. Monitoring and alerting too.

The test suite needs to run wicked fast, and be reliable. This cannot be stressed enough. If tests are flakey (i.e. often return false negatives or positives) it cannot be depended on to give us the desired confidence that we need.

Tests also need to run as early as possible in the continuous deployment pipeline to shorten the feedback loop for the developer. So ideally, most tests should be designed to run when pull requests are built (on standalone build agents that are sandboxed from the application environment). To achieve this requirement, tests must make no assumption about the availability of any of its external dependencies, and be designed to run in isolation.

Tests that are designed to be fast, reliable and isolate failure will have a short and effective feedback loop that informs the developer with confidence whether the product is working or not.

The problem with E2E integration testing

Unit tests is a good example of one such testing process that achieves the desired effectiveness being aspired to — they are fast, reliable and isolate failure. However, unit tests on their own are not enough because they give no guarantee that the tested units work well together.

In the absence of the perfect unit test suite, end-to-end (E2E) integration tests are typically the testing strategy of choice that is used to try and ascertain confidence in the system working as a whole.

Even though E2E tests are full of promise to assert the overall product working a whole from a user’s perspective, they quickly become a very bad idea because of their awful feedback loop — slow, unreliable and depend on too many things at once.

No surprise… they are darned things that sell a good story until you try them out

Integration contract testing

An alternative testing strategy that complements the deficiency of unit tests but verifies that components with external dependencies work together is integration contract testing.

An integration contract test is a test at the boundary of an external service verifying that it meets the contract expected by a consuming service — Martin Fowler

These type of tests are built on blind trust that agreed boundary contracts will be upheld, and focus on defining success as the correct verification of requests received or sent at the boundary.

To better explain the point, take the example of a simple web application that queries a movie database (e.g. OMDb API) based on a search term entered by a user:

Given that a user accesses arrives at our amazing app’s homepage
When they search for the keyword ‘titanic’
Then they see a search result of the movie ‘Titanic’

Our integration contract test in such a scenario is only interested in asserting that the right web request to the external dependency was sent when a search is invoked. The test does not need to rely on calling the real external dependency because it is outside the boundary of its concern.

So in the case of the above example, an integration contract test need only verify that a GET request with the query string of/?t=titanic&r=json was invoked on search. The tests do not need to verify that the OMDb API successfully performed a search in its own data store, and returned any expected data in any count or form. We trust that the api does what it says on the contract.

Pact as Mock Service Providers for Consumers

To aid with this type of testing, the Pact specification and its ecosystem of tools is a great way to implement consumer driven contracts testing.

Mock Service Provider — Used by tests in the Consumer project to mock out the actual service Provider, meaning that integration-like tests can be run without requiring the actual service Provider to be available.

Pact JS for example can be used to spin up mock service providers to act as a “Test Double” that is configured with expectations of what calls they are expecting to receive and pre-programmed responses to return.

So sticking with the same movie search example, Pact JS can be configured on the Given statement to expect and return the (exact) following requests and responses:

The integration tests do not need to mock any repositories or depend on network availability of any real external dependencies. Aside from a small time penalty of starting the Pact servers the first time, they run as quick as unit tests with stubbed responses returned from the mock service providers that are running on real local ports (at the HTTP level).

Upon receiving a request, Pact will verify that all the expected required details have been provided — be it headers, query parameters, cookies, etc. Because requests and responses are going through HTTP, the code tests repository logic and any de/serialization settings configured for the application as well.

Verifying Pact logs — ensuring Consumer and Provider are compatible

So far, Pact comes across as a normal HTTP Mock Server.

However, the magic that lets Pact shine as a great tool for contract driven testing is its ability to verify recorded Pact interactions.

When the contract tests run for an application with an external dependency (aka the Consumer), Pact records all interactions in a “Pact file” using a standard format. This file can then be shared with the external service dependency (aka the Provider) to continuously check how other systems (or teams) are using their services and what assumptions they are making in the process.

This feature is particularly suited for teams working on micro services to consume each other’s services and write tests that are truely trustworthy because of the two way verification between Producers and Consumers have agreed to always take place. It’s a team game!

A great talk by Ben Sayers and Mauri Edo from Atlassian on how they organised many of their teams to share their Pact logs together to easily verify contracts and pick up on wrong assumptions & breaking changes early

Wired up example

To complement this writeup, I’ve created a sample movie search web app that allows users to search for movies in the OMDb API. The web app has a basic contract integration test setup that runs on Travis CI upon every build.

The web app is implemented in React. It uses WebdriverIO as the test runner, Cucumber.js as a testing interface and Pact JS for contract testing.

There are plenty of quirks that I had to think about to get this end-to-end sample app up and running, but that is worthy of a write up of its own.

The full source is viewable on Github.

hadynz/pact-testing-example