The Aim
Our goal is to have a CI pipeline which can run Cypress browser tests against a web application which is built as a Docker container (e.g. an application that you run on Cloud Run or Amazon ECS).
Building web applications in a containerised way allows the development team to have a local coding environment that works the same for everyone, and also makes it easier to port the application from one cloud provider to another. Being able to test your application with browser tests using Cypress is enormously helpful, and bringing these two things together on an automated test pipeline enables a really slick workflow for developers.
If your web application isn’t itself containerised, then you might want to take a slightly different approach, which is mentioned at the end. For now, we’ll cover the containerised application scenario.
The Problem
Depending on your CI system, the problem, or problems, may vary…
On GitLab CI
Details aside, on GitLab CI, you might encounter a problem in which Cypress’s Chromium process runs out of memory. I haven’t done particularly extensive investigation into this yet, but if you encounter this bug, and like me are unable to find a fix, then it’s a bit of a show-stopper. It could be that this issue only affects the Docker-in-Docker (“dind”) service, but given that our aim is to run Cypress tests against an application that’s a Docker container, the dind service is rather crucial. So for now, in GitLab, this game ends here.
On Google Cloud Build
The scenario we’re focussing on is the one in which your application is a Docker container. Cypress is also available as a Docker container, so essentially we want to be able to run these two containers side-by-side and have one make requests to the other, and then terminate both containers.
In Cloud Build, each step of the pipeline is itself a Docker container. You can specify the container image for each build step by using the ‘name’ field. So you can run a Cypress container using:
steps:
- name: 'cypress/included:9.1.1'
...
And you can run a container which you’ve created yourself, e.g. your web application, using:
steps:
- name: 'gcr.io/my-project/my-application:latest'
…
The Potential Solutions
So we could use the above to have two separate steps, one which runs our web application container and another which runs our Cypress container. But each step is kind of separate, and we want our two containers to run at the same time and talk to each other. You can specify dependencies for each step, and so you can effectively make two steps run in parallel to each other by giving them a common dependency (or no dependency). But to run Cypress tests against a web application, we need the application container to stay running until the Cypress tests have finished, and we also need the two containers to be able to talk to each other. Just as importantly, we also need to be able to stop the application container once the Cypress tests have finished, otherwise the application step will just stay hanging forever and our pipeline will never complete.
An alternative approach would be to build a single container image which incorporates both our application and Cypress, so that we could then run everything within a single container. While that might work, it would rather defeat the purpose of having a containerised application, because your reliably-reproducible application container would then be having Cypress (and all its dependencies) shoved into it for the purposes of testing. So the application you’d be testing wouldn’t be the one you’d be deploying. Unless of course you used this single, combined container as your deployed application container, but then you’d be hugely bloating your application container with Cypress just to accommodate your CI setup. Neither of these scenarios seem sensible. So we want a solution that lets us run two separate containers together side-by-side.
The Solution
There’s actually already a tool for running multiple containers simultaneously, Docker Compose, which allows the containers to connect to each other and stops them.
This makes the whole process work perfectly, but it requires a slightly involved pipeline setup.
Cloud Build provides a pre-built container image for Docker (so you can have a pipeline step that runs Docker things), but perhaps surprisingly that image doesn’t include Docker Compose. So you need to use the docker-compose image from Docker Hub: 'docker/compose:1.29.2'.
So in what becomes a little bit like a Russian doll collection of Docker containers inside other Docker containers, we use the docker-compose Docker image to build a Docker Compose image, and then we use the Docker Compose image to run our application container and the Cypress container together in order to run our browser tests.
To summarise, our slightly Russian-doll-esque collection of Docker containers, in our CI pipeline needs to:
- Use
gcr.io/cloud-builders/dockerto build our web application image. - Use
docker/compose:1.29.2(which is itself a Docker container) to run docker-compose with adocker-compose.ymlfile which:- Runs our application image.
- Runs the dockerized Cypress image to run browser tests against our application.
The Gotchas
Cypress must wait for your application to be ready
In your docker-compose.yml file, you can use the depends_on option to specify that the Cypress service depends on your application service, so docker-compose will start up your application container first before starting the Cypress container. But although it will make sure that your application container is started, it won’t wait until the application is actually ready. If your application is quite heavyweight, or has startup scripts which take a few seconds to run, Cypress may well get going before your application is fully ready, so its first request to your application server will timeout and the whole thing will fall on its face before it gets going.
A way around this is to prepend the entrypoint command in your Cypress container with a script which will wait until the application is ready. You can see this in our example project below.
You need to get the WORKDIR right and pass files down to Cypress
Cloud Build defaults to setting WORKDIR to /workspace/, which probably isn’t what you set it to for anything else, so this can easily trip you up. When your web application is running inside docker-compose this probably won’t affect you, as your application container is one level removed from Cloud Build here, tucked inside docker-compose, so is safe from its meddling. But if you’ve got another step that, for example, runs backend tests on your application, Cloudbuild’s meddling might taunt you.
Additionally, if your Cypress test spec files are inside your application folder, you’ll need to route these down from /workspace/ into the place where Cypress expects to find them inside the Cypress container. In the example project linked below, you can see how we do this in the docker-compose-for-cypress.yaml file.
On a related note, Cloud Build automatically leaves artefacts that are placed in the /workspace path by one build step available for the next build step. You can take advantage of this, for example if one step installs files which are then used by the next step, but you’ll need to make sure you’ve got your WORKDIRs aligned so that you can pick up the files.
Example Project
Rather than provide a jumble of code snippets for you to pick up and staple together, we’ve created an example project to demonstrate this Cloud Build setup. If you’re familiar with Cloud Build already and just want to see the config, you can jump straight to the cloudbuild.yaml file.
When we first did this we struggled to find any articles that laid out the options and gave an example solution. I hope that this explanation and example setup saves you some of the time that we spent figuring this out for ourselves. If you have any questions or suggestions for improvement then feel free to drop us a line at p.ota.to or on Twitter, or send us a pull request to the repo!
For non-containerised applications
If your web application is not already built as a Docker image, then you can take a slightly different approach, because your application and Cypress don’t necessarily need to live in separate containers, and so you don’t necessarily need to use docker-compose to run them in tandem. You could, for example, just have a Cloud Build step in which you:
- Run
npm install cypress - Run your application dev server command, putting it into the background and storing the process ID.
- Wait for the application server to be ready (see Gotchas).
- Run the Cypress tests.
- Kill the dev server process.
Whether this brings you more or less pain than the docker-compose route is something that only you can discover!