Short-lived testing environment for web applications

06 Jan, 2020CI/CD, Testing

Gitflow is so popular that people setup up their deployment environments according to it. One common practice is having a long-lived "develop" environment and pointing develop branch to it. The benefit of having a "develop" environment apart from staging/pre-prod is that developers can have a more loosely controlled playground so they can test something that can't be setup on their own machine, such as the deployment platform itself, network proxies, databases, etc. And when multiple developers are working on the same project, measures such as branch protection, pull request and code reviewing usually being applied to keep it in a good shape.

However, there are some drawbacks of this approach:

  • First and most obvious one: the "develop" environment is only for developers to test or check temporarily, so it’s waste of resource to keep it running 24x7.
  • The whole point of having a "develop" environment is to test something that can't be reproduced locally so it has to be as close to the staging/pre-prod and production as possible. So having one more environment means more maintaining and operating overhead to keep it running and all the settings to be up to date.
  • If all the changes have to merge to develop before creating PR to merge to master, all the changes have to be reviewed twice, so each change has to create double PR. And if the reviewer has any code change request in master PR, the developer has to go back change the code, create another develop PR, wait for the CI jobs to finish and then create another master PR.
  • If it's not required to merge to develop before merging to master, people would tend to not to merge to develop anymore. And eventually the develop is diverging from master further and further away. And when someone has to merge something to develop, the "develop" environment is most likely to be broken and the the developer has to fix it even it's not their fault.

The solution is to keep only one master branch which pointing to staging/pre-prod environment and have the script to create a temporary sandbox deployment for each PR. The settings, including volume mounting, external services, environment variables are copied form the staging/pre-prod environment. The sandbox is created when the PR is opened, and be removed when the PR is merged or closed.

The sandbox solution is inspired from the ZEIT Now for GitHub. It create a deployment for each commit because it mainly dealing with static web front-end applications. But it can also apply to back-end services, especially microservices, since each service is so lightweight and cheap that can be build up and tear down with very minimum impact to the whole system.

The benefits of having such temporary sandboxes:

  • Instead of having a dedicate "develop" environment that pre-allocate the resource to support all the projects to be running 24x7, we can reduce the resource consumption by letting different projects to share a much lower sandbox pool. The minimum required resource is the maximum
  • Each PR has it’s own protected environment, so there’s no side-effects because of multiple changes are merging simultaneously to develop branch.
  • Production hot fix can be tested in the sandbox with the same code and settings as the production deployment.
  • Sandbox is replicated from staging/pre-prod and all the external resources can be shared with staging/pre-prod environment.
  • Encourages developer to keep as much changes as possible in source code, which follows the microservice concept: microservice = source code + environment variables + attached services.

Powered by Gatsby. Theme inspired by end2end.

© 2014-2022. Made withby mdluo.