Canary tests are mostly used for verifying the performance of new features or software by giving access to a small subset of users. But why is it called canary testing? The history behind the term is quite dark as coal miners have employed canary birds to detect toxic gases preemptively. Birds are more sensitive to gas leaks, so they served as an easy “tool” to identify the high toxicity levels. When birds died, coal miners knew it was time to evacuate.
In the software development world, canary releases allow testers to collect feedback from a small group of users before making the software available to everyone. The principle stays the same – testers receive an early warning from the selected subset of users and fix the detected issue before official release. How does canary testing work, and what are the main advantages?
What is Canary Testing?
Canary deployment is a performance testing technique used by software companies whereby testers release new code to a minute group of users, roughly 1%. While simultaneously keeping the older app version alive and available to all users, businesses can validate new features without downtime. Besides that, a canary test will uncover hidden bugs and errors that occur in the production environment.
The cornerstone of testing canary releases is the implementation of feature flags. They help coders remotely turn certain features on and off and control how many end users get access to them. Adopting the feature flag strategy allows the development team to limit access to the new version of the software to 1% of the user base while tracking performance metrics.
If something goes wrong during the canary release, you can easily turn off a certain feature and avoid reputational and monetary losses.
The feature flags also allow you to gradually grant access to more and more users as you start scaling up without hindering performance and other metrics. You can start with a small subset of customers and slowly increase the coverage to 10%, 30%, etc. Different user load levels might bring up new issues, making canary deployments the perfect practice to detect all hidden pitfalls.
Benefits of Canary Testing
Canary testing is one of the most popular types of software testing as it is relatively easy to execute, cost-effective, and beneficial for the customer experience. What are the most important benefits of canary releases?
- Better user experience – Even though a small number of users will be exposed to the new software, a vast majority of your customers will not experience any bugs. Keeping the number of users who get access to the new functionality as small as possible allows developers to test in the production phase without losing customers’ trust.
- Ready rollback strategy – Thanks to the simplicity of feature flags that can be turned on and off at any time, software testers always have a ready rollback strategy at hand. Once the software reaches dangerous levels of errors, you can quickly turn off the feature and remove it from the application altogether.
- Faster time to market – Canary tests are short and take just a few minutes to be executed, making this type of testing especially effective for small and frequent updates. Shorter release cycles reduce time to market while also keeping money in the budget. As a result, your company delivers more product value in a shorter time and for fewer spendings.
