Flaky Tests A Flaky test is one that has a non-deterministic outcome: it can pass sometime and fail other, for the same code, running the same test.
Why Flaky Test occurs ?
If the application interacts with browsers, external services or has asynchronous behaviour, probably there will be flaky tests.
Flaky Test are costly
Flaky test are waste of time, causing unnecessary build failure. Also, they are big waste of developer time, their failure doesn’t indicate a bug, but they do require attention. Due to this entire pipeline of code slows down
This also reduces confidence in the test suite, if these are not fixed in time.
Approach for fixing test flakiness
- Run unstable test separately Test that fails is proving it’s value, unless it’s flaky. Once you discover the flaky test, you still want them to run, remove them from CI Pipeline and execute them in separate jobs to avoid the disruption, investigate it as a part of paying down technical debt. It should be investigated to see if it is providing any actual value.
- Test in Isolation Writing focused test which do single thing is one of the effective ways to avoid them being flaky. Test which do multiple thing increases the chances of failure and can make the test non deterministic and testing features and issue in isolation helps.
- Write Helpful Errors Writing the custom error will help in reporting identifying the issues quickly and help in segregating the flaky test failure from actual failure. ( definitely for long term identification)
- Control the Environment Regularly do test run on controlled environment, which will be same for current test and future test. This ensures test always start with same conditions. This also ensure having the good CI process and able to recreate the environment from scratch when required which is good development process.
- Fixing the Async Wait Many Flaky test I came across were using sleep statement to do waiting such kind of flaky test were fixed replacing the sleep method with waitFor condition.