False-Negative is a case where your test result comes out to be positive, but in actuality, an issue is present in the system and the functionality is not working as it should. And Vice-versa is False Positive, which is a case where the test execution results show an error, even though everything is working as intended. Fundamentally, both these have always posed a challenge for web automated testing but I guess it’s ok to say that a False-negative is more hurting than a False-Positive, as the former creates a false sense of security (a lie) and would definitely end up costing much more to us.
I agree that a False-Positive consumes much of our time and worth. On average 70% of automated test case failures are False-Positives due to which testers spend an average of 1/3rd of their time analyzing, correcting, and reporting results that actually should NOT need any attention at all. In fact, with CI/CD implementation, running automated tests every night or after every commit to the master branch, time wasted by testers to triage these false positives easily outweighs the quicker test results provided by our automation suite.
Do not trust every tool that you use blindly, here are a few ways through which we can handle false Positives/false negatives:
- Put solid assertions, cover both +ve as well as -ve assertions. While the creation of your tests, make sure your tests fail when they should.
- Revisit your tests regularly. Peer-review them regularly, not only at the time of the creation of your test suite.
- Remove tests from your suite if they no longer serve a purpose. I know it’s debatable.
- Review your Test report properly.
- Go for both manual and automation testing, so you get more confidence that your system has no False-Positives/False-Negatives.
- Dynamic synchronization of objects.
- Use correct test data and revisit it regularly. Don’t always go with independent or pre-set test data- your application test data should be in line with your test strategy.
What best practices do you follow to prevent and detect false positives and false negatives in your test automation?