[Me]: - I am currently running automation in my project. What can we do to measure the effectiveness of automation?
[Guruji]: - It is no doubt that the role of test automation has become more and more important in keeping the quality of any product at the top level. We have seen companies invest a lot in test automation and seen the result where we can automatically kick off a bunch of test cases with minimum user intervention. All of these achievements are great, but we still need a way to
Measure the successfulness of our automation
Track and take the automation to next level
[Me]: Guruji, I think if we fulfill - Reliability(Successful test automation needs to deliver consistent and reliable results every time test automation is being run), Effectiveness & performance(how long it will take for the test automation to run) of the automation, we can add them as accountable to the successfulness, isn't it?
[Guruji]: - Yes, but you would need to collect a few metrics which will spell out these factors
How many times the automated test case has been run?How many times did each of them pass?
How many times the automated test case is resolved as passed due to test issue?
How many times the automated test case is resolved as passed due to environment issue?
How many times the automated test case is resolved as failed due to product bug?
How many product bugs has the automated test case found?
How many product bugs has the automated test case found?
How long is the duration for the test case to run?
[Me]: - Sounds Cool, So all we need is some kind of index to capture this?
[Guruji]: Yes, and we can call that Automation health Index. For your ease , i have created such an index. You can share it with others.