Measuring the value of test automation

Many automation projects fail due to unrealistic expectations.

To avoid this you shoul identify expectations early and communicate them to the entire team.

Reaping the benefits of test automation does not happen overnight and it really helps if everyone understands the end goal and how to measure the progress towards it.

Benefits of test automation and how to measure them

Regression testing execution time

One key benefit of test automation is the reduced time it takes to execute regression testing.

Before starting your test automation initiative take note of how long it takes to execute regression manually.

As your test automation matures you should see that regression testing takes a lot less time to run however when you first begin automating tests you won’t have much covered and there will be quite a bit of time spent learning the hard lessons of automating tests properly and updating your suite to be more robust.

Know that this will eventually pay off as you see hundreds of regression tests run in minutes as opposed to weeks so ensure that the addition of tests is not necessarily added to the overall execution time.

You can even set an SLA for the max amount of time the entire suite of automated tests can take to execute this will force your engineers to explore smarter options to make tests faster and give you a measurement that you can track another return on your investment.

Fast and Frequent Feedback

Through fast and frequent feedback ,wich you can harvest from automating your regression tests, the team is able to receive feedback as often as needed. The improvement of this is something you can track as part of the progress of your initiative, perhaps you’re only able to run the tesSt and gather results once every few days initially, because of the manual intervention required.

Eventually you should be able to run the test automatically any time a developer checks in code; the faster the feedback the less expensive it is to resolve any issues found.

Trust over tests

This is indeed a great return on investment monitor, your team’s level of trust in the automated test:

Do they believe that a failed test actually means that there is a problem?

It is a question that you can survey your team on during regular retrospectives.

This is an important return on investment because it allows your developers to move a lot faster knowing that they have the test to keep them from messing up too badly.

This improved health of the suite also means you should see a reduction in the time spent triaging brittle tests.

Test every environment

The ability to scale automated tests to run across different environments, browsers and devices it’s certainly a valuable return on investment. If this is something that is important to your business strategy, track the ability of your tests to do this.

This may be painful at first and you may only be able to tackle one of these at a time but it’s certainly something you can measure the progress toward.