What To Automate

Have you ever got the question "What tests should we automate?". Test automation is great, because it saves time, but when done wrong, it can be very time consuming and stressful for a whole project. Before you even thing about automation, you should have a test plan for your project with all requirements and all the related test cases. New test cases will be added when the project changes or new requirements will be added. In agile development this will happen quite often and you have to retest all parts of the application, to find whether the changes or addings do not break the application. These regression tests should be considered to be automated first. But is the test case worth automating? And how to find out?

Continuous delivery is one of the driving forces of test automation. But you still can´t automate 100% of your test cases, you should be aware of this fact. To ensure the quality of an application, you need human input.

Before automating a test case, you need to find out the benefits from automating. This includes factors like time benefits, what effort you need to automate the tests and what resources are necessary.

Features Used Every Day

Being a tester does not mean to find as many bugs as possible. As a tester you must know how a user uses the application and what features are used on a daily basis. This is where you should look for automation first. Does a change break one of these daily used features means, the users can´t use the application. So these features must be tested after every build to make sure they still work. Manual tests will be here time consuming, so try to automate these first.

Every application has features, which are "never" used. You may test them as well, but maybe only when these features change and not after every build. Other features are business critical and can cause a hugh damage to your business, therefore they need to be run very often.

Test Cases Having Huge Data Sets

Test cases may be very time consuming when run manually, especially when they are data-driven and have many data sets, which must be checked and verified. First make sure, that these test cases may be automated with time efforts which are worth to automate them. A good idea is to verify, how long you need to run the test with all data sets manually. Ask an experienced Test Automation Engineer for an estimation, how long it would take to automate the test case. If the time to do that is less than running the test 3 to 5 times manually, you should go for automation.

High Coverage needed

These are data-driven tests, but test case which must be run against different browsers, operating systems or even databases. When your application supports all major browsers and their different versions, you need to run the test cases against at least the most important ones. When your application supports different operating systems and theit different versions, you need to run them on as many OS and versions as possible to reach a high coverage. These test cases are also worth automating, as you need to run one test case 10 or 20 times.

Features Which Will Change Very Often

Every application has feature, which are in a very early stage and not really stable. The chance, that these feature will change very often before becoming stable is very high. So it is not a good idea to automate the test cases for these features, as maintaining the test cases will probably consume more time then running these tests manually.

Basic Smoke Tests

Usually every application has some basic smoke tests which are executed after every build and before the final release. These test cases prove, whether the application isn´t broken and still work like it should. As these test cases needs to be run very often, manual tests will be too time consuming, so you should look for automation.

Unit Tests

Automation of unit tests is always a good idea. These tests cases are very developer centric, often created and maintained by the developers, who coded this specific feature. Unit tests are always white box tests checking limited features, so usually automating these test cases is an easy task.

Performance, Stress and Load Tests

When checking non functional requirements like "Does the website finish loading under 1 second" or "The application must be usable with 1000 users logged in", then there is no way to check this manually. Performance and load testing tools are the right choice for these kind of tests and these test cases are always automated then. Nothing to add here, as any manual testing here would consume much more effort and also resources.

User Acceptance Test

You can never ever automate an user acceptance test. This kind of test cases always needs an user, who can decide, whether the application can be accepted from a pure user perspective. Also usability tests and user experience test do fall in this category of testing.

Sanity Tests

Sanity testing is about checking whether an application is up and running. This maybe tests like: Is the webpage available? Can the database be connected? Can some basic transactions be performed? Sometimes these kind of tests are named: "Start of Day Checking".

Exploratory and Ad-hoc Test Cases

What are exploratory tests? These are test cases, which do not have specific test steps. An user tries here to find different and unusual ways to navigate through the application. This can´t be automated. Also Ad-hoc tests, without a script are not a candidate for automation, as in automation you need a script, which can be repeated many times. Ad-hoc means this is an one-timer.