Manual vs Automated Testing

Manual vs automated testing is a topic continuously cycling the tech industry - should you adopt a completely automated testing strategy to ensure that your application is defect free? Or should you manually test only?

The answer to this is - Both.

Agile software development puts a high emphasis on quicker processes and higher quality outputs, but teams may use a variety of approaches to achieve these objectives. Both manual and automated testing strategies are vital in ensuring the long term efficiency and quality of a software application, as well as ensuring effective communication between testers and developers by reducing the feedback cycle. If you devote all of your QA time to only one of these methods, you'll miss out on a lot of opportunities to improve quality, the combination of both manual and automated testing would result in a higher-quality, more reliable application.

Within manual and automated testing there are two categories of testing:

• Functional
• Non-functional
  

Functional testing involves testing the real-world business application of a software product, for example, a food delivery app like Deliveroo must be able to connect end users to the restaurants in their local area.

While non-functional testing involves assessing the applications readiness based on nonfunctional requirements (performance, security, reliability, scalability) that are not covered by functional testing.

In this article we will be going through a generic view of manual and automated testing, and the best practices of implementing both into your testing strategy.

‍

Manual testing in an automated world

Manual testing is a method of testing and validating your application using test cases without the support of coded scripts and tools. For any new application to be automated, it is important that it is first manually reviewed to understand the end to end flow and to determine whether automation is possible. When an application is complete and ready for production, real world users are the ones that interact with it. The way in which a user operates with an application is very different to how an automated script will analyse the application for defects.

Manual testers, with the human eye, are able to detect if a part of the application seems visually incorrect, in a better and more effective way compared to automated test scripts. When a tester interacts with software in the same way as a user does, they will find usability problems and user interface flaws, which are aspects that are challenging to replicate with automated testing scripts alone.

As well as the above, there are several scenarios that are either technically impossible to automate or would be prohibitively expensive - as part of the software development lifecycle, it is important to assess the optimal strategy to follow for your application.

There are several different types of manual testing methods which are beneficial in understanding the software being tested, the end-user and continuous improvement requirements, such as exploratory testing.

While automated tests focus on codifying knowledge we have today, exploratory testing helps us discover and understand stuff we need tomorrow. Markus Gärtner (Gärtner, 2013).

Exploratory testing is a method of manual testing that is all about inquiry, exploration, and understanding. It involves testers exploring the application to find possible edge cases, as part of this, it is critical that testers comprehend the various personas who are likely to use the application. By exploring the software, testers will discover rare and hard to find bugs that were previously unnoticed by automated testing, which follows predefined guidelines. It’s easy to miss something that you might not have been looking for in the first place.

Aside from exploratory testing there are a multitude of manual testing methods that can be implemented into a combined automated testing strategy, for example:

• Smoke testing - A functional testing technique which involves verifying whether a build is stable enough to proceed with further testing, and that all the key features are working as expected.

• Acceptance testing - A functional testing technique performed by users or a user acceptance testing team, to ensure that the software and application has met the requirements specification.

• Usability testing - A non-functional testing technique performed with real users. During a test, real users will typically attempt to complete standard tasks while observers observe, listen, and take notes.

• Negative testing - A functional testing technique which involves checking that the application behaves as expected with invalid or unwanted inputs. The purpose of this technique is to ensure that the application remains stable with negative inputs.

• Cross-browser testing - A non-functional compatibility testing technique used to verify if the application behaves as expected across different browsers and devices.
  

At the start of a project, it is important to have a clear understanding of the business requirements for the application being built, for example:

• Who are the target market users?
• How are they expected to interact with the application?
• How many browser types, devices or screen sizes is the application expected to be compatible and responsive with?
• What happens if I take the incorrect step or input invalid values in the application?
  

This information is required to have a clear understanding of the most applicable and effective manual test methods and to support the development team in understanding the customer and user-like perspective of the application.

Manual testing is a very important process to include in your testing strategy, however, this is not to say that the solution is therefore only to run manual tests.

Manually testing an application is beneficial in adding more coverage to your testing strategy and also understanding how your target market users interact with your application, however, it can require more effort and time on its own for a complete application, hence why automation is key in supercharging your product quality lifecycle.

Automated testing

As highlighted manual testing, such as exploratory and usability is extremely beneficial, however, there are other forms of testing such as regression and functional testing that although can be completed manually, require a significant amount of repetition which is ideal for test automation.

Automated testing is the process of automatically running tests on, for example, repetitive user flows, managing test data and analysing the results in order to enhance software quality.

Test automation success necessitates meticulous preparation and design work. As part of the automated strategy, it is important to define your objective and, before defining the tests to automate and the tools required, to assess the feasibility of automation for the application being built. Without taking this into account, automation can end up increasing overall testing costs instead of saving them.

Below are a few criteria to consider when implementing an automated testing strategy:

• Are there repetitive flows?
• Are there multiple different data sets that need to be verified?
• Does the application need to run on several different configurations and platforms?
• Are there complex functionalities that introduce high risk conditions?
  

The results of a successful automated test must be repeatable. This ensures that if the application being tested does not change, each test run would produce the same results every time. If the application is not stable enough i.e. it is subject to frequent changes or significant unplanned changes throughout the development lifecycle, this can impact your ability to automate.

There are several different types of automated tests, and as previously mentioned, some of which can also be tested manually, these include:

• Unit testing - Also known as component testing, is a kind of functional testing, carried out by the developers, where individual units of code/ components are tested to ensure that it works as designed.

• Integration testing - A functional testing technique, where integrated units of code are verified as a group, to expose potential merge conflicts that could occur.

• Regression testing - A functional testing technique that is used to verify that a new code or feature implemented does not result in any unintended consequences. This usually involves testing the application end to end flow.

• Performance testing - A non-functional testing technique used to measure an application's speed, response time, consistency, durability, scalability, and resource consumption under a specific workload.
  

The right test for your application is very much dependent on your objective, the application you are testing against and the amount of coverage you require. After deciding your test objectives, the types of tests to automate and the types of automated testing to include, you should determine what actions your automated tests will take as well as the best tools to use in the process.

When creating tests it’s important to consider the following:

• When creating tests, keep them simple and based on a single goal, this makes tests more reusable.
• Ensuring that your automated tests are broken down into smaller logical chunks rather than creating much larger tests that test various aspects of the applications behaviour at once. By adding small tests that fix new features, you will be able to maintain your automated tests more frequently and easily. After creating a few automated tests, these can then be combined into a single, larger automated test.
• When testing an application, rather than waiting until an entire feature is introduced, test the functionality of your application as you integrate it. This will allow you to effectively validate sections of the feature and quickly feedback to the developers.
  

These best practices should also be considered when choosing the right automated testing tool that best meets your overall project requirements. Consider these key points when choosing an automated tool:

• What is your allocated budget?
• Do you have enough professional personnel to dedicate to automation tasks?
• What are the systems and technologies that the automated tool should support?
• Is the tool scalable as your application grows?
• What testing types does the tool support?
• Does it integrate well with your existing ecosystem, such as TestRail for test management or JIRA?
  

I recommend the following article which highlights the top 20 best automated tools to include into your strategy https://www.guru99.com/automated-testing-tools.html.

Conclusion

Both manual and automated testing are strategies that are effective in ensuring maximum end to end coverage for your application. By integrating both you have the ability to explore better methods that suit your development lifecycle, your application needs and your business needs.

It should therefore not be a question of whether either should be implemented, but instead of how both can be implemented in the best way possible to maximise efficiency in ensuring product quality throughout the process. No one test fits every scenario, be sure to plan, assess and analyse the best strategy to implement.

‍