6 reasons why automated testing fails

Automated testing is a critical component of any software development process. It not only saves time and money by automating repetitive manual tests, but it also increases the overall quality and reliability of the product. In today's fast-paced world, where software products are expected to be delivered rapidly with high quality, automated testing plays an important role in achieving these goals.

However, despite its importance, automated testing can sometimes fail to deliver the desired results. There could be several reasons for this failure, ranging from poor test design to inadequate test coverage. In this article, we will explore six common reasons why automated testing fails and what you can do to avoid them.

1. You don't follow test pyramid

The first reason why automated testing might fail is you are not following test pyramid.

What is test pyramid?

Test pyramid is a concept which describes how automated testing should be implemented. Pyramid consists of 3 layers:

  • Unit testing
  • Integration testing
  • UI testing

When you want to transition from manual testing to automated testing need to keep in mind test pyramid concept. You should always start with unit testing, then integrations testing and at the end UI or end-to-end testing. In my practice I saw a lot companies do the opposite. They start with end-to-end testing and eventually automated testing fails. The reason behind this decision is so simple, if end-to-end tests pass no need to worry about the rest of it. It might give you perception of hitting 3 rabbits with 1 bullet but in fact you are hitting none of it. Three important points to explain why it happens. First, end-to-end tests are expensive, they are brittle and slow. If I give simple analogy to explain think about solid brick wall and game Jenga.

Solid brick wall is unit and integrations tests.

Jenga tower is end-to-end tests. Now do you think you can rely on tests which look like Jenga tower? Absolutely, not. Few simple design changes and you have to re-write the UI tests and believe me you are going to change UI a lot. Maintenance cost of end-to-end tests are high.

To summarize, end-to-end tests are expensive, brittle and slow. It's the last point you should consider automating.

2. Approach is dictated by someone else

This is also common practice in IT companies where automated testing approach is dictated by business, customers, product owners and developers. Don't get me wrong I'm not saying you should not listen to other members of the company. I strongly believe you should listen and understand their needs as well. At the end of the day QA team is not the only team which will use automated tests, developers also actively use them. My point is you should gather requirements from other teams but decision should be made by you. If you are responsible for automated testing you should make the final decision not someone else. Decision should be made based on specifics of the project, QA team's skillset and ROI.

Two the most common cases, first when business or management wants end-to-end tests before everything. Above I explained what's going to happen if skip unit and integration testing and jump straight to end-to-end testing. Your responsibility as QA Lead you should explain there is an order and end-to-end tests come in the end.

Second, developers want QA team to use programming language which they use in the project. You might hear two common arguments why it's a good idea:

  • Developers can help with end-to-end testing.
  • We might re-use some project related libraries in the future.

First argument is true at the beginning. When you start implementation of automated tests you will see enthusiasm and get help you need. But sooner or later it will fade away because developers have their own primary responsibly which is delivering software. I believe if developers helping QA engineers to write end-to-end tests instead of implementing product features their time is being used inefficiently. Developers writing unit tests is going to be best contribution but leave end-to-end tests to QA team. Regarding re-using some project specific libraries in the future, I have doubt about it. Because usually integration and end-to-end tests exist in isolation from SUT(software under test). Of course, you can re-use some of the libraries or functions, but I highly doubt you are doing integration or end-to-end testing in that case. I assume you are trying to write component test not end-to-end test. For sure it all depends on the project. There are some projects there is not any alternative options rather than reusing SUT libraries but in reality, it's really rare. I prefer to isolate SUT and end-to-end tests.

3. You are not using frameworks

Another common mistake is not to utilize frameworks or tools but instead code from scratch. Usually, plan is to select programming language, spend couple of months to learn it while doing manual testing in parallel then start writing tests. Sounds like a solid plan but in reality, both creating framework and learning new programming language takes longer time. Newly created framework requires maintenance, automated testing's priority is always lower than manual testing, additionally need to write and update test cases. Eventually, you spend x number of months and you have only 3-4 tests which you execute locally. The most important lesson I learned was creating and maintaining new framework is full time job at least for single engineer. In some companies coding from scratch worked pretty well because QA team was very good in programming. In some other cases company hired junior or middle developer who is responsible for creating and maintaining it fulltime.

4. You made decision without proof of concept

Making a decision about automated testing without any proof of concept can have serious consequences. It is important to create a proof of concept to ensure that the chosen approach will work for your project. A proof of concept allows you to test the feasibility of your approach and identify potential issues early on, before investing significant time and resources into development. Skipping this step can lead to wasted time and effort, as well as a failed implementation down the line. A proof of concept should include a detailed plan for implementation, including the scope and timeline of the project. It should also include a testing strategy, outlining the types of tests that will be conducted and how they will be executed. Additionally, it should identify any potential risks or challenges that may arise during the implementation of automated testing. A proof of concept should also include a cost-benefit analysis, evaluating the expected benefits of automated testing against the associated costs. Finally, the proof of concept should include a plan for ongoing maintenance and support, ensuring that automated tests remain up-to-date and reliable over time.

5. Using single technology to solve all problems

This is also common mistake to use single technology to solve all problems. For example, you selected to write automated tests in Java. Does it mean you have to use Java for UI testing? Of course, you can but you have understood there are really good frameworks created with Javascript and such frameworks are going to be way efficient than using Java. When it comes to UI testing, you should choose the technology that is best suited for the job. For example, if you are testing a web application, a JavaScript-based framework like Cypress or TestCafe might be a better choice than Java. These frameworks are specifically designed for UI testing and offer features like automatic waiting, easy-to-use selectors, and built-in assertions. Using the right tool for the job will make your tests more reliable, easier to write, and faster to execute.

6. You don't have enough resources

The last reason why automated testing might fail is lack of resources. Automated testing requires a lot of effort, from writing test cases to maintaining the framework. It is important to have enough resources to ensure that the tests are reliable and up-to-date. The resource requirements depend on the scope of the project and the complexity of the tests. Depending on the scope of the project, additional resources may also be needed to ensure that all tests are executed properly and that all issues are addressed in a timely manner. Finally, it is important to have sufficient resources to allow for ongoing maintenance and support of automated tests. Without adequate resources, automated testing may fail due to unreliable or outdated tests.

Conclusion

In conclusion, automated testing is an essential part of modern software development. However, it requires careful planning and execution to be successful. By following the test pyramid approach, making decisions based on proof of concept, utilizing frameworks and choosing the right technology for the job, you can avoid the common pitfalls that lead to failed implementations. It is also important to involve all stakeholders in the decision-making process but ultimately QA team should make the final decision based on project-specific requirements. Remember that automated testing is a continuous process and requires ongoing maintenance and updates to keep up with changes in the software under test. With these best practices in mind, your automated testing efforts will be more efficient, effective, and reliable.