Test to Pass vs. Test to Fail: Which is Best?

When you’re testing your app or website, you want to ensure that everything is in working order. In order to do this, your QA testers will deploy several methodologies, including test to pass and test to fail. But what, exactly, is the difference between the two? Read on to learn more.

What is testing to pass?

Testing to pass is an approach to QA whereby an application or website is tested in lock-step using specific user-flow instructions. 

An example might be: Press button A, then press button B, then press the Submit button. Testing to pass is typically used when an application or a website is either in its proof of concept stage or is in its infancy and is so fragile that any deviation from controlled steps is likely to produce a fatal error.

What is testing to fail?

Testing to fail involves testing a feature in every conceivable way possible. 

Staying with the above example, a tester might click button A or button B twice before clicking Submit. He may click them out of order, click one or the other several times, or just go right for the Submit button without clicking either of the first two buttons.

Which one should you use?

While testing to pass can be useful in some circumstances, we at Beta Breakers highly recommend relying on testing to fail.

Once an application or a website has evolved beyond the initial proof of concept phase, it should be tested to fail, and aggressively. 

Why? Because beyond that point, testing to pass is misleading at best. At worst, it’s dishonest. Too often we’ve heard a dev ask us, “But, why would you even do that with that feature?” And every time we tell them, “The better question is: Why did you let me?” It is the role of QA to force clarity, and we do that best by testing to fail.

Testing to fail entails pushing a product to its limits by running it through a range of demanding scenarios. When QA testing is conducted in this manner, failure can become a kind of success. Once the software fails, the root cause of that failure can be investigated, identified and eliminated, thereby preventing it from occurring in the software’s release version.

What can you accomplish with test to fail?

This method of testing can be used to achieve a number of different goals.

It’s particularly ideal for performance and stress testing, for example, but it can also be used in localization testing to identify obscure characters in other languages that might have unforeseen effects on an application’s behavior. Testing to fail is not one type of test; instead, it’s a holistic approach to QA designed to rule out all possible scenarios that might lead to product failure.

In many cases, testing to pass becomes the go-to approach for testers who are trying to avoid confrontation, or please their project managers and software developers. This practice often results in poorly-implemented code. Testing to fail is harder and more time consuming. It asks tough questions that might otherwise be ignored or glossed over. It highlights errors and can frustrate people, but it is essential in producing a quality product. It is not, in itself, adversarial, but it must be rigorous in its scrutiny.

Why use test to fail?

It is far better to break the website or application in a controlled staging environment than it is to have a critical issue discovered by the end user once that app or site has gone live. 

In the staging environment, the issue can be clearly observed; all of the pertinent data about the issue can be noted and taken into consideration to help resolve the issue. A bug found after the app or the site has been released into the wild is rarely reported with all of the details that are necessary to fix the issue. This can result in costly work hours spent tracking down that data, and have a negative impact on your product’s reputation with users. And as we all know, it’s difficult to put a price on lost credibility.

Testing to fail is more expensive than testing to pass, but by front-loading the cost of QA, the follow up and maintenance costs of the product are significantly lower. In the long term, testing to fail pays for itself, provides a higher quality product, results in a positive end user experience, and ultimately makes you and your company look better in the eyes of your users. Keep in mind that in 2017 alone, software failures cost businesses an estimated $1.7 trillion in financial losses.

Are you looking for a QA company that will test your product to fail? You’ve come to the right place! Here at Beta Breakers we have more than 25 years of experience in product testing and a testing lab that provides the support we need for success. 

For more information on our testing process, please give us a call at 415-878-2990.