No Money and No Time, but Test Anyway…

We’re often faced with the task of providing software quality assurance for a product that, by the time it has been handed to us, is over budget and behind schedule. Diatribes about the importance of thorough testing fall on deaf ears (often those deaf ears belong to “number crunchers” who are too short-sighted to realize the damage that can be done by releasing buggy software – but that’s a separate topic), and we’re left with little time and little budget with which to work our magic.

As a 3^rd party beta testing house, we don’t have the authority to demand adequate time and resources. Nor do we have the luxury of protesting too much, as we would risk losing the client. We could be overly principled and refuse to put our mark on a product that we aren’t allowed to test to our satisfaction, but what would that accomplish? We’d just alienate our customers, and many of them would simply choose not to do any testing at all, or pass the task off to some in-house employees.  In such a case, our contribution to increasing the quality of software in the world would be zero. Wouldn’t it be better to make the most of the budget and time that we DO have – knowing that we can provide more benefit than a couple marketing people or overworked developers rushing to finish the testing so they can enjoy the rest of their weekend?

The test effort then becomes one of risk management. It is critical in these circumstances that the sparse testing resources be allocated in the most efficient manner and in a way that mitigates as much of the risk as possible.

Automated quality assurance tools should be used only if they can provide useful feedback quickly and without much tinkering. For example, scripting pieces of the main functionality of a web application using an automated tool is very useful as a smoke test when the test cycle is long enough to accommodate several builds.  When time is short, however, even such simple scripting takes too long to create and maintain, and often there won’t be enough builds to justify the effort. On the other hand, automated link checkers can be configured quickly and can provide useful feedback almost immediately without significant effort on the part of the tester.

Another area to look at closely (for websites / web applications) is browser compatibility.  Ideally, you would test backward compatibility for the last 2 – 3 versions of each major browser.  However, focusing on the last two IE versions, plus the latest Firefox, Chrome and Safari versions will cover over 90% of the desktop browser market.  The time and money you save by neglecting the 5% of the market that you miss by not testing on older browsers can and should be used more effectively elsewhere.  It also helps if you have accurate demographics for your product that will allow you to focus testing on the most applicable browsers.

In general, when these situations arise, we remind our clients that some testing is better than no testing, and that even a small amount of properly focused quality assurance by a professional QA firm is better than lots of directionless testing by inexperienced individuals.  We also warn them to be prepared for a large number of significant bugs –projects that are behind schedule and over budget tend to go hand-in-hand with rushed development that usually results in buggy code.  It is not unusual for us to find so many significant bugs in the 2 – 3 days that we are given to test an application that the powers-that-be had no choice but to postpone release.