Housing Technology: Stopping defects from escaping into live operations and disrupting users

Article originally published at Housing Technology magazine – issue May/2024 – Author: Kristen Hansen, Principal Consultant of Acutest

“I never want to pay someone to run that type of test again”— Reducing the cost of software quality

Is anything more important than stopping defects from escaping into live operations and disrupting users? In almost half of organisations, it seems reducing development, testing and bug fixing costs is more important. This insight came from a 2023 survey conducted by IT research advisory firm IDC/ Foundry commissioned by Acutest. Of the 109 organisations surveyed across varied industries and sizes, almost half commit less than 20% of development budgets to testing.

In our previous article we discussed approaches to prevent joining the 93% of organisations that release severe defects into live operations. These include:

• Effective requirements and design reviews
• Leveraging generative AI for streamlined requirement engineering
• Enhancing non-functional coverage to avoid performance issues and downtime

Issues cost more to fix the later they are found

We highlighed over 60% of defects identified during testing stem from issues introduced during requirements specification and solution design. Early reviews find defects almost as soon as they are introduced, significantly reducing the cost of fixing them.  According to Software Engineering: A Practitioner’s Guide by R. S. Pressman, preventing defects at this stage is 15 times more cost-effective than addressing them during testing. Waiting until live operations to uncover a defect is even more expensive—60 to 100 times more costly, according to Pressman.

Early reviews not only enhance documentation quality but also yield substantial cost savings in project and test delivery.

Drive out duplication

If a registered engineer serviced the boiler in a property, you would not reinspect the flue and disassemble the boiler to check the internal components for wear yourself. However, this is what organisations do when they test IT solutions. Running identical tests through the same workflows to obtain the same results consumes expensive quality assurance resource and does not surface defects missed in earlier phases of testing.

For one client, Acutest reviewed a test suite of 400 tests and found more than 50% were essentially duplicates. Though each test used different data, the functional behaviour remained identical. If one test passed, the duplicate tests would inevitably pass as well.

Duplicated tests consume time and cost money throughout development and release cycles. Writing, then repeatedly running duplicate tests through multiple cycles, and analysing the results all contribute to the cost. When multiple functionally identical tests fail, the resulting alarm leads to further resource wastage in fixing what may be a minor issue.

Look for duplication when reviewing requirement and design — collaborative with business users, suppliers and testers to develop user stories. These stories not only avoid any confusion between suppliers and customers but also reveal duplicate functional behaviours that require testing only once.

Do not waste time, save cost

To reduce test costs while preventing serious issues from reaching live operations, organisations must improve test efficiency:

• Write fewer, better tests—perversely, large numbers of tests lead to more duplicates and higher costs. Our cross-client analysis has found that doubling the number of tests triples the duplication rate.  Do not measure test team productivity by the numbers of tests they write as spending time writing more tests does not guarantee better outcomes or coverage.
• Don’t write tests too early—in Acutest’s experience if a test isn’t executed within three months of development it is unlikely to be run at all due to the world changing by the time execution arrives. Our analysis of client projects, shows that for test suites with over 1000 tests typically 30% are never run. Writing tests just before they are needed minimises the risk of obsolesce and ensures their relevance.
• Monitor test team efficiency —track key metrics such as unexecuted tests, time from test scripting to execution, test reuse, and earned confidence. These indicators will alert you to inefficient and money wasting money projects delivering the same level of quality that others may be delivering for less.

Know what is good enough

Achieving “100% of tests completed and passed” is often a key milestone for deployment and just as frequently a source of unnecessary cost. Executing the final 10% of tests may well take as much effort as the preceding 30% and not significantly boost confidence. For instance, in an Acutest review, stakeholders agreed that passing 104 of the 258 tests originally planned would provide sufficient confidence to go live.

Our analysis shows that in test reviews, key stakeholders conclude that many tests either could not fail, or the impact of failure would be so low that it did not justify the expense of running them. Implementing the right risk-based approach means that decisions on going live can be made earlier and confidently, even before testing all testing has completed. This saves considerable cost without significantly compromising the delivered solution’s quality.

Simple changes give big returns

The survey highlighted how important a consideration the cost of assurance is to many organisations, often more so than quality, especially where there is such a small percentage of overall budget available. Significant cost savings can be achieved by internal delivery and test teams making simple but important changes to the way they work.

• Fail fast by removing defects at requirements and design phases rather than when they become much more expensive to fix.
• Avoid incurring the cost of writing and running duplicate tests or tests that will never be run.
• Focus on executing tests that will build confidence and enable decisions to be taken on going live earlier.