To put this another way, auditors don’t care just about test results. They want to see an end-to-end journey that covers everything from test design, to execution, to interpretation, to maintenance, and to updates. The ability to prove that the organization meets its obligations across all of these stages is essential for demonstrating true compliance.
But for a business that runs hundreds or thousands of automated tests (as many now do to keep pace with burgeoning codebases that they update on a daily basis), manually generating test evidence isn’t practical. Hence there is a need to augment test automation frameworks with orchestration platforms that include the ability to collect and document governance and compliance data across the entire testing lifecycle.
What auditors actually want is to know what the tests checked for, why they checked for it, who approved the tests and who maintains them to ensure they remain up-to-date with changing governance and compliance rules. Without this context, auditors’ ability to judge compliance and governance performance based on test results is very limited.
An automated test can show that a control worked under a specific set of conditions. On its own, it rarely shows why the test exists, which requirement it maps to, who approved it, whether it still matters as the application changes, or how failures and exceptions are handled.
Test Evidence vs. Test Activity: What Auditors Actually Want From Your Automation
In this article, James Carpenter, VP of Product at Leapwork, looks at what auditors actually want from automated testing, and why organizations need to treat the evidence trail around their tests as seriously as the tests themselves.
How can you prove to auditors that your organization’s software conforms with security, governance and privacy requirements?
Businesses that develop software typically need to demonstrate that their applications meet governance and compliance requirements. They may need to be able to show, for example, that an application reliably anonymizes sensitive user data before writing it to a database, or that it validates user input to prevent injection attacks (a type of cybersecurity risk in which threat attackers inject malicious commands into application input fields).
The role of software testing in audits
According to a PwC survey, 85% of business leaders reported that compliance has grown more complex in recent years. In a business environment like this, running automated tests may seem like a simple, efficient solution for proving adherence to compliance and other requirements that involve software.
Historically, solutions like this weren’t essential for most organizations because compliance was simpler, and tests occurred on a smaller scale. But going forward, all signs point to ever-increasing complexity on the compliance and governance front as more and more mandates come online, and as trends like AI-assisted code generation introduce new types of risks into codebases. The number of tests that businesses run is also bound to grow as more and more tests become automated. Against this backdrop, testing platforms that include governance documentation capabilities fill a vital and growing gap.
A challenge facing teams tasked with managing and reporting on software tests, however, is that test automation frameworks don’t capture all of the evidence described above. They allow engineers to write and execute tests, but they don’t document test design, maintenance roles, approval records and the like.
The gap between automated testing and compliance
In reality, though, running the tests is merely a step toward demonstrating compliance. The reason is that auditors (who are responsible for reviewing compliance evidence and assessing how well an organization meets its obligations) are not satisfied with evidence that tests ran. On its own, evidence of testing does not translate into evidence of effective governance and compliance.
For organizations that run tests on a small scale, this isn’t a problem. They can manually generate the complete set of evidence that auditors care about.
An audit-friendly test automation strategy
For organizations that maintain tens of thousands of lines of code, manually collecting evidence to prove compliance with requirements like these isn’t practical. Instead, they use test automation frameworks, which allow them to write and execute tests that systematically evaluate applications based on whichever criteria test engineers establish. While these frameworks help with validating application functionality, they may not always be sufficient for the scrutiny needed in highly regulated industries.
If you answered, “we test it and show auditors the results,” you may be missing the mark when it comes to the relationship between software testing and audits. On their own, test results are evidence of little more than the fact that you ran tests.
Going beyond test automation
That gap between test activity and meaningful test evidence matters more as compliance requirements grow more complex and development environments grow more automated.
As software development becomes more automated, organizations are generating more testing data than ever. For security, governance, and compliance teams, that volume is not the real problem. The problem is proving what the tests actually mean.
After all, the job of auditors, in a sense, is to doubt that an organization meets its obligations until it proves otherwise. The more holistically and efficiently businesses can leverage automated software tests and orchestration tools to present detailed evidence of compliance, the better positioned they are to turn those doubters into believers.
True alignment between software testing strategies and audit-friendly operations requires a more detailed and nuanced approach to leveraging the evidence produced by tests. The sooner businesses step up to this challenge, the better positioned they are to thrive in an era when audits are becoming increasingly complex.
To explain why, let’s first step back a bit and take stock of the role that software testing plays in modern audits.
