Introducing Applied Intuition’s V&V Handbook
Safely developing and deploying autonomous systems—such as advanced driver-assistance systems (ADAS) and automated driving systems (ADS)—is a challenging task. Autonomy programs need to define and follow a rigorous verification & validation (V&V) process to ensure the safety of the systems they develop. Unfortunately, autonomous systems are especially complex due to the unbounded conditions they need to handle safely in their operational space. On top of that, there is limited regulatory guidance surrounding autonomous system safety and how programs can work towards safe commercial deployment.
As a result of these challenges, some autonomy programs are hesitant to invest in V&V early on in their development, hoping that it will get easier to establish V&V processes once their system is developed further. However, those programs might overlook the fact that foundational development and testing practices are best established jointly. By setting up the right foundations early and incrementally maturing V&V processes over time, teams can develop autonomous systems more efficiently against clearly defined goals, avoid delays, and achieve a safer, more performant end product.
Applied’s V&V handbook provides autonomy programs with principles to consider for their safety framework (“I. Safety Framework Best Practices”) and best practices for setting up their V&V processes (“II. V&V Best Practices”). The Applied Intuition team has leveraged its unique position in the autonomous vehicle industry to create this first edition, and we look forward to engaging in conversations and hearing feedback from the industry.
This blog post is the first in a three-part series that highlights a subset of the many topics included in Applied Intuition’s V&V handbook. The full-length handbook is available for download below.
Stages of V&V: From Early-Stage to Late-Stage Development
The first part of this blog post series explains the V&V efforts that autonomy programs across the industry can follow in different stages of their development. The following table is a shortened version of Figure 3 in Applied Intuition’s V&V handbook. It outlines what early-, mid-, and late-stage autonomy programs might look like across various V&V dimensions such as safety governance, safety case, release validation process, requirements management, test methods, coverage analysis, and performance analysis (Figure 1).
“Early-stage” refers to teams who are just starting their V&V journey. “Mid-stage” refers to teams that have already established some of their V&V processes but are roughly more than two years away from commercial deployment. “Late-stage” refers to teams eyeing commercial deployment within the next one to two years.
As seen in Figure 1, early-stage autonomy programs do not typically have a formal safety governance structure or a formal release validation process yet. They usually, however, already define their target operational design domain (ODD), define their end use case, and set up minimal requirements tracking in spreadsheets or documents. In terms of test environments and coverage, autonomy programs in earlier stages often heavily rely on closed test track testing while also utilizing model-in-the-loop (MIL), hardware-in-the-loop (HIL), and potentially software-in-the-loop (SIL) testing. They might track coverage lightly by the number of overall tests. Mid-stage programs use a mix of MIL, SIL, and test track testing while ramping up HIL, vehicle integration, and real-world testing. These programs measure coverage as the number of tests split by scenario category. Late-stage programs use all test environments and execute the majority of tests in simulation to save cost and scale more effectively. They pursue rigorous requirements coverage, scenario parameter space coverage, map coverage, and statistical ODD coverage.
Our V&V handbook lays out best practices around these and other V&V aspects that autonomy programs can follow during early-, mid-, and late-stage development to accelerate their progress beyond the status quo.
Conclusion
By following V&V best practices specific to their maturity stage, programs can build out strong V&V processes over time and safely develop, test, and deploy autonomous systems for commercialization. In the next part of this blog post series, we will summarize the handbook’s key insights regarding scenario creation and test execution. Stay tuned for the next blog post, or download our handbook today to gain access to all its topics, including safety framework best practices, the V&V lifecycle, requirements management & traceability, and analytics & reporting.
Contact our engineering team if you have questions about this handbook or would like to learn more about Applied Intuition’s V&V platform Validation Toolset*.
*Note: Validation Toolset was previously known as Basis.
.webp)
.webp)
