Operational Design Domains (ODDs) are crucial for defining the boundaries within which autonomous systems can operate safely. These specifications encompass the environmental, geographic, and temporal conditions under which autonomous systems are designed to function. A rigorous definition of an Operational Design Domain is essential for developing safe and reliable autonomous systems, ensuring compliance with regulatory requirements, and facilitating scenario-based testing.
Phase 1: Defining an Operational Design Domain
A common misconception is that ODDs are static, “one size fits all” documents which are defined comprehensively from the start. This is not the case; they require continuous iteration and refinement to incorporate new data and operational insights. An ODD should expand over time. However, ODDs do not need to be defined from scratch.
Certain industry standards such as ASAM OpenODD, ISO 34503, and Pegasus provide guidelines for general ODD structure. These standards can be very helpful when drafting an ODD for the first time and are widely used across automated driving systems (ADS) OEMs.
ASAM OpenODD aims to create a standardized data model for defining and exchanging ODD information. It focuses on digital interoperability, allowing different systems and tools to communicate ODD details seamlessly. This standard is particularly valuable for the testing, validation, and certification processes, as it provides a consistent format for ODD representation in coverage analysis.
ISO 34503 offers a framework for the classification and description of ODDs specifically for ADS. Unlike ASAM OpenODD, ISO 34503 emphasizes a broader framework rather than data modeling specifics. Out of the three ODD standards mentioned, ISO 34503 is also the most abstract, which may be useful to companies that want a flexible ODD standard.
Finally, Pegasus's 5-Layer ODD Model breaks down ODDs into five distinct layers, offering a detailed conceptual analysis. This model covers functional, system, operational, situational, and behavioral ODD attribute classes, allowing for a thorough exploration of vehicle capabilities and operational contexts. Unlike the other standards, the Pegasus model is more focused on structuring the analysis and safety assurance processes by delineating these different layers of considerations for ADS.
While there are differences among them, all these standards are excellent baseplates for ODD definitions and all are supported by Applied Intuition’s Validation Toolset.
Phase 2: Understanding How ODDs Relate to Requirements
Requirement libraries are a separate concept from ODDs, but the two are often closely intertwined in functional verification and validation (V&V) workflows. Requirements provide a detailed set of situations, conditions, and constraints which ADS must be able to handle without issue. ODDs can elaborate the test cases used to verify these requirements with additional details and classifications. For example, consider the situation described below in a Euro NCAP AEB/LSS VRU Test Protocol (February 2024) requirement:
“7.41. Car-to-Motorcyclist Rear stationary (CMRs)
The CMRs scenario will be performed with 5km/h incremental steps in speed within the complete speed range and with a 50% hit-point as shown in the figure [a diagram showing an ADS with a speed range of 10-60 km/h approaching a stationary motorcyclist] below.”
By using an ODD definition, one can describe a test case which validates the requirement above using ODD attributes. Here’s an arbitrary example of such a test case:
“The ego navigates an urban road with a speed limit of 50 km/h and, after performing a left turn, encounters a stationary motorcyclist.
The ODD taxonomy of this test case would contain “Urban road, 50 km/h speed limit, stationary actor, motorcyclist, left turn”.
By clarifying requirement and test case definitions with ODD attributes, engineers can ensure their requirements library is comprehensive across their ODD.
The relationship between ODDs and requirements also extends to scenario-based testing. By mapping requirements to specific ODD parameters, developers can create a comprehensive testing framework that covers both typical scenarios and edge cases. Thorough testing is vital for ensuring the reliability of automated driving systems, particularly for when they encounter unexpected situations that fall outside the current ODD.
ODD Issues in the Real World
An incident involving a pair of autonomous Cruise vehicles underscored critical shortcomings in their systems’ perception and control stacks, as recounted by SFGate and the San Francisco Chronicle. Severe storms felled two trees on Clay Street in San Francisco, and when they fell the trees tore down power lines over the street. The San Francisco Fire Department roped off the area with caution tape, which was sufficient to reroute most road users.
However, the two Cruise vehicles nevertheless drove into the hazardous area and became entangled in the debris. Their inability to respond properly to the danger-implying environmental cues—the downed wires and caution tape—suggests an insufficiently defined ODD was used in training their autonomous driving system. These vehicles were operating in intense weather conditions with unexpected road features, a scenario that likely exceeded their previously tested operating parameters.
This edge case oversight highlights the necessity of consistently iterating on and drafting of new ODD attributes. The process can help ensure an autonomous system’s robust handling of diverse and unpredictable real-world events, thereby enhancing the overall safety and reliability of the system.
Applied Intuition’s Approach
Applied Intuition’s ADAS and AD development platform features advanced software tools which support comprehensive, version-controlled ODD definitions. In Validation Toolset, users can define an ODD from scratch or import their existing ODDs. The ODD can be used to tag many artifacts in ADP such as test cases, simulations, and drive logs.
ODD tagging can be completed with three sub-processes:
- Manually by a human labeler using any supported ADP product.
- Automatically with deterministic logic rules based on raw simulation data.
- Automatically with latest-generation machine learning and computer vision models based on drive log data (such as camera or lidar).
After artifacts are labeled, Validation Toolset’s analysis tools provide insights into the simulation and real-world test coverage over the ODD. From customizable visuals using scalable dashboards to Python notebooks designed for in-depth data processing, Validation Toolset can deliver reports for any required ODD coverage analysis.
Other products such as Data Explorer are used to manage and analyze log data collected during ADS operations. This enables the correlation of real-world driving data with specific ODD definitions, helping to refine system capabilities and identify gaps in current ODD specifications. For instance, Applied Intuition's triage systems allow for the detailed analysis of disengagements—instances where the ADS encounters an ODD which it was not prepared for and is forced to hand control back to a human operator. Identifying and triaging the cause of these events is crucial for uncovering "unknown unknowns”—scenarios that were not initially considered during system design but were nevertheless encountered in the real world.
Conclusion
Comprehensive ODD definitions are essential for the safe and effective operation of autonomous systems, providing a clear framework for testing and validation. By leveraging advanced tools like Applied Intuition's Validation Toolset and Data Explorer, developers can enhance system reliability and address potential gaps in ODD coverage with ease.
What’s Next?
In the next ODD taxonomy blog post, we will focus on ODD coverage analysis—the process mentioned briefly herein, which answers the question, “When have I tested my automated driving systems enough?”
.webp)
.webp)
