The goal of Oztech Inc. MoVal approach is to perform model-based validation of autonomous vehicle systems for predefined risky scenarios.
Motivation for analysis setup
The approach is to decompose the expected operation of the system into a functional hierarchy, that allows to perform the analysis in three stages: unit testing, integration testing, and scenario testing.
The method has been divided in some stages:
1. Scene description
This stage describes the scenario that will be studied. The use case requirements and assumptions are defined along with the operational design domain and the top level hazard.
The use case requirements, assumptions and top-level hazard are defined following the guidelines of standard HARA (ISO 26262-3) Hazard Analysis and Risk Assessment. The purpose of HARA is to identify faulty functions that could lead to system hazards and to assess the risks they pose. It also allows us to find and define the top risk event (understand the safety goal).
We also consider the standard SOTIF (ISO/PAS 21448) Safety of the Intended Function, which is concerned with ensuring safety when there is no fault in the system, thus reducing potential unknown and unsafe conditions.
Then a Preliminary Hazard Analysis (PHA) is used to reduce the number of variables and operational design domain specifications of the scenario. This analysis allows to identify potential hazards and accidental events that could lead to an accident. This information is used to constraint the operational design domain specification.
A Hybrid State System is then developed to describe the intended operation of the system and includes the HARA and SOTIF considerations of operation.
2. Testing
The principles of Functional Hazard Analysis (FHA) and some ideas from the military standard MIL-STD-882E are considered for this second stage. FHA is a complete examination of the system functions to identify and classify failure conditions. It is used for the Functional Hierarchical Decomposition of the system, a method introduced by L. Acar and U. Ozguner, by which a system is decomposed into a series of functions and associating knowledge-rich controllers to each functionality.
Reference: L. Acar and U. Ozguner, "Design of knowledge-rich hierarchical controllers for large functional systems," in IEEE Transactions on Systems, Man, and Cybernetics, vol. 20, no. 4, pp. 791-803, July-Aug. 1990, doi: 10.1109/21.105079.
With the system expanded into a functional hierarchy, it is possible to divide testing into stages. Unit Testing is used to analyze individual functionalities, like the capacity of braking/acceleration for a vehicle in general. Integration Testing is used to analyze two or more individual functionalities together, like for example, testing the cruising/tracking capabilities of a convoy of vehicles . Finally, Scenario Testing allows to test the system as a whole.
Since doing Scenario Testing can become computationally expensive, we recur to Integration Testing to analyze specific dangerous situations for the system. The purpose is to then group the analysis of all these tests to have comprehensive Scenario Testing results.
Since the dynamics of scenario configuration may be unknown or too complicated, a simulation-based approach is used to obtain a probabilistic cell to cell mapping of the states of the system. This technique allows to obtain a probability distribution for the state evolution of the system in one timestep. Under different configurations of the system, we obtain a certain number of different mappings that will be used for analysis. For more details on this technique, please see A Quantitative Risk Based Framework for UAS Control System Assurance.
Reference: Hejase, M., Kurt, A., Aldemir, T., Ozguner, U., Guarro, S., Yau, M. K., & Knudson, M. (2017). A quantitative and risk-based framework for UAS control system assurance. In AIAA Information Systems-AIAA Infotech@ Aerospace (p. 0882).
3. Validation
After obtaining the probabilistic cell-to-cell maps in the second stage, the validation is performed using the Backtracking Process Algorithm, a Dynamic Probabilistic Risk Assessment method that allows to provide a framework to track risk while considering epistemic and aleatory uncertainties in physical processes and system safety responses.
After defining the threshold probability of interest and the search depth, this algorithm allows to track down from the top hazard event definition, all the possible initial states that lead to the top hazard. Then, the process is repeated until the desired search depth is reached.
4. Results
As results from the Validation part, we are able to identify sequences of actions that consider the changes in the system dynamics and operational modes that might not be obvious nor easy to capture.
This provides useful information regarding combination of initial states that turn out hazardous after a certain time T. From there, we are able to extract operational characteristics/constraints for the modules involved in the system operation, which can include sensor sensitivity, tolerable communication delays, control tolerance, among others.
After this is done, we can go back to stage 2 and refine the simulator, i.e. use a higher fidelity simulator, and focus on the most dangerous cases found in the first iteration. This allows to analyze the unknown unknowns that could not be captured by the first coarse analysis.
Cases of study
Occluded pedestrian scenario
The MoVal methodology is demonstrated in a scenario with an occluded pedestrian crossing the road. We are able to identify risks associated with the actor classification problem and sudden changes in the behavior of the pedestrian.
Reference:M. Hejase, M. Barbier, Ü. Özgüner, J. Ibanez-Guzman and T. Acarman, "A Validation Methodology for the Minimization of Unknown Unknowns in Autonomous Vehicle Systems," Accepted to 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, USA, 2020.
Traffic light at intersection scenario
The methodology is tested within a scenario with an intersection with a traffic light. The Top Event is considered as the vehicle's failure to stop within the required position during a red light.
Reference: M. Hejase, M. Barbier, Ü. Özgüner, and J. Ibanez-Guzman, "A Methodology for Model-Based Validation of Autonomous Vehicle Systems," Accepted to 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, USA, 2020.
Two-truck system (under study)
We consider a two-truck system that travels together at highway speed. The leader truck and the follower truck maintain a short gap as long as the communication link is active. We study the different situations that can provoke crashes using the Integration Testing from our approach.
