SiMOOD: Evolutionary Testing Simulation with Out-Of-Distribution Images - LAAS - Laboratoire d'Analyse et d'Architecture des Systèmes Access content directly
Conference Papers Year : 2022

SiMOOD: Evolutionary Testing Simulation with Out-Of-Distribution Images


Testing perception functions for safety-critical autonomous systems is a crucial task. The reason is that accurate ML models applied in computer vision tasks still fail in scenarios where humans perform well. Out-of-distribution (OOD) images are usually a source of such failures. For this reason, literature usually applies data augmentation techniques or runtime monitors such as OOD detectors to increase robustness. Evaluating such solutions is usually performed by analyzing metrics based on positive and negative rates over a dataset containing several perturbations. However, using such metrics on such datasets can be misleading since not all OOD data lead to failures in the perception system. Hence, testing a perception system cannot be reduced to measuring Machine Learning (ML) performances on a dataset but rely on the images captured by the system at runtime. However, the amount of time spent to generate diverse test cases during a simulation of perception components can grow quickly since it is a combinatorial optimization problem. Aiming to provide a solution for this challenging task, we present SiMOOD, an evolutionary simulation testing of safety-critical perception systems, which comes integrated into the CARLA simulator. Unlike related works that simulate scenarios that raise failures for control or specific perception problems such as adversarial and novelty, we provide an approach that finds the most relevant OOD perturbations that can lead to hazards in safety-critical perception systems. Moreover, our approach can decrease, at least 10 times, the amount of time to find a set of hazards in safety-critical scenarios such as autonomous emergency braking system simulation. Besides, code is publicly available for use.
Fichier principal
Vignette du fichier
SiMOOD_PRDC2022_FINAL.pdf (1.73 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03779723 , version 1 (17-09-2022)



Raul Sena Ferreira, Joris Guérin, Jérémie Guiochet, Hélène Waeselynck. SiMOOD: Evolutionary Testing Simulation with Out-Of-Distribution Images. 27th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC 2022), Nov 2022, Beijing, China. ⟨10.1109/PRDC55274.2022.00021⟩. ⟨hal-03779723⟩
110 View
66 Download



Gmail Facebook X LinkedIn More