Probabilistically Extended Ontologies a basis for systematic testing of ML-based systems

Autonomous driving is a hot topic in the automotive domain, and there is an increasing need to prove its reliability. They use machine learning techniques, which are themselves stochastic techniques based on some kind of statistical inference. The occurrence of incorrect decisions is part of this approach and often not directly related to correctable errors. The quality of the systems is indicated by statistical key figures such as accuracy and precision. Numerous driving tests and simulations in simulators are extensively used to provide evidence. However, the basis of all descriptive statistics is a random selection from a probability space. The difficulty in testing or constructing the training and test data set is that this probability space is usually not well defined. To systematically address this shortcoming, ontologies have been and are being developed to capture the various concepts and properties of the operational design domain. They serve as a basis for the specification of appropriate tests in different approaches. However, in order to make statistical statements about the system, information about the realistic frequency of the inferred test cases is still missing. Related to this problem is the proof of completeness and balance of the training data. While an ontology may be able to check the completeness of the training data, it lacks any information to prove its representativeness. We make a proposal for the extension of ontologies to include probabilistic information. These concepts allow to evaluate the completeness and balance of training sets. Moreover, it serves as a basis for a random sampling of test cases, which allows mathematically sound statistical proofs of the quality of the ML system. We demonstrate our approach by extending published ontologies that capture typical scenarios of autonomous driving systems with probabilistic information.