The increasing inclusion of Deep Learning  (DL) models in safety-critical systems such as autonomous vehicles have led to the development of multiple model-based DL testing techniques. One key limitation of these approaches is the lack of diversity in their generated test cases, which often results in poor performance. Also, they assume that training programs are adequate and bug-free. An assumption that is not always true since DL programs like any software program can contain bugs. In this talk, I will introduce the audience to a TensorFlow-based library that we have developed to help DL engineers detect issues in their training programs.  I will also present DeepEvolution, a novel search-based approach for testing DL models that relies on meta-heuristics to ensure a maximum diversity in generated test cases.

To appear

Increasingly, autonomous systems' behavior is driven my machine learning, and in particular by (deep) neural networks. One crucial question then is how to effectively test such systems given that they don’t have (complete) specifications or even source code corresponding to some of their critical behavior. Further, adding even more complexity to an already difficult problem, many of these autonomous systems are usually integrated into larger systems, thus possibly leading to undesirable feature interactions. This talk will report on recent work done to address these intricate problems and will attempt to better define the challenges ahead. Examples from the automotive domain will be used to illustrate the main points.