Metamorphic relations via relaxations: An approach to obtain oracles for action-policy testing

Abstract

Testing is a promising way to gain trust in a learned action policy π, in particular if π is a neural network. A “bug” in this context constitutes undesirable or fatal policy behavior, e.g., satisfying a failure condition. But how do we distinguish whether such behavior is due to bad policy decisions, or whether it is actually unavoidable under the given circumstances? This requires knowledge about optimal solutions, which defeats the scalability of testing. Related problems occur in software testing when the correct program output is not known. Metamorphic testing addresses this issue through metamorphic relations, specifying how a given change to the input should affect the output, thus providing an oracle for the correct output. Yet, how do we obtain such metamorphic relations for action policies? Here, we show that the well explored concept of relaxations in the Artificial Intelligence community can serve this purpose. In …

@inproceedings{eniserMetamorphicRelationsViaRelaxations2022,
 author = {Hasan Ferit Eniser and Timo P. Gros and Valentin Wüstholz and Jörg Hoffmann and Maria Christakis},
 booktitle = {ISSTA 2022: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA-2022), July 18-22},
 doi = {10.1145/3533767.3534392},
 isbn = {978-1-4503-9379-9},
 month = {7},
 pages = {52-63},
 publisher = {ACM, New York, NY, USA},
 title = {Metamorphic relations via relaxations: An approach to obtain oracles for action-policy testing},
 year = {2022}
}