About

Defect4ML is a benchmark of bugs, faultload benchmark, in Machine Learning (ML)-based systems. Defect4ML consists of bugs in two of the most popular ML frameworks, including Keras and Tensorflow. But we are working on it to add bugs from other widespread ML frameworks (e.g., PyTorch). Defect4ML has been started by Mohammad Mehdi Morovati, Amin Nikanajam, and Foutse Khomh from the SWAT lab of Polytechnique Montreal. All researchers and practitioners of ML-based systems are welcome to collaborate on extending, improving, and maintaining Defect4ML.