Kill Them Faster: Speeding up Mutation Testing Through Dynamic Timeouts and Adaptive Test Execution Ordering
Authors
Krogh, Esben Baadsgaard ; Würtz, Andreas
Term
4. term
Education
Publication year
2026
Pages
14
Abstract
Mutation testing inserts small, artificial changes (mutants) into the code to check whether tests catch them. It provides a more precise measure of test quality than traditional coverage metrics, but it is time-consuming because many mutants must be generated and executed. This is especially challenging in Continuous Integration (CI), where fast feedback is crucial. This project investigates how to reduce runtime using dynamic timeout mechanisms and adaptive test prioritization strategies without changing the results (mutation score). The work builds on MUT4CI, a tool for mutation testing in CI. We develop three dynamic timeout strategies that adjust timeout values based on observed runtimes from previous test runs and compensate for extra overhead during parallel execution. The goal is to avoid waiting unnecessarily for mutants that will not finish in a reasonable time. We also design and study nine test prioritization strategies based on static, dynamic, and sampling-based criteria. These strategies aim to run tests first that are most likely to kill a mutant early, and we evaluate sampling approaches where only a subset of tests is executed to balance time savings against the accuracy of the mutation score (the share of mutants the tests detect). We evaluate our methods on three open-source C++ projects: Google Test, Cppcheck, and CMake. Results show that timeout handling has the largest impact. In particular, a lightweight timeout strategy with compensation for the number of parallel threads reduced runtime significantly without harming the mutation score: 5.35x faster for Google Test, 2.13x for Cppcheck, and 1.23x for CMake compared to our ground truth. Test prioritization provided additional but smaller, project-dependent gains; the largest improvements appeared in projects with slower test suites. Sampling strategies that favor fast tests can cut runtime substantially with only moderate reductions in mutation score accuracy. Overall, mutation testing can be made substantially faster by optimizing timeout handling and test execution. Dynamic timeouts can markedly reduce execution time without degrading the mutation score, and adaptive test prioritization can add further gains depending on project characteristics, making the approach more scalable and practical in environments that require rapid feedback.
Mutationstestning indsætter små, kunstige fejl (mutanter) i koden for at se, om tests opdager dem. Det giver et mere præcist mål for testkvalitet end traditionelle dækningsmål, men er tidskrævende, fordi mange mutanter skal genereres og køres. Det er særligt et problem i Continuous Integration (CI), hvor hurtig feedback er vigtig. I dette projekt undersøger vi, hvordan eksekveringstiden kan reduceres med dynamiske timeout-mekanismer og adaptive testprioriteringsstrategier uden at ændre resultaterne (mutation score). Arbejdet bygger på MUT4CI, et værktøj til mutationstestning i CI. Vi udvikler tre dynamiske timeout-strategier, som justerer timeout-værdier ud fra observerede køretider fra tidligere testkørsler og kompenserer for ekstra overhead ved parallel eksekvering. Formålet er at undgå at vente unødigt på mutanter, der ikke afsluttes i rimelig tid. Derudover udvikler og undersøger vi ni testprioriteringsstrategier baseret på statiske, dynamiske og sampling-baserede kriterier. Strategierne forsøger at køre de tests først, der mest sandsynligt kan dræbe en mutant tidligt, og vi evaluerer også sampling, hvor kun et udsnit af testene køres, for at afveje tidsbesparelse mod nøjagtigheden af mutation scoren (andelen af mutanter, som testene afslører). Vi evaluerer metoderne på tre open source C++-projekter: Google Test, Cppcheck og CMake. Resultaterne viser, at timeout-håndtering har størst effekt. Særligt en letvægts timeout-strategi med kompensation for antal parallelle tråde reducerede køretiden markant uden at påvirke mutation scoren negativt: 5,35x hurtigere for Google Test, 2,13x for Cppcheck og 1,23x for CMake sammenlignet med vores grundsandhed. Testprioritering gav yderligere, men mindre og projektafhængige, forbedringer; de største gevinster sås i projekter med langsomme testsuites. Sampling-strategier, der prioriterer hurtige tests, kunne skære tiden betydeligt med kun moderate fald i mutation score-nøjagtighed. Samlet set kan mutationstestning gøres væsentligt hurtigere gennem optimeret timeout-håndtering og testeksekvering. Dynamiske timeouts kan sænke eksekveringstiden uden at forringe mutation scoren, og adaptive testprioriteringer kan give ekstra gevinster afhængigt af projektets karakteristika, hvilket gør metoden mere skalerbar og anvendelig i miljøer med behov for hurtig feedback.
[This apstract has been rewritten with the help of AI based on the project's original abstract]
Keywords