Incremental mutation testing in the Bitcoin Core

In their research, Jain et al. introduced an innovative concept known as the oracle gap, which serves to highlight the discrepancy between a test suite's coverage and its mutation score for a given file. This gap is quantitatively defined as the difference between the coverage (cov) of a file by its test suite (T) and the mutation score (mut) achieved, expressed as gap(f,T) = cov(f,T) - mut(f,T). To offer a more nuanced view, they further delineate the covered oracle gap, which focuses on the mutation score but only within the ambit of lines that are actually covered by tests. This is represented as gap_cov(f,T) = cov(f,T) - mut_cov(f,T), aiming to provide deeper insights into the effectiveness of testing efforts.

Their exploration into the realm of software testing revealed a particularly intriguing finding in the context of Bitcoin Core. Despite the application of fuzzing techniques, which are designed to detect bugs or vulnerabilities by inputting massive amounts of random data to the system, there was only a marginal 2% increase in the mutation score. This insight suggests that while fuzzing is a valuable tool in the arsenal of software testing methodologies, its contribution to enhancing the mutation score, and by extension potentially uncovering more subtle bugs or weaknesses, might be less significant than previously presumed.

These findings were discussed in detail across two notable publications. The first, presented at the International Symposium on Software Reliability Engineering (ISSRE) in 2023, laid the groundwork for understanding the oracle gap and its implications for testing efforts. The subsequent study, shared at the International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) in 2022, specifically examined the application and impact of fuzzing on Bitcoin Core, shedding light on areas where this approach may fall short and prompting a reevaluation of strategies to identify and address lacunae in software testing practices.