Differential Test

Differential Test

What is differential testing?

Foundry Book

Differential testing cross references multiple implementations of the same function by comparing each one's output. Imagine we have a function specification F(X), and two implementations of that specification: f1(X) and f2(X). We expect f1(x) == f2(x) for all x that exist in an appropriate input space. If f1(x) != f2(x), we know that at least one function is incorrectly implementing F(X). This process of testing for equality and identifying discrepancies is the core of differential testing.

Differential fuzzing is an extension of differential testing. Differential fuzzing programatically generates many values of x to find discrepancies and edge cases that manually chosen inputs might not reveal.

Example: Exp.sol and exp.py

We are testing the correctness of Exp.sol against an implementation exp.py that we trust:

https://github.com/t4sk/hello-foundry/blob/main/src/Exp.sol

https://github.com/t4sk/hello-foundry/blob/main/exp.py

We need the eth_abi package:

pip3 install eth_abi

Writing the differential test:

pragma solidity ^0.8.18;

import "forge-std/Test.sol";
import "forge-std/console.sol";
import {exp} from "../src/Exp.sol";
import {Strings} from "@openzeppelin/contracts/utils/Strings.sol";

// FOUNDRY_FUZZ_RUNS=100 forge test --match-path test/DifferentialTest.t.sol --ffi -vvv

contract DifferentialTest is Test {
    using Strings for uint256;

    function setUp() public {}

    function ffi_exp(int128 x) private returns (int128) {
        require(x >= 0, "x < 0");

        string[] memory inputs = new string[](3);
        inputs[0] = "python";
        inputs[1] = "exp.py";
        inputs[2] = uint256(int256(x)).toString();

        bytes memory res = vm.ffi(inputs);
        // console.log(string(res));

        int128 y = abi.decode(res, (int128));

        return y;
    }

    function test_exp(int128 x) public {
        // 2**64 = 1 (64.64 bit number)
        vm.assume(x >= 2 ** 64);
        vm.assume(x <= 20 * 2 ** 64);

        int128 y0 = ffi_exp(x);
        int128 y1 = exp(x);

        // Check |y0 - y1| <= 1
        uint256 DELTA = 2 ** 64;
        assertApproxEqAbs(uint256(int256(y0)), uint256(int256(y1)), DELTA);
    }
}

The test contains two functions. In ffi_exp(), we are collecting the output of exp.py via FFI:

    function ffi_exp(int128 x) private returns (int128) {
        require(x >= 0, "x < 0");

        string[] memory inputs = new string[](3);
        inputs[0] = "python";
        inputs[1] = "exp.py";
        inputs[2] = uint256(int256(x)).toString();

        bytes memory res = vm.ffi(inputs);
        // console.log(string(res));

        int128 y = abi.decode(res, (int128));

        return y;
    }