# Differential Test

{% embed url="<https://youtu.be/WhZQhxOG124>" %}
Differential Test
{% endembed %}

## What is differential testing?

{% embed url="<https://book.getfoundry.sh/forge/differential-ffi-testing>" %}

> [Differential testing](https://en.wikipedia.org/wiki/Differential_testing) cross references multiple implementations of the same function by comparing each one's output. Imagine we have a function specification `F(X)`, and two implementations of that specification: `f1(X)` and `f2(X)`. We expect `f1(x) == f2(x)` for all x that exist in an appropriate input space. If `f1(x) != f2(x)`, we know that at least one function is incorrectly implementing `F(X)`. This process of testing for equality and identifying discrepancies is the core of differential testing.
>
> Differential fuzzing is an extension of differential testing. Differential fuzzing programatically generates many values of `x` to find discrepancies and edge cases that manually chosen inputs might not reveal.

## Example: Exp.sol and exp.py

We are testing the correctness of Exp.sol against an implementation exp.py that we trust:

{% embed url="<https://github.com/t4sk/hello-foundry/blob/main/src/Exp.sol>" %}

{% embed url="<https://github.com/t4sk/hello-foundry/blob/main/exp.py>" %}

We need the `eth_abi` package:

```bash
pip3 install eth_abi
```

Writing the differential test:

```solidity
pragma solidity ^0.8.18;

import "forge-std/Test.sol";
import "forge-std/console.sol";
import {exp} from "../src/Exp.sol";
import {Strings} from "@openzeppelin/contracts/utils/Strings.sol";

// FOUNDRY_FUZZ_RUNS=100 forge test --match-path test/DifferentialTest.t.sol --ffi -vvv

contract DifferentialTest is Test {
    using Strings for uint256;

    function setUp() public {}

    function ffi_exp(int128 x) private returns (int128) {
        require(x >= 0, "x < 0");

        string[] memory inputs = new string[](3);
        inputs[0] = "python";
        inputs[1] = "exp.py";
        inputs[2] = uint256(int256(x)).toString();

        bytes memory res = vm.ffi(inputs);
        // console.log(string(res));

        int128 y = abi.decode(res, (int128));

        return y;
    }

    function test_exp(int128 x) public {
        // 2**64 = 1 (64.64 bit number)
        vm.assume(x >= 2 ** 64);
        vm.assume(x <= 20 * 2 ** 64);

        int128 y0 = ffi_exp(x);
        int128 y1 = exp(x);

        // Check |y0 - y1| <= 1
        uint256 DELTA = 2 ** 64;
        assertApproxEqAbs(uint256(int256(y0)), uint256(int256(y1)), DELTA);
    }
}
```

The test contains two functions. In `ffi_exp()`, we are collecting the output of exp.py via FFI:

```solidity
    function ffi_exp(int128 x) private returns (int128) {
        require(x >= 0, "x < 0");

        string[] memory inputs = new string[](3);
        inputs[0] = "python";
        inputs[1] = "exp.py";
        inputs[2] = uint256(int256(x)).toString();

        bytes memory res = vm.ffi(inputs);
        // console.log(string(res));

        int128 y = abi.decode(res, (int128));

        return y;
    }
```