Andreas Bigger

base/base runs four tiers of tests before a change lands. Unit tests cover a function in isolation. Action tests exercise protocol logic like batching and derivation against in-memory actors. System tests bring the full L1 and L2 stack up in Docker on the merge queue. A nightly fuzz job throws randomized transaction streams at sync-parity. Static checks handle formatting, clippy, licenses, and feature flags. Together they cover correctness across the stack.

None of them measure speed. Regress state-root computation, trie access, or block building by any margin and every check on the pull request stays green, because they test whether the code is correct, not whether it is fast. For a sequencer, where latency and throughput are the product, that is a strange thing to leave unmeasured.

This post is about closing that gap, and the harder problem hiding inside it: getting a wall-clock benchmark on shared CI hardware to produce a number you can trust.

The instruments point the wrong way

Most of the work is already done. Around ten crates ship criterion benchmarks over the hot paths, covering the precompiles, the Merkle Patricia trie, the execution trie, state root, sender recovery, and batch encoding and derivation. Separately, a scheduler builds our client from main and runs the base-bench suite on real hardware across devnet, testnet, and mainnet, with warning and error thresholds on block-building and validation latency and a dashboard to watch. It also runs live load tests, like a daily Sepolia run against a gas-per-second target.

So the instruments exist, and so do the acceptance criteria, the actual numbers a healthy chain should hit. The problem is when they run. Everything fires on a fixed schedule against a pinned copy of main, every few days, never against a change. A regression surfaces late, on a dashboard, blamed on a batch of a dozen commits rather than the one that caused it, and every result is advisory. We have everything except a signal at the point a change lands.

A fifth tier, in two tracks

Track 1 is a per-PR tripwire. Reuse the criterion benchmarks, run a curated CPU-bound subset on our own bare-metal runners against both the base commit and the PR head, and post a comment, but only when a benchmark clears a threshold. Slowdown is a caution, speedup is a win, silence otherwise. It never fails the build. The comment looks like this:

The 53 benchmarks that barely moved are collapsed; a wall of "within noise" rows is how you train people to stop reading the comment. Benchmarks that can't reproduce their own number are flagged unstable rather than reported as confident-looking fiction. The one real regression sits at the top, tied to the change that caused it, in front of the person who wrote it.

Track 2 matters more. Wire the scheduled macro suite into the merge loop: run it per-merge, tie each result to its commit, compare against a stored baseline, and alert on a real regression. It catches what Track 1 cannot, the lock contention and allocation churn that leave instruction counts flat while throughput drops. It costs dedicated hardware and a trustworthy baseline, so it follows Track 1 rather than blocking on it.

Making wall-clock honest

A stopwatch on shared hardware lies. Run the same benchmark twice and you get two numbers, from warm-up, thermal throttling, or a noisy neighbor on the box. Run the base once, the head once, and subtract, and you manufacture half your regressions and bury the rest in the wobble.

The fix is a stack of disciplines. Pin the work to a single dedicated core. Interleave the runs, A, B, B, A, so whatever state the machine is in gets shared across both sides instead of landing on one. Repeat each side and combine the runs with a geometric mean, so one outlier can't dominate. Then check each side against itself: if the base can't reproduce its own number within 10%, the machine is too noisy to trust and you report that instead of a fabricated delta. Only when both paired comparisons agree, and both clear the threshold, does it say anything.

// Compare two sides of a change without trusting a single stopwatch reading.
// Runs are interleaved (A, B, B, A, ...) so machine noise is shared evenly.
let base = geomean(&base_runs); // combine repeats; one outlier can't dominate
let head = geomean(&head_runs);
 
// If a side can't reproduce its own number, the comparison is meaningless.
if spread(&base_runs) > 0.10 || spread(&head_runs) > 0.10 {
    return Report::Unstable;
}
 
// Only speak up when the move is real and clears the threshold.
let delta = (head - base) / base;
match delta {
    d if d >  0.10 => Report::Regression(d), // slower: a caution
    d if d < -0.10 => Report::Speedup(d),    // faster: a win
    _              => Report::Quiet,          // within the noise floor
}

None of this makes the number deterministic. It makes it honest, good enough to comment on, not good enough to block on. That is why both tracks stay advisory. reth keeps its bare-metal macro benchmarks advisory for exactly this reason, and revm, foundry, and the rest converge on the same rule: gate a merge only on a deterministic metric, and treat wall-clock as something you track and alert on.

Where this goes next

The exciting part is that an advisory tripwire is not the ceiling, it's the on-ramp. The way you turn a performance check into a required one is to stop timing the clock and start counting simulated CPU cycles: a deterministic measure, run-to-run variance well under 1%, fair enough to block a merge on. That gate is genuinely within reach.

revm already runs it through CodSpeed on every PR, and we can reach the same bar with tooling we host ourselves: iai-callgrind to count cycles, a self-hosted Bencher to store baselines and render the comparison. Same accuracy, nothing leaving the org, no third-party App to install. It's the natural follow-on to Track 1, and it's the direction we're heading.

That's the real reason to start with the honest stopwatch: the per-PR comment is already the exact shape that gate will take, so a cycle counter drops straight into the slot the wall-clock check lives in today. And even before the gate arrives, the tripwire earns its keep. "All green" can no longer quietly mean "half as fast."