Engineer. Investor. Writer.
Missing Years Is a Bug
A routine version bump. A database that wouldn't open. And a linker three years out of date, hiding in a build container nobody had looked at since the day it was written.
A routine deploy. Mostly dependency bumps, a few logic changes, a green end-to-end devnet. We were rolling out a new consensus client, and that's where our attention was. The execution client was along for the ride — the kind of change where you almost skip staging. Almost.
If anything was going to break, it wasn't going to be the execution client.
The node didn't start. Not a config error, not a port clash — an immediate panic on boot, before it did anything useful:
MDBX_INVALID. The storage engine refused to open a database that had been running fine — same data directory, same permissions, same config as the previous deploy. Nothing about the database had changed. The only new variable was the binary sitting on top of it, out of an unremarkable version bump. A bug that won't tell you what's wrong, only that the answer is no.
We Couldn't Reproduce It
Locally: fine. Fresh cloud dev boxes: fine. Built by hand anywhere we tried: fine. It failed in exactly one place — the binary produced by the actual build container we deploy from.
"Works on my machine" usually means prod is right and your laptop is wrong. This was inverted. Every binary built outside the container opened the database. The container's binary couldn't.
Everything visible looks the same. The binaries are not.
Same Cargo.lock, same target triple, same compiler version in the config. So how was the binary different? While we were in there we ruled out the container's Ubuntu 22.04 base — not a factor, though pulling up its EOL date was its own small surprise: April 2027, a lot closer than "LTS is basically forever" makes it feel.
Chasing the Ghost
MDBX_INVALID on database open. The database file exists and was created by the same binary version, but the new node refuses to start. No grace, just a panic.
Ran mdbx_chk and walked through integrity checks. Database is structurally sound -- pages, trees, and checksums all pass. It's not a corruption issue.
Diffed every environment variable and config file between working and broken environments. Identical. Same image, same flags, same data directory, same everything.
Binaries built locally open the database fine. Binaries that come out of the actual build container don't. Same source, same Cargo.lock, same target. The difference has to be in how the container builds.
Compared the compiled artifacts directly using readelf and ldd. Different section layouts, different relocation patterns. The binaries are structurally different despite identical source.
The build container had mold pinned to a 1.x release from years ago. Dev environments were installing the latest 2.x from the package registry. mold shipped breaking changes between those major versions, including section alignment behavior. A silent drift that finally surfaced as an incompatible binary.
We ran mdbx_chk. We diffed every environment variable between working and broken deploys, stared at configs side by side, ran the binary under strace. Someone floated subtle database corruption the integrity check had missed, so we opened it with a known-good binary from another machine. It opened fine.
The database was healthy. The binary was wrong — specifically ours, the one from the real container, producing something MDBX rejected on page-geometry grounds.
So we stopped reading the source and started reading the ELF. readelf, ldd, symbol tables, section layouts. The two binaries were structurally different in ways they had no business being.
The Linker
Both environments used mold, the fast linker. Different versions.
The build container had mold pinned — installed once, forgotten. Dev and runtime pulled latest from the package registry, so they tracked current releases. The container was years behind. Not a patch, not a minor version. The ecosystem had shipped mold 2.x; the container was still on 1.x. Nobody noticed because nobody looked. It was just "how the container is set up."
Dependency drift over time
The 1.x to 2.x transition wasn't only performance work. It changed how mold handles certain relocation types and section alignment. For nearly all software that's invisible. libmdbx is not nearly all software: it bakes hard page-geometry assumptions into a C library and enforces them with internal consistency checks. Old mold emitted a binary with subtly different section alignment, those checks fired, and MDBX refused the file. Binaries linked with current mold passed.
MDBX_INVALID. A full day of debugging a perfectly healthy database.
This Isn't Debt
The Dockerfile's git log had exactly one entry for the mold line — the commit that first stood up the build environment, years back, untouched since. There was no decision to stay on 1.x. It was whatever version was current when someone wired the container together, and then everyone moved on.
Nobody had tested this configuration in years. Not the mold maintainers, not the Rust toolchain team, not libmdbx. Three years of releases, three years of the ecosystem building and testing against current behavior — and we were sitting outside all of it.
GitHub Has the Data
630 million repositories. 1.9 million with Dockerfiles. 15.5 billion documents indexed, 115 terabytes of raw code. GitHub knows which mold version is in your build container, when that line last changed, and what the current release is — for every repository on the platform.
build container pinned 1.11.0, current 2.x line is 2.35.1 (3+ years behind, section alignment semantics changed in 2.x)
build container pinned 12.3, current is 14.2 (ABI changes may affect C FFI boundaries)
Dependabot runs on 2.66 million repos and opened 75 million pull requests in 2022 alone. It's genuinely good at what it does: runtime dependencies, known CVEs, manifests it understands. What it doesn't do is read apt-get install lines in a Dockerfile. As far as Dependabot is concerned, the build environment doesn't exist.
That's a choice, not a technical limit. The Dockerfile is in the repo, its history is right there, mold's release history is public, and the committed Cargo.lock says exactly which ecosystem you're in. A check that reads "your build container pins mold 1.11.0, the current release is 2.35.1, and that gap includes breaking changes to linker semantics" is not a research problem. It's a query over data GitHub already has. They're just not running it.
The Fix
One line. Pin mold to a specific 2.x release in the build container. Rebuild. The node opened its database without complaint.
Wednesday ended late.