GAUNTLET started as a question: what does the infrastructure behind a chip-design regression product actually do — and could I rebuild the core loop from scratch, honestly, in a weekend? This is what came out.
Everyone can build the demo. The thing I kept bumping into — reading about EDA flows, design-verification pipelines, the tooling teams live inside — was that the interesting engineering isn't in any single tool. It's in the boring connective tissue: the thing that runs forty tools in the right order, decides what "failed" even means when a tool lies about its exit code, tracks which failures are known debt, and tells you what changed since last night.
That connective tissue is a whole product category, and it's mostly invisible. So I rebuilt a miniature of it end to end — real subprocesses, real signals, real exit codes, logs as files on disk — with a hard rule: no faking the hard parts. If real EDA tools under-report via exit codes, mine had to too. If real logs are gigabytes, mine had to stream. The point wasn't to ship a product; it was to earn the intuition.
Scoped tight on purpose. Built strictly in phases — core first, stretch features only after the demo was already complete.
Mapped the real loop — flow → run → parse → waive → diff — and, just as deliberately, wrote down the scope guards: no plugins, no asyncio, no web UI, no config server. Naming a floor and a ceiling before writing code is what kept the weekend a weekend.
Frozen dataclasses as the single source of truth, then flow loading with unique-id checks, unknown-dependency detection, and a three-color DFS cycle finder. A broken flow has to fail fast — before anything executes — not halfway through a run.
The ready-set scheduler and ShellStep came together: steps submitted only when dependencies pass, each running as a real process, logs streamed to disk. First end-to-end green run. This is the moment it stopped being a diagram.
The honest bit. A timed-out shell wrapper would exit while its child simulator kept running — a zombie eating a worker. Fixing it meant process groups, a SIGTERM → grace → SIGKILL escalation, and later making that whole dance portable to Windows. Killing a process cleanly is genuinely harder than starting one.
Rulepacks compile to regex; a tool can exit 0 and still FAIL on an unwaived ERROR; waivers turn a would-be failure into visible WAIVED debt. Then the payoff: the diff that buckets NEW FAILURES / FIXED / STILL FAILING and exits non-zero on regressions.
Persistence so history/report/diff work from the DB alone. A deterministic mock-tool harness (GAUNTLET_SEED for byte-identical runs, GAUNTLET_FLAKE to inject regressions). A fast, fully-offline test suite — because a regression platform with a flaky test suite is a bad joke.
Only after the core was demo-complete: self-contained HTML reports, a GitHub Actions workflow that archives run evidence, honest flaky-pass retries — and a cross-platform test rewrite so the whole thing is green on Linux, macOS, and Windows.
The scheduler's entire contract is run() → StepResult. ShellStep is one implementation; a future PythonStep or RemoteStep touches zero scheduler code. The abstraction earns its keep by what it lets you add later.
The tempting shortcut is to treat a waived failure as a pass. That's how debt goes invisible. Keeping WAIVED distinct — with owner and reason surfaced in every report — was a product decision disguised as an enum value.
When an upstream step fails, its dependents are SKIPPED — and the diff deliberately doesn't count that as a new failure. A skipped step tells you nothing new about itself; blaming it would bury the real culprit.
Seeding the mock tools makes runs byte-identical, so tests assert on exact output; a separate flake knob injects regressions on demand. The test suite proves the diff catches a regression — the whole reason the thing exists.
One language, one dependency. Every choice below is a "why this," not a default.
Batteries-included stdlib covers subprocesses, threads, signals, SQLite and regex — the exact primitives an EDA harness needs, with nothing to install.
Workers block in subprocess.wait() where the GIL is released, so threads give true parallelism without the cognitive tax of asyncio or multiprocessing.
One file, three tables, no ORM, no server. It's the platform's memory — durable, queryable, zero-ops.
Flows, rulepacks and waivers are declarative YAML. The only third-party package in the whole project — and it earns the slot.
Compiled-once regex rulepacks classify logs line-by-line; the HTML report is a stdlib template — a self-contained page with no build step and no new deps.
A fast, deterministic, fully-offline suite, run in CI on every push — with run artifacts uploaded even when a run fails, so the evidence survives.
The source is on GitHub — clone it and run the money shot in three commands. If you want to talk regression infra, LLM systems, or anything in between, I'm around.