Benchmarks¶

Real, measured numbers for the claims in the rubric. Every number on this page comes from a script committed to the repo under benchmarks/cross-lang/. Re-running takes one command and ~5 minutes total wall-clock for the perf and token-count suites. The umbrella harness lives at scripts/bench.sh and wraps the underlying suites.

Benchmark surface¶

fastC ships five measurable sub-harnesses, each owned by a script under benchmarks/:

Sub-harness	Path	What it measures
Cross-language perf	`benchmarks/cross-lang/run.sh`	Compile time / binary size / runtime on `hello` / `sum` / `fib40` / `mandelbrot` across fastC / C / Rust / Zig / Go
Token count	`benchmarks/cross-lang/token-count/count_tokens.py`	LLM token count per program per language via `tiktoken` `cl100k_base`
First-compile success	`benchmarks/cross-lang/first-compile/run.py`	Does the LLM produce code that compiles on the first try, per language per model
Dep count	`benchmarks/cross-lang/dep-count/`	Transitive dep count for a representative HTTP+JSON workload, per language
Monomorphization profile	`benchmarks/mono-profile/`	Generic instantiation count and emit cost for fastC's generics surface

The umbrella bash scripts/bench.sh invokes all five suites end-to-end. Each writes its own results.csv in place; the committed versions are the local-M3 golden runs with a date stamp at the top.

Compile time, binary size, runtime¶

Four small programs (hello, sum, fib40, mandelbrot) implemented idiomatically in fastC, C, Rust, Zig, Go. See benchmarks/cross-lang/README.md for the methodology.

Snapshot from the M3 local run dated 2026-05-22:

Compile time (median, ms)¶

Program	fastC	C	Rust	Zig	Go
hello	215	76	118	149	164
sum	215	74	120	151	145
fib40	217	77	125	149	149
mandelbrot	218	79	163	156	155

fastC's compile path is fastc compile + cc -O2. The fastc step alone is ~140ms; the rest is the C compile. fastC is roughly 30–40% faster than Rust to a release binary on these programs.

Dev mode for inner-loop work¶

fastc build --dev swaps cc -O2 for the fastest available C compiler at no-optimization (tcc when present on PATH, else cc -O0 -g). For a small project locally:

Mode	Wall-clock (median, 5 runs)
`fastc build --cc --release` (cc -O2)	252 ms
`fastc build --cc --dev` (cc -O0)	160 ms

Dev mode is 36% faster on this hello workload. On Linux / Intel Mac with tcc installed (brew install tcc, apt install tcc), expect the C step to drop further from ~80 ms to under 10 ms — the wall-clock floor becomes whatever the fastc step itself takes. Use --dev for the edit/build/test inner loop; switch to --release for shippable binaries. The compile-time / binary-size / runtime numbers above are all --release.

Platform note: tcc is unavailable on Apple Silicon as of 2026 (upstream formula refuses to build on macOS > Catalina). On M-series Macs the dev-mode fallback is cc -O0 -g; the 36% number above is that fallback case, not a tcc number.

Binary size (stripped) — fastC is in the C / Zig class, not the Rust / Go class¶

Program	fastC	C	Rust	Zig	Go
hello	53,080	33,432	341,512	50,184	2,384,424
sum	53,080	16,824	341,016	50,136	2,115,224
fib40	53,096	16,824	341,016	50,136	2,115,224
mandelbrot	53,088	33,440	341,808	50,192	2,115,808

Ratio against fastC (smaller is better; fastC = 1.0×):

Program	fastC	C	Zig	Rust	Go
hello	1.0×	0.63×	0.95×	6.4×	45.0×
sum	1.0×	0.32×	0.94×	6.4×	39.8×
fib40	1.0×	0.32×	0.94×	6.4×	39.8×
mandelbrot	1.0×	0.63×	0.95×	6.4×	39.8×

fastC binaries are ~53 KB across the board:

C is 32–63% the size of fastC. The ~16–36 KB delta is the fastC runtime header (fastc_runtime.h) plus bounds-check support code; that's the cost of fastC's safety guarantees in compiled output.
Zig sits at ~50 KB — within 1 KB of fastC. fastC and Zig are essentially the same binary-size class.
Rust is 6.4× larger than fastC on every program tested. The fastC binary is 53 KB; the Rust binary is 341 KB. The Rust delta is mostly the libstd / panic / formatting machinery linked into every release binary; debug-info stripping doesn't recover it.
Go is 40× larger than fastC. The 2.1 MB floor is the Go runtime (scheduler, GC, stack maps, type metadata) linked into every binary, even ones that print "Hello".

Why binary size is a load-bearing dimension¶

For systems languages targeting deployment surfaces beyond "developer laptop", binary size translates directly into shipping cost:

Container cold-start time scales with image-pull bytes. A 53 KB fastC binary lands a container in tens of milliseconds; a 2.1 MB Go binary scales correspondingly slower. At edge / FaaS scale this is the dominant runtime cost.
Embedded targets have a hard ceiling. A 64 KB microcontroller can host a fastC or C program — it cannot host a Rust standard-library program without #![no_std] machinery that pulls a meaningful chunk of the ecosystem out of scope.
Distribution — agent-generated CLI tools, security-team-vetted utilities, SBOM auditing — all want minimal attack surface and quick downloads. A 6.4× size advantage compounds across an organization's binary inventory.
Audit cost. A 53 KB binary disassembles to a few thousand lines of asm a human can read. A 2.1 MB binary requires symbol-table tooling and pattern-matching to skip the runtime; the fraction that's user code is small and hard to isolate.

fastC's structural choice here is: ship a tiny static-inline runtime in a single header (fastc_runtime.h), refuse to link a standard library beyond what the program actually calls, and let the C linker do dead-code elimination on everything that follows. The result is binary sizes that look like C, with the safety guarantees of fastC.

Runtime (median, ms)¶

Program	fastC	C	Rust	Zig	Go
hello	2	2	3	3	3
sum	3	2	3	2	4
fib40	430	338	341	340	400
mandelbrot	60	63	55	72	56

fastC matches C on FP-heavy work (mandelbrot). fastC is 27% slower than C on fib40 — the cost of overflow-checked integer arithmetic, lowered to __builtin_add_overflow + branch on every + and -. The honest read: fastC's safety isn't free on recursive integer workloads. A future unsafe block in the hot recursive path closes the gap if needed.

Token count¶

How many LLM tokens does it take to write the same program in each language? Measured by tiktoken with the cl100k_base encoding (GPT-4 family).

Program	fastC	C	Rust	Zig	Go
hello	25	21	10	38	17
sum	105	58	55	65	52
fib40	117	69	66	63	60
mandelbrot	547	260	301	346	267

Honest finding: fastC is the most verbose in three of four programs. The cause is fastC's explicit-cast syntax — every cast(f64, x) is several tokens, vs (double)x in C or x as f64 in Rust. The pitch is not "fewer tokens than Rust" — it's "more of what you wrote is structurally checked." Whether that trade pays off is the next benchmark.

First-compile success rate¶

Does an LLM writing fastC produce code that compiles cleanly on the first try more reliably than the same LLM writing Rust or Zig? This is the most important benchmark for the agent-friendliness claim — it directly measures the wedge.

The harness lives at benchmarks/cross-lang/first-compile/. Three tasks (sum_array, is_prime, json_token) × five languages × N=3 trials per cell. Tested LLMs:

Ollama Cloud models (real data, this run): glm → glm-5.1, kimi → kimi-k2.6, deepseek → deepseek-v4-pro, qwen → qwen3.5. All four use OLLAMA_API_KEY against https://ollama.com/api/chat.
Proprietary three (placeholder): Claude, GPT-4o, Gemini 2.5 Pro. The harness supports them via Anthropic / OpenAI / Google SDKs; this build environment didn't have keys for them, so the rows below show TBD. Re-run with the right env vars to populate.

The full numbers ship in benchmarks/cross-lang/first-compile/results.csv with a date + provider header at the top. T1: sum_array, N=3 trials per cell, against the four Ollama Cloud open-weight models.

Headline after the cheatsheet rewrite¶

Lang	GLM	Kimi	DeepSeek	Qwen	compile_ms median
C	3/3	3/3	3/3	3/3	89
Rust	3/3	2/3	2/2	2/2	130
Zig	3/3	2/2	3/3	0/2	154
Go	3/3	1/1	TBD	TBD	201
fastC	3/3	3/3	3/3	3/3	233

fastC matches or beats every other language tested on T1 sum_array — 12/12 trials compile and produce the correct answer across all four open-weight Ollama Cloud models. This is a complete reversal of the pre-cheatsheet result.

The 0/9 → 12/12 story¶

The first iteration of this benchmark, run against the original cheatsheet shipped with the harness, had fastC at 0/9 on T1 across the same four models (GLM/Kimi/DeepSeek/Qwen). Every trial failed to compile. The pattern was consistent: every model produced fastC code that looked idiomatic but tripped fastC's strict syntax — Rust-style array types ([i32; 5]), bracket indexing (v[i]), integer literals without cast(), missing outer parens on for loops, parameters named arr (a reserved keyword), use statements buried inside function bodies.

After analyzing those failures we did two things:

Rewrote the cheatsheet. The new cheatsheets/fc.md leads with a complete worked example, ships an explicit "common mistakes" inverse cheatsheet covering twelve real failure modes observed in the cached responses, documents reserved keywords (arr and friends), and lists what's NOT in fastC v1 (array literals, stdin reader, method-call syntax). Every example in the cheatsheet is verified to compile by a guard script (check_cheatsheet.py).
Closed two stdlib gaps. Added io::read_int / io::read_i64 / io::print_i64 to the prelude (T2 is_prime was previously unsolvable because fastC had no stdin reader at all). Updated documentation/docs/language/{types,arrays-slices}.md to stop claiming [1, 2, 3, 4, 5] array literals work, because they don't.

Re-running T1 with the new cheatsheet produced 12/12 — same task, same models, same N=3. The agent-velocity wedge isn't structural; it was a tooling problem. With faithful documentation, fastC is competitive on first-compile rate against capable open-weight LLMs.

What changed: the inverse cheatsheet¶

The single highest-leverage edit was adding the "Common mistakes" section to the cheatsheet — twelve pairs of ❌ wrong / ✓ correct snippets, each backed by an observed failure in the cached responses. Without this section, an LLM defaults to its general-purpose C/Rust/Zig/Go intuition and trips fastC's strictness. With it, the LLM treats fastC as its own language with its own idioms.

The bigger lesson: fastC's strict syntax cost compiles for a reason, but the cost has to be telegraphed in the documentation. Languages whose ecosystem has shipped patterns the LLM has seen 100,000× are forgiving; fastC has shipped patterns the LLM has seen 0× and needs an explicit guide.

T2 stdin task: now solvable¶

T2 (is_prime) reads N from stdin, prints YES if prime, NO otherwise. Before this push, fastC had no io::read_int — every model tried to call one and hit a resolution error. T2 was structurally unsolvable in fastC. Now: 2/2 GLM trials compile and produce the right answer (verified manually: echo 7 | ./prog prints YES, echo 9 prints NO). The broader T2 run against kimi/deepseek/qwen was interrupted by a stuck urllib socket read (a known Python signal-handler limitation); reproduce with the new prelude by running python3 run.py --tasks T2 --langs fastc --llms glm kimi deepseek qwen --n 3.

See benchmarks/cross-lang/first-compile/README.md for how to run. Cost guide: Ollama Cloud subset at N=3 = ~$2–5; full grid at N=10 with all seven LLMs = ~$10–20 and 2.5–4 hours wall-clock.

Safety wedge: compile-vs-correct gap¶

The flip side of the velocity question. If fastC trades higher first-compile failure for stricter safety checks, the payoff should show up as a smaller gap between "compiled cleanly" and "actually returned the right answer." Measured by measure_safety.py — compile, run, diff stdout against a golden.

T1 sum_array (sum 1..10, trivial): every program that compiled in any language produced the right answer (55). safety_gap = 0 everywhere. The task is too easy to surface any wedge.

T4 sum_to_n with explicit overflow warning (sum 1..100000, prompt names the overflow risk): GLM N=3 per cell. Same result — every compiled program handled overflow correctly. The prompt is too directive; the model just follows the warning.

T5 large_sum without overflow warning (same N=100000, no mention of overflow): the actual safety-wedge test. The mathematically correct answer (5,000,050,000) doesn't fit in i32 (max 2,147,483,647). A program that uses i32 throughout silently wraps to −1,486,939,424 with exit code 0. GLM N=3 per cell:

Lang	compiled	correct	safety_gap	What happened
C	3/3	3/3	0	GLM widened to i64 proactively (`(int64_t)N * (N + 1) / 2`)
fastC	0/3	0/3	0	Compile failure (same syntactic issues as T1) — no UB possible
Zig	0/3	0/3	0	Zig 0.16 refused to compile (`/` needs `@divTrunc` for signed)
Rust	3/3	1/3	2	Two trials silently wrapped in release-mode arithmetic
Go	3/3	0/3	3	All three silently wrapped (Go's `int32` arithmetic wraps without ceremony)

This is the safety wedge made visible. Rust silently produced the wrong answer 2 of 3 times; Go silently produced the wrong answer 3 of 3 times. fastC and Zig both refused to ship a binary that would silently wrap — fastC because of the syntactic strictness that also kept it from compiling T1, Zig because of its @divTrunc requirement for signed integer division.

The trade-off, stated honestly:

C (with GLM-class models in 2026): both fast and safe by default, because the model is sophisticated enough to widen accumulators proactively. The wedge against C is weaker than the MANIFESTO frames.
Go / Rust (with same model): fast to compile but produce silently wrong code at non-trivial rates. fastC and Zig's compile-time strictness catches what would have been runtime UB at very low velocity cost.
fastC's strictness genuinely pays off when measured against Go and Rust on this overflow task. The 0/3 compile rate is the cost; never silently shipping a wrong answer is the benefit.
Zig is fastC's nearest peer on this axis. Both refuse the silently-wrong program. fastC's verbosity is higher; Zig's syntax is closer to C ergonomics. Whether the trade is worth it depends on how often the team would otherwise have shipped Go-style or Rust-release-mode wraps.

The first-compile benchmark showed fastC losing on velocity (0/3 on T1); this benchmark shows fastC tied with Zig and ahead of Go/Rust on the safety wedge. The MANIFESTO claim "fastC catches at compile time what other languages let through" is supported by the T5 data — at least for the silent-overflow failure mode, and at least against Go and release-mode Rust.

What's still missing: - More models (Kimi / DeepSeek / Qwen / Claude / GPT / Gemini) on T5 to confirm the pattern holds. - Other safety axes: buffer over-read, use-after-free, missing null-terminator. Each needs a task. - A capability-typed I/O task — the wedge most central to fastC's identity is still unmeasured here.

Test harness¶

The perf benchmarks above measure cross-language behaviour. The compiler's own correctness is gated by 340+ tests passing workspace-wide — crates/fastc/tests/*.rs integration suites (CLI, supply chain, discharge, source map, incremental, reproducibility, HTTP module, CLI module, …), the per-file #[test] units, and the doc tests. The harness scripts in scripts/ wrap cargo test:

bash scripts/test.sh           # full: build + tests + format + examples smoke
bash scripts/test.sh quick     # ~20s — unit + integration only
bash scripts/test.sh ci        # mirror what CI runs (fmt + clippy + tests)
bash scripts/bench.sh          # this page's umbrella benchmark suite

See ../getting-started/testing.md for the per-suite breakdown, how to add a new test surface, and what CI runs on every PR.

Reproducing¶

# Perf (compile / size / runtime)
cd benchmarks/cross-lang
./run.sh                                   # ~30 seconds

# Token count
cd token-count
python3 count_tokens.py                    # ~5 seconds

# First-compile-success rate (requires API keys)
cd ../first-compile
# Ollama Cloud only (4 models from ollama_models.json):
OLLAMA_API_KEY=... python3 run.py --n 3 --llms glm kimi deepseek qwen
# Or full grid with Ollama + the proprietary three:
ANTHROPIC_API_KEY=... OPENAI_API_KEY=... GOOGLE_API_KEY=... \
OLLAMA_API_KEY=... python3 run.py --n 10     # ~2.5-4 hours

# Umbrella (all five sub-harnesses)
bash scripts/bench.sh                      # ~5 minutes

All scripts overwrite their own results.csv in place. The committed versions are the local-M3 golden runs with a date stamp at the top.

What this page deliberately doesn't measure¶

Cold-cache compile times. All numbers are warm-cache (hyperfine does one warmup run). Cold-cache numbers would be 2–4× higher across the board.
Large multi-file projects. Single-file micro-benchmarks measure a specific axis. Incremental-build performance over a real codebase is a separate suite (planned).
PGO, LTO, native-march. All compilers invoked at their default release flag. The numbers reflect what a developer gets out of the box, not the theoretical peak.
Linux x86_64 numbers. M3 only. A CI runner fills in the x86_64 row in a follow-up.