# Benchmarks

This directory holds the performance benchmarks for the derivation tactics.
They run as part of `make test` and produce one CSV plus one PNG per sweep under
`bench/results/latest/`.

## Sweeps

A *sweep* varies one factor that we expect to affect derivation cost and holds
the rest fixed.  The five sweeps currently defined are:

| sweep    | x-axis                                          |
|----------|-------------------------------------------------|
| `types`  | number of mutually inductive types              |
| `ctors`  | number of constructors per type                 |
| `rec`    | recursive arguments per constructor             |
| `nonrec` | nonrecursive arguments per constructor          |
| `args`   | both kinds of arguments together (rec = nonrec) |

Each sweep is described by a plain CSV file at `bench/<sweep>.csv`.  The header
lists `value` (the x-axis) followed by one column per `gen_bench.ml` flag; every
row becomes one synthetic Rocq file with those parameters.  Adding a data point
is a one-line edit; adding a new sweep is a new CSV plus an entry in
`BENCH_DIMS` (and an x-axis label in `bench/plot.ml`).

## Pipeline

```
bench/<sweep>.csv  ──▶  gen_mk.awk  ──▶  bench/build/bench-deps.mk
                                            │
                                            ▼
                          gen_bench.ml ──▶  build/<sweep>/v<n>.v
                                            │
                                            ▼
                                  rocq c ──▶ build/<sweep>/v<n>-<r>.out
                                            │
                                            ▼
                              summarize ──▶ results/latest/<sweep>.csv
                                            │
                                            ▼
                                plot ──▶ results/latest/<sweep>.png
```

`bench/gen_mk.awk` reads every `bench/<sweep>.csv` and emits
`bench/build/bench-deps.mk`, an auto-generated make fragment that contains, for
each row, a `.v` recipe and per-rep `.out`→`.v` dependencies.  `bench/bench.mk`
(the hand-written companion) provides the small number of generic pattern rules
that drive the rest of the pipeline:

- `%.v`         — generated by `gen_bench.ml` from the CSV row.
- `%.out`       — captured stdout of `rocq c` on the matching `.v`.
- `summarize.ml` — small OCaml script (run under the bytecode
                   interpreter) that parses a batch of `.out` files
                   into a long-form summary CSV.
- `plot.ml`      — small OCaml script that reads a summary CSV, takes
                   the median across reps for each
                   `(value, derivation step)`, and shells out to
                   `gnuplot` to render the PNG.

Each benchmark is run several times (`BENCH_REPS` in `bench/bench.mk`) so that
`plot` can take the median and smooth out noise.  Because every `.out` file is
its own make target, `make -j` parallelises the runs across cores for free.

## Files

| file                   | role                                                    |
|------------------------|---------------------------------------------------------|
| `<sweep>.csv`          | sweep configuration (one row per data point)            |
| `gen_bench.ml`         | generates a synthetic `.v` file from CLI flags          |
| `gen_mk.awk`           | turns the sweep CSVs into `build/bench-deps.mk`         |
| `bench.mk`             | hand-written make rules driving the whole pipeline      |
| `summarize.ml`         | aggregates per-run `.out` files into one summary CSV    |
| `plot.ml`              | renders one summary CSV into a PNG via `gnuplot`        |
| `build/`               | gitignored; intermediate `.v`, `.out`, compiled tools   |
| `results/latest/`      | gitignored; final summary CSVs and PNGs                 |
