# Testing

There are three layers. Use all three.

1. **Headless suites** (`test/t_*.js`) run the simulation in Node with no browser. They are fast
   and deterministic, and they are where most checks live.
2. **Browser tests** (`test/play.py`, `test/t_smooth.py`) drive the real page in headless
   Chromium with real key, mouse and gamepad input. They check the UI, saving, the smooth look,
   and that the page throws no errors.
3. **Screenshots** (`tools/shot.py`) show what something looks like. Open the PNG and look at it.

Run everything with `python3 test/run_all.py --browser`. It exits non-zero if anything failed and
prints the failing checks.

## Headless suites

Each suite is a plain Node script:

```js
// What this suite covers, in a sentence or two.
const { load, suite, flatWorld, rect, count, ticks } = require('./harness');
const { run } = load();                 // loads src/sim/ (the files in src/manifest.json)
const { check, done } = suite();

flatWorld(run, 240, 160, 100);          // 240x160 grains, sky above row 100, stone below
run('money = 5000; bot.x = 60; bot.y = 40;');
run('for (let bx = 20; bx < 30; bx++) placeBlock(BP_BELT, 1, bx, 20, true);'); // true: free, no reach check
rect(run, 84, 60, 95, 79, run('SAND'));
ticks(run, 300);
check('sand rides the belt off its end', count(run, run('SAND'), 120, 80, 239, 99) > 100, count(run, run('SAND')));
done();                                 // prints ALL PASS or N FAILED, sets the exit code
```

- `run(code)` evaluates JavaScript inside the simulation and returns the result. Every top-level
  name in `src/sim/` is available. Return plain values (numbers, strings, `JSON.stringify(...)`).
- Harness helpers:
  - `flatWorld(run, w, h, ground)`;
  - `put(run, x, y, m)`;
  - `rect(run, x0, y0, x1, y1, m)`;
  - `count(run, m, x0, y0, x1, y1)`;
  - `maxTemp(run, ...)`;
  - `ticks(run, n)`;
  - `dump(run, x0, y0, x1, y1, withTemp)` for an ASCII picture of an area.
- `check(name, ok, detail)`: name the behaviour the player would notice ("a pump moves water
  over a wall"). Put the measured numbers in `detail`, so a failure shows how far off it was.
- Coordinates are in grains. Block coordinates (for `placeBlock`, `placeMachineAt`) are grains
  divided by 4.
- Errors name the real file and line (`src/sim/pipes.js:42`).
- A new suite named `test/t_<topic>.js` is picked up by `run_all.py` automatically.

**Determinism.** The simulation's randomness comes from seeded generators. The same code and the
same test give the same output every run, apart from lines with timings in ms. To prove a
refactor changed nothing, save every suite's output before and after, and diff them:

```sh
for t in test/t_*.js; do node $t > /tmp/before/$(basename $t).txt; done   # before the change
for t in test/t_*.js; do node $t > /tmp/after/$(basename $t).txt; done    # after
diff -r /tmp/before /tmp/after
```

## Browser tests

- They need the `.venv` from README.md → Develop; plain `python3` switches to it on its own.
- `test/play.py` (about 2.5 minutes) builds a world through the title screen and plays it with real
  input. It checks the tools, building, copy/cut/paste, undo, the build menu, filters, pipes,
  the notebook and market, saving and continuing, exporting and importing a world file, the
  gamepad and world sizes. Screenshots go to
  `shots/play/`. Pixel look by default; `python3 test/play.py shots/play smooth` uses the smooth
  look.
- `test/t_smooth.py` checks that the WebGL2 smooth look compiles and draws, that it falls back
  to pixels cleanly, and that the thermal lens works.
- Run them one at a time. Two headless browsers at once can starve each other, and
  `t_smooth.py`'s half-resolution check then fails. `run_all.py` runs them one after the other.
- Rebuild first (`python3 build.py`); `run_all.py --browser` rebuilds for you. All browser tools
  use `tools/browser.py` and `tools/server.py`: a free-port background HTTP server serves the
  built page with COOP/COEP headers and shuts down after the browser. No `file://` loads.
- Choose Chromium graphics with `--gpu real|soft` or `DEEPGRAIN_GPU=real|soft` (the command line
  wins). This works for the suite runner, individual browser tests, screenshots and profiler.
  Real mode uses ANGLE Vulkan on the host GPU; soft uses SwiftShader and sets
  `window.__allowSoftwareGL` before load. The explicit software-fallback test still uses soft
  mode with that override off. Firefox uses its own default graphics backend.
  The default is **real**, including the profiler and `caps.py`. No colour tolerances were
  changed. The play test holds the slow-save fixture open during its screenshot, waits for
  tool/selection UI updates, disables camera lead after testing it, and gives the gamepad drill
  a known reachable target; these checks do not depend on GPU speed or earlier digging.
- `python3 tools/caps.py` prints isolation, actual WebGPU adapter info and the GL renderer for
  Chromium and installed Firefox. Real mode requires a hardware Chromium adapter.
- `test/t_caps.py` checks missing, null and rejected WebGPU adapters, both title notices, warning
  mode and the future `REQUIRE_GPU_AND_THREADS` startup gate. Its forced notice screenshot is
  `shots/t88-p1-notice.png`.
- Install Firefox with `.venv/bin/python -m playwright install firefox`.
  `python3 test/firefox_smoke.py` checks isolation and the WebGPU notice, starts a pixel world,
  runs three seconds and checks page errors. `run_all.py --browser` includes it and prints a
  skip note if Firefox is absent; an installed browser that fails to launch is a failure.
- Linux WebGPU currently needs Firefox Nightly; the notice points to `about:config` →
  `dom.webgpu.enabled`. Sources: [Mozilla's experimental features](https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Experimental_features#webgpu_api)
  and [GPUWeb implementation status](https://github.com/gpuweb/gpuweb/wiki/Implementation-Status#firefox).

## Screenshots: tools/shot.py

```sh
python3 tools/shot.py shots/silo.png --setup tools/scenes/clear-sky.js \
    --js "$(cat tools/scenes/stone-uses.js)" --ticks 240 --zoom 7 --no-light
```

- It starts a new world, pauses it at once, runs your setup code in the page, runs `--ticks`
  simulation steps, then saves the picture. Because the world is paused otherwise, the same
  options always give the same picture.
- Setup code can use every game global, plus `SX`, `SY` (the bot's starting position).
  `tools/scenes/clear-sky.js` clears a flat work area and sets `FLOOR`, `BX`, `BY` and `CAM`. Set
  `CAM = [x, y]` to frame the shot.
- The HUD and the cursor previews are hidden unless you pass `--hud` or `--previews`.
- `--renderer webgpu|legacy` selects the pixel renderer (default WebGPU). WebGPU screenshots
  fail clearly when no device is available; Smooth still uses WebGL2.
- `test/t_gpu.py` compares the rendered composite against Legacy for ordinary cells, built pieces,
  ghosts, fog and all thermal palettes, and checks world replacement, resize, timing and device loss.
  World-pixel tests use `worldPixelContext()` to read both stacked canvases. The helper
  presents the retained GPU result again before copying, because canvas textures expire after
  presentation ([WebGPU specification](https://gpuweb.github.io/gpuweb/#abstract-opdef-expire-the-current-texture),
  [Mozilla implementation notes](https://bugzilla.mozilla.org/show_bug.cgi?id=1965377)). Tests that change
  static `noiseB` in place set `GPU.world = null` before drawing to upload their new fixture.
- `--look smooth` for the smooth look. Test in the pixel look unless you are checking graphics.
- `--freeze` draws the last frame at a fixed time and frame count, so animations are caught at
  the same moment. Two shots of the same scene then match pixel for pixel, and a before / after
  pair differs only where something really changed. A quick comparison with Pillow:

  ```python
  from PIL import Image, ImageChops
  a, b = (Image.open(p).convert('RGB') for p in ('before.png', 'after.png'))
  print(ImageChops.difference(a, b).getbbox())   # None: identical
  ```

  To compare against an older version, build it somewhere else (for example from a git
  worktree) and pass that page with `--page`.
- Page errors are printed with their source file and line, and the exit status is 1.
- Look at the picture. A shot that "ran without errors" can still show the wrong thing.

## Pitfalls

- **GPU access in headless Chromium.** Real mode needs access to GPU devices and driver caches;
  a filesystem sandbox can silently force software rendering. Use `tools/caps.py` to check.
  Soft mode is available for machines without a GPU. Both modes load isolated HTTP pages.
- **Top-level `const` and `let` are not `window` properties.** In the page, `window.SM` is
  `undefined` even though `SM` exists. Write `typeof SM !== 'undefined'`, or use the bare name.
  Functions declared with `function` are on `window`.
- **Playwright calls a string that evaluates to a function.** `page.evaluate("x = function () {}")`
  assigns the function and then calls it. Wrap statements in an arrow function:
  `page.evaluate("() => { ... }")`.
- **Starting a world takes a moment.** `startNewWorld(seed, w, h)` generates on a timer. Wait
  for `GAME.running === true`. Then remove the welcome card if it is in the way:
  `document.querySelectorAll('.card, .toast').forEach(e => e.remove())`.
- **Real-time play is not deterministic.** In the browser the simulation runs as fast as frames
  arrive. For repeatable results, pause (`GAME.paused = true`) and call `simTick()` yourself, as
  `tools/shot.py` does.
- **World sizes** offered by the game: 2048×1024, 3072×1280, 4096×1536 grains.
- **Rebuild after every source change** before any browser test or screenshot. They all load
  `dist/index.html`.

## Measuring

- `tools/perf.py`: the phase 0 CPU/frame profiler, served over HTTP with COOP/COEP isolation.
  Rebuilds automatically. Run on an otherwise idle machine, without another browser test:

  ```sh
  python3 tools/perf.py --json tools/perf-baseline.json
  python3 tools/perf.py --json /tmp/perf-now.json --compare tools/perf-baseline.json
  python3 tools/perf.py --viewport 1920x1080 --world 2048x1024 --zooms 12,6,4,3,2 \
      --looks pixel,smooth --scenes settled,factory,heat --seconds 10 --json /tmp/perf.json
  python3 tools/test_perf.py
  ```

  The default matrix is three scenes × two looks × five zooms × lighting full/off. The world
  seed is 424242; fog is off so all visible grains are shaded. Settings otherwise use game
  defaults (including glow and the minimap), with autosave, camera lead and shake disabled.
  `settled` runs 600 ticks; `factory` reuses the furnace/transport layout from
  `experiments/factory_cost.js`, adding screws, a sand filter and full silos for sustained sand
  and ore flow; `heat` places water over magma beside a rock floor. The latter two settle for
  another 60 ticks. Each rebuild resets the power-rebuild scheduler, weather random stream and all statistics to
  fresh-page values; fixture hashes are checked by `test/t_perf.js`. No save/load snapshot is used: it would quantize heat and land flyers.

  **World** is the whole `drawWorld` CPU call. **Light** is the lighting work inside that call;
  **Prep** sets up the view/masks/joins; **Colour** fills CPU grain buffers (including ghosts and
  flyers); **Upload** submits the pixel buffer or smooth textures; **Blit** submits drawing,
  including pixel light/glow composition or the smooth shader and Canvas2D copy. These are CPU
  wall times, including driver stalls; they are not GPU timestamp measurements. In pixel look,
  the glow upload is part of Blit. The separate `computeLight_ms` times an unconditional light
  call even when the row has lighting off. World includes Light: do not add them together.
  Overlay totals and each individual draw call are in JSON, with ms/frame and invocation counts; conditional
  overlays absent from the fixture have zero calls, not a measured zero execution cost.

  Five calls warm each render configuration before 30 measured calls. Simulation warms five
  full 120-tick periods on a disposable scene, then rebuilds the same seeded fixture and
  measures 240 consecutive ticks. Every direct `simTick` callee is discovered from its source;
  JSON records per-call and amortized per-tick costs and invocation counts, including periodic
  functions. Inactive paths (e.g. load wake, saplings, flow tracking) have zero calls and null
  per-call cost. Unattributed time includes tick bookkeeping and clock/wrapper overhead.
  Awake/hot chunk counts are recorded at the start/end of simulation and real runs.

  Real-loop tests independently rebuild each scene for zooms 2 and 6 in both requested looks,
  regardless of `--zooms`. `--seconds` applies to each of these twelve default runs. Raw samples
  retain frame interval (rAF timestamps, including waiting/scheduling), CPU frame work, total
  simulation work per frame and number of ticks. p50/p95/p99 use interpolated sorted samples;
  five warm-up frames are excluded. Zero-tick frames correctly have zero simulation time.
  JSON also records the smooth renderer's adaptive resolution scale: compare it as well as time.
  Comparisons accept subsets of the baseline scenes, looks and zooms, with identical world/viewport dimensions and run duration, print all timing changes, and mark >10% regressions;
  tiny timings and tails can be noisy, and a zero baseline has no meaningful percent change.

  Performance runs launch Chromium with
  `--use-angle=vulkan --enable-features=Vulkan,VulkanFromANGLE --enable-gpu --ignore-gpu-blocklist --enable-gpu-rasterization --enable-unsafe-webgpu`.
  They record the unmasked GPU/renderer, cores, browser version, viewport/world size,
  `crossOriginIsolated`, `navigator.gpu`, source hash and elapsed time. Real mode refuses software
  renderers; both modes require isolation. Chromium needs access to the host GPU devices; a filesystem
  sandbox that hides `/dev/dri`/NVIDIA devices must be lifted for this measurement. The browser
  tests' SwiftShader numbers (and explicit `--gpu soft` profiler runs) are not a hardware performance baseline.

  The existing in-game Performance stats toggle shows smoothed CPU world/light/overlay times
  and movement/heat/rest per simulation tick. Detailed renderer clocks are off by default
  (`PERF_MARKS = false`); the raw frame hook (`PERF_FRAME`) is also off outside the profiler.
  `tools/server.py` shares the isolation/no-cache headers with `tools/serve.py`, and supplies a
  free-port background server with cleanup for measurement tools.
- `tools/experiments/factory_cost.js`: what running factories cost the simulation.
- `tools/experiments/flow.js`: belt and screw conveyor throughput in grains a second.
- `tools/experiments/worldgen.js`: material counts and timings of world generation.

WebGPU Phase 2a: Colour measures the CPU override fill (including loose objects and ghosts).
`frame_ms` is the complete paused-frame CPU call (including overlays and UI).
Upload measures strided visible-array writes and the override/light uploads, and Blit measures
GPU command submission. `override_cells` records the selected CPU cells; GPU timing is a
completed timestamp sample for that configuration across colour, bloom and drawing when `timestamp-query` is
available. The timestamp excludes uploads and the CPU lighting pass. Lighting remains on the
CPU. The GPU canvas draws the world; the transparent game canvas holds the existing overlays.

The comparison fixture is `tools/scenes/t88-renderer.js`. Capture with `--freeze --zoom 12`
(and 6 or 2), `--viewport 1920x1080`, and each `--renderer`. It includes terrain, water, fire,
magma, fog, a tree, flat/slope belts, screws, pipe, mold, bar and blueprint ghost.