deepgrain-foundry/docs/TESTING.md

# Testing

There are three layers. Use all three.

1. **Headless suites** (`test/t_*.js`) run the simulation in Node with no browser. They are fast
   and deterministic, and they are where most checks live.
2. **Browser tests** (`test/play.py`, `test/t_smooth.py`) drive the real page in headless
   Chromium with real key, mouse and gamepad input. They check the UI, saving, the smooth look,
   and that the page throws no errors.
3. **Screenshots** (`tools/shot.py`) show what something looks like. Open the PNG and look at it.

Run everything with `python3 test/run_all.py --browser`. It exits non-zero if anything failed and
prints the failing checks.

## Headless suites

For t69, `node test/t_screw_heat.js` checks full-cost repair, load preservation, undo,
all screw directions, the unchanged melt threshold, and saved damage/message state.
`python3 test/t_screw_heat.py` is the focused browser check for both looks, hover,
first-melt message, fog, and real click/G repair with keyboard undo. It saves
`shots/t69-{pixel,smooth}.png` (cold tube, orange heat warning, cooled fractured tube).

Each suite is a plain Node script:

```js
// What this suite covers, in a sentence or two.
const { load, suite, flatWorld, rect, count, ticks } = require('./harness');
const { run } = load();                 // loads src/sim/ (the files in src/manifest.json)
const { check, done } = suite();

flatWorld(run, 240, 160, 100);          // 240x160 grains, sky above row 100, stone below
run('money = 5000; bot.x = 60; bot.y = 40;');
run('for (let bx = 20; bx < 30; bx++) placeBlock(BP_BELT, 1, bx, 20, true);'); // true: free, no reach check
rect(run, 84, 60, 95, 79, run('SAND'));
ticks(run, 300);
check('sand rides the belt off its end', count(run, run('SAND'), 120, 80, 239, 99) > 100, count(run, run('SAND')));
done();                                 // prints ALL PASS or N FAILED, sets the exit code
```

- `run(code)` evaluates JavaScript inside the simulation and returns the result. Every top-level
  name in `src/sim/` is available. Return plain values (numbers, strings, `JSON.stringify(...)`).
- Harness helpers:
  - `flatWorld(run, w, h, ground)`;
  - `put(run, x, y, m)`;
  - `rect(run, x0, y0, x1, y1, m)`;
  - `count(run, m, x0, y0, x1, y1)`;
  - `maxTemp(run, ...)`;
  - `ticks(run, n)`;
  - `dump(run, x0, y0, x1, y1, withTemp)` for an ASCII picture of an area.
- `check(name, ok, detail)`: name the behaviour the player would notice ("a pump moves water
  over a wall"). Put the measured numbers in `detail`, so a failure shows how far off it was.
- Coordinates are in grains. Block coordinates (for `placeBlock`, `placeMachineAt`) are grains
  divided by 4.
- Errors name the real file and line (`src/sim/pipes.js:42`).
- A new suite named `test/t_<topic>.js` is picked up by `run_all.py` automatically.

**Determinism.** The simulation's randomness comes from seeded generators. The same code and the
same test give outputs identical across thread counts and between runs, apart from lines with timings in ms. To prove a
refactor changed nothing, save every suite's output before and after, and diff them:

```sh
for t in test/t_*.js; do node $t > /tmp/before/$(basename $t).txt; done   # before the change
for t in test/t_*.js; do node $t > /tmp/after/$(basename $t).txt; done    # after
diff -r /tmp/before /tmp/after
```

`test/t_threads.js` runs the real shared-memory scheduler inline and with 1, 2 and 8
Node workers, including repeats, comparing all mutable cell/chunk bytes and JS side effects.
The browser creates up to `min(16, hardwareConcurrency - 2, CPU workers setting)` helpers.
Small passes run inline; larger passes use batches to amortize dispatch. Wakes and heat flags
merge after each pass; long liquid searches are deferred until that merge. Global machine and
weather randomness remains separate from the chunk streams. World generation uses the same
inline schedule while settling. The t88 re-baseline changes worldgen material hashes and the
seeded sand pile bounds in `t_corner_seal.js`; the heat oracle now supports pass outputs.
The conveyor fixture uses a wider feed pile, the screen fixture warms longer, and `t_load.js`
checks waking within eight ticks separately from the probabilistic consumption of fuel.

`test/t_lakes.js` pins SHA-256 over raw `mat` followed by Float32 `temp` bytes for seeds
7, 42 and 424242 at 2048x1024; `t_corner_seal.js` pins three smaller worlds and `t_world.js`
the shallow-ore hash, all re-baselined for the t96 lake rebuild (changes only inside lake spans).
For seeds 7, 42, 424242, 2, 1, 5 and 6 (and Huge seed 766733 without ticks) every lake is at
least 12 deep with one at least 120 wide, exactly flat, held by a dry bank at or above its
surface on both sides, clay-bedded, with no ground step over 3 grains within 40 columns, clear
of the dock, with no submerged tree base; after 3000 ticks each lake keeps its exact surface and
water count (no leaks). A hand-made surface checks `spillLevels`.
`python3 test/t_lakes.py` checks pixel startup and fresh/reloaded lakes after 600 ticks with
two browser helpers and no page errors; `shots/t96-level.png` shows the new lake.

`test/t_bot_rescue.js` covers deep basalt burial, nearest/upward and diagonal nudges,
unexplored-pocket rejection, newest open trail points, ring wrap/load reset, bounded searches,
and flight through rock stopping at open space. Run `python3 test/t_bot_rescue.py` for the
focused pixel-browser check: both rescue messages and real keyboard flight out of basalt;
screenshots go to `shots/t72-{retrace,escape,open}.png`.

## Browser tests

- They need the `.venv` from README.md → Develop; plain `python3` switches to it on its own.
- `test/play.py` (about 2.5 minutes) builds a world through the title screen and plays it with real
  input. It checks the tools, building, copy/cut/paste, undo, the build menu, filters, pipes,
  the notebook and market, saving and continuing, exporting and importing a world file, the
  gamepad and world sizes. Screenshots go to
  `shots/play/`. Pixel look by default; `python3 test/play.py shots/play smooth` uses the smooth
  look.
- `test/t_vacuum_collapse.py` cuts settled stone and mixed heaps with one normal-width
  vacuum stroke per simulation tick in the real frame loop. It uses the shared
  `test/pile-cut.js` fixture, requires actual helper dispatch (forced for this small world),
  checks both faces within 180 ticks, and saves time-series pixel screenshots in `shots/`.
  `test/t_vacuum_collapse.js` uses the same cuts inline and with Node helpers, including
  bot-near/far, grain conservation, equal material bytes and forced-awake resting stability.
  It also checks a broad 120-grain wall within 600 ticks and a spill stopped by cleats
  across chunk edges, where the source chunk's one-grain barrier flag is clear.
- `test/t_blow_collapse.py` continuously blows bagged sand straight above a stationary bot
  at nozzle radius 6 for 1200 real frame-loop ticks, then checks the pile after 480 idle
  ticks with actual helper dispatch. Pixel captures are `shots/t93c-before.png` and
  `shots/t93c-after.png`. `test/t_blow_collapse.js` repeats the recipe inline and with
  1/2/8 Node helpers, compares material and motion flags, conserves all bag/flying/landed
  sand, and forces visits to check resting stability. It also directly checks a supported
  face releasing immediately and a spill stopped by a cleat across a chunk boundary.
- `test/t_smooth.py` checks WebGPU smooth shapes, seams, joins, fog and the thermal lens.
- `test/t_gpu_golden.py` compares 28 cropped frozen reference PNGs (384x288, under
  `test/fixtures/golden/`) against WebGPU: renderer, joins, sky and four light scenes,
  both looks at zoom 12 and 2. ImageMagick decodes RGB; the per-channel tolerance is 8,
  mean RGB error must stay below 0.5/255 and fewer than 0.3% of pixels may exceed 8
  (smooth AO/edge rounding). Seven smooth crops were refreshed for t91 (renderer 12/2,
  joins 2, sky 12/2, lamp 12, magma 12): the old noise had seams in the references too.
  The remaining crops are frozen pre-removal legacy images. The compressed
  `golden-world-640x*.json.gz` fixtures retain the Phase 2 world bytes that produced them,
  isolating renderer checks from worldgen RNG changes; t96 round 3 added the old `surfY`, `dirtY` and `AMB`
  columns too, because the renderer reads them and lake generation changed them. Inspect the matching
  `shots/golden-*.png` after visible changes.
- `test/t_gpu.py` checks default limits (four storage textures per stage), dirty-upload
  readback after 240 busy ticks with scrolling and both looks, exact dirty/full image equality,
  noise continuity and pan/zoom equality against fresh view textures/full uploads with a
  frozen clock, world replacement, transparent overlays, drill contours, shader compilation
  timing, successful device recreation and failed recovery with an autosave and Reload. `gpuCheckUploads()` reads actual resident GPU texture bytes and
  compares them to the full CPU arrays, including Float32 temperatures, machine and block data.
- `test/t_render_dirty.js` checks invalidation of sleeping chunks, heat edges, swaps, fog and trees.
- The old CPU light checks now run as `test/t_gpu_light.py` / `test/light-cases.js`, reading the
  actual WGSL float light grid. Transmission tolerance is 0.001 because the GPU work textures
  store float16 (the removed CPU implementation used float32). Behavioural expectations remain.
- Browser suites run one at a time; concurrent GPU work distorts timing and can starve tests.
- Rebuild first (`python3 build.py`); `run_all.py --browser` rebuilds for you. All browser tools
  use `tools/browser.py` and `tools/server.py`: a free-port background HTTP server serves the
  built page with COOP/COEP headers and shuts down after the browser. No `file://` loads.
- Choose Chromium graphics with `--gpu real|soft` or `DEEPGRAIN_GPU=real|soft` (the command line
  wins). This works for the suite runner, individual browser tests, screenshots and profiler.
  Real mode uses ANGLE Vulkan on the host GPU; soft uses SwiftShader.
  Unsupported software adapters show the same startup notice. Firefox uses its own default graphics backend.
  The default is **real**, including the profiler and `caps.py`. No colour tolerances were
  changed. The play test holds the slow-save fixture open during its screenshot, waits for
  tool/selection UI updates, disables camera lead after testing it, and gives the gamepad drill
  a known reachable target; these checks do not depend on GPU speed or earlier digging.
- `python3 tools/caps.py` prints isolation, actual WebGPU adapter info and the GL renderer for
  Chromium and installed Firefox. Real mode requires a hardware Chromium adapter.
- `test/t_caps.py` checks absent, null, rejected and insufficient-limit adapters, notices and
  the mandatory WebGPU startup gate. Cross-origin isolation is required for threads. Firefox smoke
  checks that a browser without WebGPU shows the notice and cannot start a world.
- Install Firefox with `.venv/bin/python -m playwright install firefox`.
  `python3 test/firefox_smoke.py` checks isolation and the WebGPU notice, starts a pixel world
  only when WebGPU is available, and otherwise verifies the startup block. `run_all.py --browser` includes it and prints a
  skip note if Firefox is absent; an installed browser that fails to launch is a failure.
- Linux WebGPU currently needs Firefox Nightly; the notice points to `about:config` →
  `dom.webgpu.enabled`. Sources: [Mozilla's experimental features](https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Experimental_features#webgpu_api)
  and [GPUWeb implementation status](https://github.com/gpuweb/gpuweb/wiki/Implementation-Status#firefox).

## Screenshots: tools/shot.py

```sh
python3 tools/shot.py shots/silo.png --setup tools/scenes/clear-sky.js \
    --js "$(cat tools/scenes/stone-uses.js)" --ticks 240 --zoom 7 --no-light
```

- It starts a new world, pauses it at once, runs your setup code in the page, runs `--ticks`
  simulation steps, then saves the picture. Because the world is paused otherwise, the same
  options always give the same picture.
- Setup code can use every game global, plus `SX`, `SY` (the bot's starting position).
  `tools/scenes/clear-sky.js` clears a flat work area and sets `FLOOR`, `BX`, `BY` and `CAM`. Set
  `CAM = [x, y]` to frame the shot.
- The HUD and the cursor previews are hidden unless you pass `--hud` or `--previews`.
- Both looks require WebGPU; there is no renderer switch or legacy fallback.
- `test/t_gpu.py` checks GPU uploads, dirty/full equality, contours, resize, timing and device loss.
  World-pixel tests use `worldPixelContext()` to read both stacked canvases. The helper
  presents the retained GPU result again before copying, because canvas textures expire after
  presentation ([WebGPU specification](https://gpuweb.github.io/gpuweb/#abstract-opdef-expire-the-current-texture),
  [Mozilla implementation notes](https://bugzilla.mozilla.org/show_bug.cgi?id=1965377)). Tests that change
  static `noiseB` in place set `GPU.world = null` before drawing to upload their new fixture.
- `--look smooth` for the smooth look. Test in the pixel look unless you are checking graphics.
- `--freeze` draws the last frame at a fixed time and frame count, so animations are caught at
  the same moment. Two shots of the same scene then match pixel for pixel, and a before / after
  pair differs only where something really changed. A quick comparison with Pillow:

  ```python
  from PIL import Image, ImageChops
  a, b = (Image.open(p).convert('RGB') for p in ('before.png', 'after.png'))
  print(ImageChops.difference(a, b).getbbox())   # None: identical
  ```

  To compare against an older version, build it somewhere else (for example from a git
  worktree) and pass that page with `--page`.
- Page errors are printed with their source file and line, and the exit status is 1.
- Look at the picture. A shot that "ran without errors" can still show the wrong thing.

## Pitfalls

- **GPU access in headless Chromium.** Real mode needs access to GPU devices and driver caches;
  a filesystem sandbox can silently force software rendering. Use `tools/caps.py` to check.
  Soft mode is available for machines without a GPU. Both modes load isolated HTTP pages.
- **Top-level `const` and `let` are not `window` properties.** In the page, `window.SM` is
  `undefined` even though `SM` exists. Write `typeof SM !== 'undefined'`, or use the bare name.
  Functions declared with `function` are on `window`.
- **Playwright calls a string that evaluates to a function.** `page.evaluate("x = function () {}")`
  assigns the function and then calls it. Wrap statements in an arrow function:
  `page.evaluate("() => { ... }")`.
- **Starting a world takes a moment.** `startNewWorld(seed, w, h)` generates on a timer. Wait
  for `GAME.running === true`. Then remove the welcome card if it is in the way:
  `document.querySelectorAll('.card, .toast').forEach(e => e.remove())`.
- **Real-time play is not deterministic.** In the browser the simulation runs as fast as frames
  arrive. For repeatable results, pause (`GAME.paused = true`) and call `simTick()` yourself, as
  `tools/shot.py` does.
- **World sizes** offered by the game: 2048×1024, 3072×1280, 4096×1536 grains.
- **Rebuild after every source change** before any browser test or screenshot. They all load
  `dist/index.html`.

## Measuring

- `tools/perf.py`: the phase 0 CPU/frame profiler, served over HTTP with COOP/COEP isolation.
  Rebuilds automatically. Run on an otherwise idle machine, without another browser test:

  ```sh
  python3 tools/perf.py --json tools/perf-baseline.json
  python3 tools/perf.py --json /tmp/perf-now.json --compare tools/perf-baseline.json
  python3 tools/perf.py --viewport 1920x1080 --world 2048x1024 --zooms 12,6,4,3,2 \
      --looks pixel,smooth --scenes settled,factory,heat --seconds 10 --json /tmp/perf.json
  python3 tools/test_perf.py
  ```

  The default matrix is three scenes × two looks × five zooms × lighting full/off. The world
  seed is 424242; fog is off so all visible grains are shaded. Settings otherwise use game
  defaults (including glow and the minimap), with autosave, camera lead and shake disabled.
  `settled` runs 600 ticks; `factory` reuses the furnace/transport layout from
  `experiments/factory_cost.js`, adding screws, a sand filter and full silos for sustained sand
  and ore flow; `heat` places water over magma beside a rock floor. The latter two settle for
  another 60 ticks. Each rebuild resets the power-rebuild scheduler, weather random stream and all statistics to
  fresh-page values; fixture hashes are checked by `test/t_perf.js`. No save/load snapshot is used: it would quantize heat and land flyers.

  **World** is the whole `drawWorld` CPU call. **Prep** sets up the view/masks/joins;
  **Colour** fills CPU rare-cell overrides (including ghosts and flyers); **Upload** writes
  dirty world chunks and override textures; **Blit** submits GPU drawing and composition.
  These are CPU wall times, including driver stalls; they are not GPU timestamp measurements.
  Bloom is computed on the GPU. GPU light timestamps are recorded separately as `gpuLight`.
  Overlay totals and each individual draw call are in JSON, with ms/frame and invocation counts; conditional
  overlays absent from the fixture have zero calls, not a measured zero execution cost.

  Five calls warm each render configuration before 30 measured calls. Simulation warms five
  full 120-tick periods on a disposable scene, then rebuilds the same seeded fixture and
  measures 240 consecutive ticks. Every direct `simTick` callee is discovered from its source;
  JSON records per-call and amortized per-tick costs and invocation counts, including periodic
  functions. Inactive paths (e.g. load wake, saplings, flow tracking) have zero calls and null
  per-call cost. Unattributed time includes tick bookkeeping and clock/wrapper overhead.
  Awake/hot chunk counts are recorded at the start/end of simulation and real runs.

  Real-loop tests independently rebuild each scene for zooms 2 and 6 in both requested looks,
  regardless of `--zooms`. `--seconds` applies to each of these twelve default runs. Raw samples
  retain frame interval (rAF timestamps, including waiting/scheduling), CPU frame work, total
  simulation work per frame and number of ticks. p50/p95/p99 use interpolated sorted samples;
  five warm-up frames are excluded. Zero-tick frames correctly have zero simulation time.
  JSON records a fixed render scale of 1 for both looks.
  Comparisons accept subsets of the baseline scenes, looks and zooms, with identical world/viewport dimensions and run duration, print all timing changes, and mark >10% regressions;
  tiny timings and tails can be noisy, and a zero baseline has no meaningful percent change.

  Performance runs launch Chromium with
  `--use-angle=vulkan --enable-features=Vulkan,VulkanFromANGLE --enable-gpu --ignore-gpu-blocklist --enable-gpu-rasterization --enable-unsafe-webgpu`.
  They record the unmasked GPU/renderer, cores, browser version, viewport/world size,
  `crossOriginIsolated`, `navigator.gpu`, source hash and elapsed time. Real mode refuses software
  renderers; both modes require isolation. Chromium needs access to the host GPU devices; a filesystem
  sandbox that hides `/dev/dri`/NVIDIA devices must be lifted for this measurement. The browser
  tests' SwiftShader numbers (and explicit `--gpu soft` profiler runs) are not a hardware performance baseline.

  The existing in-game Performance stats toggle shows smoothed CPU world/light/overlay times
  and movement/heat/rest per simulation tick. Detailed renderer clocks are off by default
  (`PERF_MARKS = false`); the raw frame hook (`PERF_FRAME`) is also off outside the profiler.
  `tools/server.py` shares the isolation/no-cache headers with `tools/serve.py`, and supplies a
  free-port background server with cleanup for measurement tools.
- `tools/experiments/factory_cost.js`: what running factories cost the simulation.
- `tools/experiments/flow.js`: belt and screw conveyor throughput in grains a second.
- `tools/experiments/worldgen.js`: material counts and timings of world generation.

Both looks use WebGPU. Colour measures the CPU rare-cell override fill, including loose
objects and ghosts. `frame_ms` is the complete paused-frame CPU call, with overlays and UI.
Upload measures dirty world chunks, override buffers and small lighting tables; Blit measures
GPU command submission. `override_cells` counts CPU cells. GPU timestamp samples include
colour, lighting, bloom and drawing, excluding uploads. CPU lighting is gone. The GPU canvas
draws the world; the transparent game canvas holds overlays. Override colours and glow live in
plain typed arrays, without offscreen Canvas2D backing stores.

Use `tools/scenes/t88-renderer.js` with `--freeze --zoom 12` (or 6/2),
`--viewport 1920x1080` and `--look pixel|smooth`. It includes terrain, water, fire, magma,
fog, a tree, flat/slope belts, screws, pipe, mold, bar and blueprint ghost. Frozen legacy
comparisons are in `test/t_gpu_golden.py`; no legacy renderer remains in the game.

The smooth shader retains the nine-tap glow filter and samples field AO at mip level 2.2;
three render passes generate field levels 1-3, including non-power-of-two view sizes.
Only smooth uploads back/tree/field overrides and a separate presence mask. Profiler scale is
always 1. The screenshot tool invalidates the parallax horizon after setup changes `surfY`.
Mip pass references: [Chrome's porting guide](https://developer.chrome.com/docs/web-platform/webgpu/from-webgl-to-webgpu#mipmap_generation)
and [WebGPU Fundamentals](https://webgpufundamentals.org/webgpu/lessons/webgpu-importing-textures.html#generating-mips-on-the-gpu).

`perfSim` reports synchronous tick wall time (also occupied main-thread time), polling in
`threadWait`, and main compute excluding polling. The latter includes serial simulation,
dispatch/merge and the main thread's own chunk batch; it is not pure scheduling overhead.
Final t88 full-matrix results are saved in `tools/perf-after-t88.json`; keep
`tools/perf-baseline.json` unchanged as the original comparison.

The required pipeline limits are checked in `gpuLimitsOK` before requesting the device: four
storage textures, sixteen sampled textures and eight storage buffers per stage, plus 8192 for
2D textures; no non-default limit is requested. References: [WebGPU limits](https://www.w3.org/TR/2026/CRD-webgpu-20260109/#limits)
and [Chromium Dawn defaults](https://dawn.googlesource.com/dawn.git/+/refs/heads/chromium/7945/src/dawn/native/Limits.cpp).
All game pipelines use async creation during capability checking. World loading submits both
looks before `GAME.running` becomes true. Shader and loading timings are printed by `t_gpu.py`;
look/zoom switches must not call synchronous pipeline creation.