278 lines
19 KiB
Markdown
278 lines
19 KiB
Markdown
# Testing
|
||
|
||
There are three layers. Use all three.
|
||
|
||
1. **Headless suites** (`test/t_*.js`) run the simulation in Node with no browser. They are fast
|
||
and deterministic, and they are where most checks live.
|
||
2. **Browser tests** (`test/play.py`, `test/t_smooth.py`) drive the real page in headless
|
||
Chromium with real key, mouse and gamepad input. They check the UI, saving, the smooth look,
|
||
and that the page throws no errors.
|
||
3. **Screenshots** (`tools/shot.py`) show what something looks like. Open the PNG and look at it.
|
||
|
||
Run everything with `python3 test/run_all.py --browser`. It exits non-zero if anything failed and
|
||
prints the failing checks.
|
||
|
||
## Headless suites
|
||
|
||
Each suite is a plain Node script:
|
||
|
||
```js
|
||
// What this suite covers, in a sentence or two.
|
||
const { load, suite, flatWorld, rect, count, ticks } = require('./harness');
|
||
const { run } = load(); // loads src/sim/ (the files in src/manifest.json)
|
||
const { check, done } = suite();
|
||
|
||
flatWorld(run, 240, 160, 100); // 240x160 grains, sky above row 100, stone below
|
||
run('money = 5000; bot.x = 60; bot.y = 40;');
|
||
run('for (let bx = 20; bx < 30; bx++) placeBlock(BP_BELT, 1, bx, 20, true);'); // true: free, no reach check
|
||
rect(run, 84, 60, 95, 79, run('SAND'));
|
||
ticks(run, 300);
|
||
check('sand rides the belt off its end', count(run, run('SAND'), 120, 80, 239, 99) > 100, count(run, run('SAND')));
|
||
done(); // prints ALL PASS or N FAILED, sets the exit code
|
||
```
|
||
|
||
- `run(code)` evaluates JavaScript inside the simulation and returns the result. Every top-level
|
||
name in `src/sim/` is available. Return plain values (numbers, strings, `JSON.stringify(...)`).
|
||
- Harness helpers:
|
||
- `flatWorld(run, w, h, ground)`;
|
||
- `put(run, x, y, m)`;
|
||
- `rect(run, x0, y0, x1, y1, m)`;
|
||
- `count(run, m, x0, y0, x1, y1)`;
|
||
- `maxTemp(run, ...)`;
|
||
- `ticks(run, n)`;
|
||
- `dump(run, x0, y0, x1, y1, withTemp)` for an ASCII picture of an area.
|
||
- `check(name, ok, detail)`: name the behaviour the player would notice ("a pump moves water
|
||
over a wall"). Put the measured numbers in `detail`, so a failure shows how far off it was.
|
||
- Coordinates are in grains. Block coordinates (for `placeBlock`, `placeMachineAt`) are grains
|
||
divided by 4.
|
||
- Errors name the real file and line (`src/sim/pipes.js:42`).
|
||
- A new suite named `test/t_<topic>.js` is picked up by `run_all.py` automatically.
|
||
|
||
**Determinism.** The simulation's randomness comes from seeded generators. The same code and the
|
||
same test give outputs identical across thread counts and between runs, apart from lines with timings in ms. To prove a
|
||
refactor changed nothing, save every suite's output before and after, and diff them:
|
||
|
||
```sh
|
||
for t in test/t_*.js; do node $t > /tmp/before/$(basename $t).txt; done # before the change
|
||
for t in test/t_*.js; do node $t > /tmp/after/$(basename $t).txt; done # after
|
||
diff -r /tmp/before /tmp/after
|
||
```
|
||
|
||
`test/t_threads.js` runs the real shared-memory scheduler inline and with 1, 2 and 8
|
||
Node workers, including repeats, comparing all mutable cell/chunk bytes and JS side effects.
|
||
The browser creates up to `min(16, hardwareConcurrency - 2, CPU workers setting)` helpers.
|
||
Small passes run inline; larger passes use batches to amortize dispatch. Wakes and heat flags
|
||
merge after each pass; long liquid searches are deferred until that merge. Global machine and
|
||
weather randomness remains separate from the chunk streams. World generation uses the same
|
||
inline schedule while settling. The t88 re-baseline changes worldgen material hashes and the
|
||
seeded sand pile bounds in `t_corner_seal.js`; the heat oracle now supports pass outputs.
|
||
The conveyor fixture uses a wider feed pile, the screen fixture warms longer, and `t_load.js`
|
||
checks waking within eight ticks separately from the probabilistic consumption of fuel.
|
||
|
||
## Browser tests
|
||
|
||
- They need the `.venv` from README.md → Develop; plain `python3` switches to it on its own.
|
||
- `test/play.py` (about 2.5 minutes) builds a world through the title screen and plays it with real
|
||
input. It checks the tools, building, copy/cut/paste, undo, the build menu, filters, pipes,
|
||
the notebook and market, saving and continuing, exporting and importing a world file, the
|
||
gamepad and world sizes. Screenshots go to
|
||
`shots/play/`. Pixel look by default; `python3 test/play.py shots/play smooth` uses the smooth
|
||
look.
|
||
- `test/t_smooth.py` checks WebGPU smooth shapes, seams, joins, fog and the thermal lens.
|
||
- `test/t_gpu_golden.py` compares 28 cropped legacy reference PNGs (384x288, under
|
||
`test/fixtures/golden/`) against WebGPU: renderer, joins, sky and four light scenes,
|
||
both looks at zoom 12 and 2. ImageMagick decodes RGB; the per-channel tolerance is 8,
|
||
mean RGB error must stay below 0.5/255 and fewer than 0.3% of pixels may exceed 8
|
||
(smooth AO/edge rounding). These are frozen pre-removal legacy images, not generated
|
||
from the new renderer. The compressed `golden-world-640x*.json.gz` fixtures retain the
|
||
Phase 2 world bytes that produced them, isolating renderer checks from worldgen RNG changes. Inspect the matching `shots/golden-*.png` after visible changes.
|
||
- `test/t_gpu.py` checks default limits (four storage textures per stage), dirty-upload
|
||
readback after 240 busy ticks with scrolling and both looks, exact dirty/full image equality
|
||
with a frozen clock, world replacement, transparent overlays, drill contours, shader compilation timing, successful device recreation and failed recovery with
|
||
an autosave and Reload. `gpuCheckUploads()` reads actual resident GPU texture bytes and
|
||
compares them to the full CPU arrays, including Float32 temperatures, machine and block data.
|
||
- `test/t_render_dirty.js` checks invalidation of sleeping chunks, heat edges, swaps, fog and trees.
|
||
- The old CPU light checks now run as `test/t_gpu_light.py` / `test/light-cases.js`, reading the
|
||
actual WGSL float light grid. Transmission tolerance is 0.001 because the GPU work textures
|
||
store float16 (the removed CPU implementation used float32). Behavioural expectations remain.
|
||
- Browser suites run one at a time; concurrent GPU work distorts timing and can starve tests.
|
||
- Rebuild first (`python3 build.py`); `run_all.py --browser` rebuilds for you. All browser tools
|
||
use `tools/browser.py` and `tools/server.py`: a free-port background HTTP server serves the
|
||
built page with COOP/COEP headers and shuts down after the browser. No `file://` loads.
|
||
- Choose Chromium graphics with `--gpu real|soft` or `DEEPGRAIN_GPU=real|soft` (the command line
|
||
wins). This works for the suite runner, individual browser tests, screenshots and profiler.
|
||
Real mode uses ANGLE Vulkan on the host GPU; soft uses SwiftShader.
|
||
Unsupported software adapters show the same startup notice. Firefox uses its own default graphics backend.
|
||
The default is **real**, including the profiler and `caps.py`. No colour tolerances were
|
||
changed. The play test holds the slow-save fixture open during its screenshot, waits for
|
||
tool/selection UI updates, disables camera lead after testing it, and gives the gamepad drill
|
||
a known reachable target; these checks do not depend on GPU speed or earlier digging.
|
||
- `python3 tools/caps.py` prints isolation, actual WebGPU adapter info and the GL renderer for
|
||
Chromium and installed Firefox. Real mode requires a hardware Chromium adapter.
|
||
- `test/t_caps.py` checks absent, null, rejected and insufficient-limit adapters, notices and
|
||
the mandatory WebGPU startup gate. Cross-origin isolation is required for threads. Firefox smoke
|
||
checks that a browser without WebGPU shows the notice and cannot start a world.
|
||
- Install Firefox with `.venv/bin/python -m playwright install firefox`.
|
||
`python3 test/firefox_smoke.py` checks isolation and the WebGPU notice, starts a pixel world
|
||
only when WebGPU is available, and otherwise verifies the startup block. `run_all.py --browser` includes it and prints a
|
||
skip note if Firefox is absent; an installed browser that fails to launch is a failure.
|
||
- Linux WebGPU currently needs Firefox Nightly; the notice points to `about:config` →
|
||
`dom.webgpu.enabled`. Sources: [Mozilla's experimental features](https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Experimental_features#webgpu_api)
|
||
and [GPUWeb implementation status](https://github.com/gpuweb/gpuweb/wiki/Implementation-Status#firefox).
|
||
|
||
## Screenshots: tools/shot.py
|
||
|
||
```sh
|
||
python3 tools/shot.py shots/silo.png --setup tools/scenes/clear-sky.js \
|
||
--js "$(cat tools/scenes/stone-uses.js)" --ticks 240 --zoom 7 --no-light
|
||
```
|
||
|
||
- It starts a new world, pauses it at once, runs your setup code in the page, runs `--ticks`
|
||
simulation steps, then saves the picture. Because the world is paused otherwise, the same
|
||
options always give the same picture.
|
||
- Setup code can use every game global, plus `SX`, `SY` (the bot's starting position).
|
||
`tools/scenes/clear-sky.js` clears a flat work area and sets `FLOOR`, `BX`, `BY` and `CAM`. Set
|
||
`CAM = [x, y]` to frame the shot.
|
||
- The HUD and the cursor previews are hidden unless you pass `--hud` or `--previews`.
|
||
- Both looks require WebGPU; there is no renderer switch or legacy fallback.
|
||
- `test/t_gpu.py` checks GPU uploads, dirty/full equality, contours, resize, timing and device loss.
|
||
World-pixel tests use `worldPixelContext()` to read both stacked canvases. The helper
|
||
presents the retained GPU result again before copying, because canvas textures expire after
|
||
presentation ([WebGPU specification](https://gpuweb.github.io/gpuweb/#abstract-opdef-expire-the-current-texture),
|
||
[Mozilla implementation notes](https://bugzilla.mozilla.org/show_bug.cgi?id=1965377)). Tests that change
|
||
static `noiseB` in place set `GPU.world = null` before drawing to upload their new fixture.
|
||
- `--look smooth` for the smooth look. Test in the pixel look unless you are checking graphics.
|
||
- `--freeze` draws the last frame at a fixed time and frame count, so animations are caught at
|
||
the same moment. Two shots of the same scene then match pixel for pixel, and a before / after
|
||
pair differs only where something really changed. A quick comparison with Pillow:
|
||
|
||
```python
|
||
from PIL import Image, ImageChops
|
||
a, b = (Image.open(p).convert('RGB') for p in ('before.png', 'after.png'))
|
||
print(ImageChops.difference(a, b).getbbox()) # None: identical
|
||
```
|
||
|
||
To compare against an older version, build it somewhere else (for example from a git
|
||
worktree) and pass that page with `--page`.
|
||
- Page errors are printed with their source file and line, and the exit status is 1.
|
||
- Look at the picture. A shot that "ran without errors" can still show the wrong thing.
|
||
|
||
## Pitfalls
|
||
|
||
- **GPU access in headless Chromium.** Real mode needs access to GPU devices and driver caches;
|
||
a filesystem sandbox can silently force software rendering. Use `tools/caps.py` to check.
|
||
Soft mode is available for machines without a GPU. Both modes load isolated HTTP pages.
|
||
- **Top-level `const` and `let` are not `window` properties.** In the page, `window.SM` is
|
||
`undefined` even though `SM` exists. Write `typeof SM !== 'undefined'`, or use the bare name.
|
||
Functions declared with `function` are on `window`.
|
||
- **Playwright calls a string that evaluates to a function.** `page.evaluate("x = function () {}")`
|
||
assigns the function and then calls it. Wrap statements in an arrow function:
|
||
`page.evaluate("() => { ... }")`.
|
||
- **Starting a world takes a moment.** `startNewWorld(seed, w, h)` generates on a timer. Wait
|
||
for `GAME.running === true`. Then remove the welcome card if it is in the way:
|
||
`document.querySelectorAll('.card, .toast').forEach(e => e.remove())`.
|
||
- **Real-time play is not deterministic.** In the browser the simulation runs as fast as frames
|
||
arrive. For repeatable results, pause (`GAME.paused = true`) and call `simTick()` yourself, as
|
||
`tools/shot.py` does.
|
||
- **World sizes** offered by the game: 2048×1024, 3072×1280, 4096×1536 grains.
|
||
- **Rebuild after every source change** before any browser test or screenshot. They all load
|
||
`dist/index.html`.
|
||
|
||
## Measuring
|
||
|
||
- `tools/perf.py`: the phase 0 CPU/frame profiler, served over HTTP with COOP/COEP isolation.
|
||
Rebuilds automatically. Run on an otherwise idle machine, without another browser test:
|
||
|
||
```sh
|
||
python3 tools/perf.py --json tools/perf-baseline.json
|
||
python3 tools/perf.py --json /tmp/perf-now.json --compare tools/perf-baseline.json
|
||
python3 tools/perf.py --viewport 1920x1080 --world 2048x1024 --zooms 12,6,4,3,2 \
|
||
--looks pixel,smooth --scenes settled,factory,heat --seconds 10 --json /tmp/perf.json
|
||
python3 tools/test_perf.py
|
||
```
|
||
|
||
The default matrix is three scenes × two looks × five zooms × lighting full/off. The world
|
||
seed is 424242; fog is off so all visible grains are shaded. Settings otherwise use game
|
||
defaults (including glow and the minimap), with autosave, camera lead and shake disabled.
|
||
`settled` runs 600 ticks; `factory` reuses the furnace/transport layout from
|
||
`experiments/factory_cost.js`, adding screws, a sand filter and full silos for sustained sand
|
||
and ore flow; `heat` places water over magma beside a rock floor. The latter two settle for
|
||
another 60 ticks. Each rebuild resets the power-rebuild scheduler, weather random stream and all statistics to
|
||
fresh-page values; fixture hashes are checked by `test/t_perf.js`. No save/load snapshot is used: it would quantize heat and land flyers.
|
||
|
||
**World** is the whole `drawWorld` CPU call. **Prep** sets up the view/masks/joins;
|
||
**Colour** fills CPU rare-cell overrides (including ghosts and flyers); **Upload** writes
|
||
dirty world chunks and override textures; **Blit** submits GPU drawing and composition.
|
||
These are CPU wall times, including driver stalls; they are not GPU timestamp measurements.
|
||
Bloom is computed on the GPU. GPU light timestamps are recorded separately as `gpuLight`.
|
||
Overlay totals and each individual draw call are in JSON, with ms/frame and invocation counts; conditional
|
||
overlays absent from the fixture have zero calls, not a measured zero execution cost.
|
||
|
||
Five calls warm each render configuration before 30 measured calls. Simulation warms five
|
||
full 120-tick periods on a disposable scene, then rebuilds the same seeded fixture and
|
||
measures 240 consecutive ticks. Every direct `simTick` callee is discovered from its source;
|
||
JSON records per-call and amortized per-tick costs and invocation counts, including periodic
|
||
functions. Inactive paths (e.g. load wake, saplings, flow tracking) have zero calls and null
|
||
per-call cost. Unattributed time includes tick bookkeeping and clock/wrapper overhead.
|
||
Awake/hot chunk counts are recorded at the start/end of simulation and real runs.
|
||
|
||
Real-loop tests independently rebuild each scene for zooms 2 and 6 in both requested looks,
|
||
regardless of `--zooms`. `--seconds` applies to each of these twelve default runs. Raw samples
|
||
retain frame interval (rAF timestamps, including waiting/scheduling), CPU frame work, total
|
||
simulation work per frame and number of ticks. p50/p95/p99 use interpolated sorted samples;
|
||
five warm-up frames are excluded. Zero-tick frames correctly have zero simulation time.
|
||
JSON records a fixed render scale of 1 for both looks.
|
||
Comparisons accept subsets of the baseline scenes, looks and zooms, with identical world/viewport dimensions and run duration, print all timing changes, and mark >10% regressions;
|
||
tiny timings and tails can be noisy, and a zero baseline has no meaningful percent change.
|
||
|
||
Performance runs launch Chromium with
|
||
`--use-angle=vulkan --enable-features=Vulkan,VulkanFromANGLE --enable-gpu --ignore-gpu-blocklist --enable-gpu-rasterization --enable-unsafe-webgpu`.
|
||
They record the unmasked GPU/renderer, cores, browser version, viewport/world size,
|
||
`crossOriginIsolated`, `navigator.gpu`, source hash and elapsed time. Real mode refuses software
|
||
renderers; both modes require isolation. Chromium needs access to the host GPU devices; a filesystem
|
||
sandbox that hides `/dev/dri`/NVIDIA devices must be lifted for this measurement. The browser
|
||
tests' SwiftShader numbers (and explicit `--gpu soft` profiler runs) are not a hardware performance baseline.
|
||
|
||
The existing in-game Performance stats toggle shows smoothed CPU world/light/overlay times
|
||
and movement/heat/rest per simulation tick. Detailed renderer clocks are off by default
|
||
(`PERF_MARKS = false`); the raw frame hook (`PERF_FRAME`) is also off outside the profiler.
|
||
`tools/server.py` shares the isolation/no-cache headers with `tools/serve.py`, and supplies a
|
||
free-port background server with cleanup for measurement tools.
|
||
- `tools/experiments/factory_cost.js`: what running factories cost the simulation.
|
||
- `tools/experiments/flow.js`: belt and screw conveyor throughput in grains a second.
|
||
- `tools/experiments/worldgen.js`: material counts and timings of world generation.
|
||
|
||
Both looks use WebGPU. Colour measures the CPU rare-cell override fill, including loose
|
||
objects and ghosts. `frame_ms` is the complete paused-frame CPU call, with overlays and UI.
|
||
Upload measures dirty world chunks, override buffers and small lighting tables; Blit measures
|
||
GPU command submission. `override_cells` counts CPU cells. GPU timestamp samples include
|
||
colour, lighting, bloom and drawing, excluding uploads. CPU lighting is gone. The GPU canvas
|
||
draws the world; the transparent game canvas holds overlays. Override colours and glow live in
|
||
plain typed arrays, without offscreen Canvas2D backing stores.
|
||
|
||
Use `tools/scenes/t88-renderer.js` with `--freeze --zoom 12` (or 6/2),
|
||
`--viewport 1920x1080` and `--look pixel|smooth`. It includes terrain, water, fire, magma,
|
||
fog, a tree, flat/slope belts, screws, pipe, mold, bar and blueprint ghost. Frozen legacy
|
||
comparisons are in `test/t_gpu_golden.py`; no legacy renderer remains in the game.
|
||
|
||
The smooth shader retains the nine-tap glow filter and samples field AO at mip level 2.2;
|
||
three render passes generate field levels 1-3, including non-power-of-two view sizes.
|
||
Only smooth uploads back/tree/field overrides and a separate presence mask. Profiler scale is
|
||
always 1. The screenshot tool invalidates the parallax horizon after setup changes `surfY`.
|
||
Mip pass references: [Chrome's porting guide](https://developer.chrome.com/docs/web-platform/webgpu/from-webgl-to-webgpu#mipmap_generation)
|
||
and [WebGPU Fundamentals](https://webgpufundamentals.org/webgpu/lessons/webgpu-importing-textures.html#generating-mips-on-the-gpu).
|
||
|
||
`perfSim` reports synchronous tick wall time (also occupied main-thread time), polling in
|
||
`threadWait`, and main compute excluding polling. The latter includes serial simulation,
|
||
dispatch/merge and the main thread's own chunk batch; it is not pure scheduling overhead.
|
||
Final t88 full-matrix results are saved in `tools/perf-after-t88.json`; keep
|
||
`tools/perf-baseline.json` unchanged as the original comparison.
|
||
|
||
The required pipeline limits are checked in `gpuLimitsOK` before requesting the device: four
|
||
storage textures, sixteen sampled textures and eight storage buffers per stage, plus 8192 for
|
||
2D textures; no non-default limit is requested. References: [WebGPU limits](https://www.w3.org/TR/2026/CRD-webgpu-20260109/#limits)
|
||
and [Chromium Dawn defaults](https://dawn.googlesource.com/dawn.git/+/refs/heads/chromium/7945/src/dawn/native/Limits.cpp).
|
||
All game pipelines use async creation during capability checking. World loading submits both
|
||
looks before `GAME.running` becomes true. Shader and loading timings are printed by `t_gpu.py`;
|
||
look/zoom switches must not call synchronous pipeline creation.
|