215 lines
14 KiB
Markdown
215 lines
14 KiB
Markdown
# Testing
|
||
|
||
There are three layers. Use all three.
|
||
|
||
1. **Headless suites** (`test/t_*.js`) run the simulation in Node with no browser. They are fast
|
||
and deterministic, and they are where most checks live.
|
||
2. **Browser tests** (`test/play.py`, `test/t_smooth.py`) drive the real page in headless
|
||
Chromium with real key, mouse and gamepad input. They check the UI, saving, the smooth look,
|
||
and that the page throws no errors.
|
||
3. **Screenshots** (`tools/shot.py`) show what something looks like. Open the PNG and look at it.
|
||
|
||
Run everything with `python3 test/run_all.py --browser`. It exits non-zero if anything failed and
|
||
prints the failing checks.
|
||
|
||
## Headless suites
|
||
|
||
Each suite is a plain Node script:
|
||
|
||
```js
|
||
// What this suite covers, in a sentence or two.
|
||
const { load, suite, flatWorld, rect, count, ticks } = require('./harness');
|
||
const { run } = load(); // loads src/sim/ (the files in src/manifest.json)
|
||
const { check, done } = suite();
|
||
|
||
flatWorld(run, 240, 160, 100); // 240x160 grains, sky above row 100, stone below
|
||
run('money = 5000; bot.x = 60; bot.y = 40;');
|
||
run('for (let bx = 20; bx < 30; bx++) placeBlock(BP_BELT, 1, bx, 20, true);'); // true: free, no reach check
|
||
rect(run, 84, 60, 95, 79, run('SAND'));
|
||
ticks(run, 300);
|
||
check('sand rides the belt off its end', count(run, run('SAND'), 120, 80, 239, 99) > 100, count(run, run('SAND')));
|
||
done(); // prints ALL PASS or N FAILED, sets the exit code
|
||
```
|
||
|
||
- `run(code)` evaluates JavaScript inside the simulation and returns the result. Every top-level
|
||
name in `src/sim/` is available. Return plain values (numbers, strings, `JSON.stringify(...)`).
|
||
- Harness helpers:
|
||
- `flatWorld(run, w, h, ground)`;
|
||
- `put(run, x, y, m)`;
|
||
- `rect(run, x0, y0, x1, y1, m)`;
|
||
- `count(run, m, x0, y0, x1, y1)`;
|
||
- `maxTemp(run, ...)`;
|
||
- `ticks(run, n)`;
|
||
- `dump(run, x0, y0, x1, y1, withTemp)` for an ASCII picture of an area.
|
||
- `check(name, ok, detail)`: name the behaviour the player would notice ("a pump moves water
|
||
over a wall"). Put the measured numbers in `detail`, so a failure shows how far off it was.
|
||
- Coordinates are in grains. Block coordinates (for `placeBlock`, `placeMachineAt`) are grains
|
||
divided by 4.
|
||
- Errors name the real file and line (`src/sim/pipes.js:42`).
|
||
- A new suite named `test/t_<topic>.js` is picked up by `run_all.py` automatically.
|
||
|
||
**Determinism.** The simulation's randomness comes from seeded generators. The same code and the
|
||
same test give the same output every run, apart from lines with timings in ms. To prove a
|
||
refactor changed nothing, save every suite's output before and after, and diff them:
|
||
|
||
```sh
|
||
for t in test/t_*.js; do node $t > /tmp/before/$(basename $t).txt; done # before the change
|
||
for t in test/t_*.js; do node $t > /tmp/after/$(basename $t).txt; done # after
|
||
diff -r /tmp/before /tmp/after
|
||
```
|
||
|
||
## Browser tests
|
||
|
||
- They need the `.venv` from README.md → Develop; plain `python3` switches to it on its own.
|
||
- `test/play.py` (about 2.5 minutes) builds a world through the title screen and plays it with real
|
||
input. It checks the tools, building, copy/cut/paste, undo, the build menu, filters, pipes,
|
||
the notebook and market, saving and continuing, exporting and importing a world file, the
|
||
gamepad and world sizes. Screenshots go to
|
||
`shots/play/`. Pixel look by default; `python3 test/play.py shots/play smooth` uses the smooth
|
||
look.
|
||
- `test/t_smooth.py` checks that the WebGL2 smooth look compiles and draws, that it falls back
|
||
to pixels cleanly, and that the thermal lens works.
|
||
- Run them one at a time. Two headless browsers at once can starve each other, and
|
||
`t_smooth.py`'s half-resolution check then fails. `run_all.py` runs them one after the other.
|
||
- Rebuild first (`python3 build.py`); `run_all.py --browser` rebuilds for you. All browser tools
|
||
use `tools/browser.py` and `tools/server.py`: a free-port background HTTP server serves the
|
||
built page with COOP/COEP headers and shuts down after the browser. No `file://` loads.
|
||
- Choose Chromium graphics with `--gpu real|soft` or `DEEPGRAIN_GPU=real|soft` (the command line
|
||
wins). This works for the suite runner, individual browser tests, screenshots and profiler.
|
||
Real mode uses ANGLE Vulkan on the host GPU; soft uses SwiftShader and sets
|
||
`window.__allowSoftwareGL` before load. The explicit software-fallback test still uses soft
|
||
mode with that override off. Firefox uses its own default graphics backend.
|
||
The default is **real**, including the profiler and `caps.py`. No colour tolerances were
|
||
changed. The play test holds the slow-save fixture open during its screenshot, waits for
|
||
tool/selection UI updates, disables camera lead after testing it, and gives the gamepad drill
|
||
a known reachable target; these checks do not depend on GPU speed or earlier digging.
|
||
- `python3 tools/caps.py` prints isolation, actual WebGPU adapter info and the GL renderer for
|
||
Chromium and installed Firefox. Real mode requires a hardware Chromium adapter.
|
||
- `test/t_caps.py` checks missing, null and rejected WebGPU adapters, both title notices, warning
|
||
mode and the future `REQUIRE_GPU_AND_THREADS` startup gate. Its forced notice screenshot is
|
||
`shots/t88-p1-notice.png`.
|
||
- Install Firefox with `.venv/bin/python -m playwright install firefox`.
|
||
`python3 test/firefox_smoke.py` checks isolation and the WebGPU notice, starts a pixel world,
|
||
runs three seconds and checks page errors. `run_all.py --browser` includes it and prints a
|
||
skip note if Firefox is absent; an installed browser that fails to launch is a failure.
|
||
- Linux WebGPU currently needs Firefox Nightly; the notice points to `about:config` →
|
||
`dom.webgpu.enabled`. Sources: [Mozilla's experimental features](https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Experimental_features#webgpu_api)
|
||
and [GPUWeb implementation status](https://github.com/gpuweb/gpuweb/wiki/Implementation-Status#firefox).
|
||
|
||
## Screenshots: tools/shot.py
|
||
|
||
```sh
|
||
python3 tools/shot.py shots/silo.png --setup tools/scenes/clear-sky.js \
|
||
--js "$(cat tools/scenes/stone-uses.js)" --ticks 240 --zoom 7 --no-light
|
||
```
|
||
|
||
- It starts a new world, pauses it at once, runs your setup code in the page, runs `--ticks`
|
||
simulation steps, then saves the picture. Because the world is paused otherwise, the same
|
||
options always give the same picture.
|
||
- Setup code can use every game global, plus `SX`, `SY` (the bot's starting position).
|
||
`tools/scenes/clear-sky.js` clears a flat work area and sets `FLOOR`, `BX`, `BY` and `CAM`. Set
|
||
`CAM = [x, y]` to frame the shot.
|
||
- The HUD and the cursor previews are hidden unless you pass `--hud` or `--previews`.
|
||
- `--look smooth` for the smooth look. Test in the pixel look unless you are checking graphics.
|
||
- `--freeze` draws the last frame at a fixed time and frame count, so animations are caught at
|
||
the same moment. Two shots of the same scene then match pixel for pixel, and a before / after
|
||
pair differs only where something really changed. A quick comparison with Pillow:
|
||
|
||
```python
|
||
from PIL import Image, ImageChops
|
||
a, b = (Image.open(p).convert('RGB') for p in ('before.png', 'after.png'))
|
||
print(ImageChops.difference(a, b).getbbox()) # None: identical
|
||
```
|
||
|
||
To compare against an older version, build it somewhere else (for example from a git
|
||
worktree) and pass that page with `--page`.
|
||
- Page errors are printed with their source file and line, and the exit status is 1.
|
||
- Look at the picture. A shot that "ran without errors" can still show the wrong thing.
|
||
|
||
## Pitfalls
|
||
|
||
- **GPU access in headless Chromium.** Real mode needs access to GPU devices and driver caches;
|
||
a filesystem sandbox can silently force software rendering. Use `tools/caps.py` to check.
|
||
Soft mode is available for machines without a GPU. Both modes load isolated HTTP pages.
|
||
- **Top-level `const` and `let` are not `window` properties.** In the page, `window.SM` is
|
||
`undefined` even though `SM` exists. Write `typeof SM !== 'undefined'`, or use the bare name.
|
||
Functions declared with `function` are on `window`.
|
||
- **Playwright calls a string that evaluates to a function.** `page.evaluate("x = function () {}")`
|
||
assigns the function and then calls it. Wrap statements in an arrow function:
|
||
`page.evaluate("() => { ... }")`.
|
||
- **Starting a world takes a moment.** `startNewWorld(seed, w, h)` generates on a timer. Wait
|
||
for `GAME.running === true`. Then remove the welcome card if it is in the way:
|
||
`document.querySelectorAll('.card, .toast').forEach(e => e.remove())`.
|
||
- **Real-time play is not deterministic.** In the browser the simulation runs as fast as frames
|
||
arrive. For repeatable results, pause (`GAME.paused = true`) and call `simTick()` yourself, as
|
||
`tools/shot.py` does.
|
||
- **World sizes** offered by the game: 2048×1024, 3072×1280, 4096×1536 grains.
|
||
- **Rebuild after every source change** before any browser test or screenshot. They all load
|
||
`dist/index.html`.
|
||
|
||
## Measuring
|
||
|
||
- `tools/perf.py`: the phase 0 CPU/frame profiler, served over HTTP with COOP/COEP isolation.
|
||
Rebuilds automatically. Run on an otherwise idle machine, without another browser test:
|
||
|
||
```sh
|
||
python3 tools/perf.py --json tools/perf-baseline.json
|
||
python3 tools/perf.py --json /tmp/perf-now.json --compare tools/perf-baseline.json
|
||
python3 tools/perf.py --viewport 1920x1080 --world 2048x1024 --zooms 12,6,4,3,2 \
|
||
--looks pixel,smooth --scenes settled,factory,heat --seconds 10 --json /tmp/perf.json
|
||
python3 tools/test_perf.py
|
||
```
|
||
|
||
The default matrix is three scenes × two looks × five zooms × lighting full/off. The world
|
||
seed is 424242; fog is off so all visible grains are shaded. Settings otherwise use game
|
||
defaults (including glow and the minimap), with autosave, camera lead and shake disabled.
|
||
`settled` runs 600 ticks; `factory` reuses the furnace/transport layout from
|
||
`experiments/factory_cost.js`, adding screws, a sand filter and full silos for sustained sand
|
||
and ore flow; `heat` places water over magma beside a rock floor. The latter two settle for
|
||
another 60 ticks. Each rebuild resets the power-rebuild scheduler, weather random stream and all statistics to
|
||
fresh-page values; fixture hashes are checked by `test/t_perf.js`. No save/load snapshot is used: it would quantize heat and land flyers.
|
||
|
||
**World** is the whole `drawWorld` CPU call. **Light** is the lighting work inside that call;
|
||
**Prep** sets up the view/masks/joins; **Colour** fills CPU grain buffers (including ghosts and
|
||
flyers); **Upload** submits the pixel buffer or smooth textures; **Blit** submits drawing,
|
||
including pixel light/glow composition or the smooth shader and Canvas2D copy. These are CPU
|
||
wall times, including driver stalls; they are not GPU timestamp measurements. In pixel look,
|
||
the glow upload is part of Blit. The separate `computeLight_ms` times an unconditional light
|
||
call even when the row has lighting off. World includes Light: do not add them together.
|
||
Overlay totals and each individual draw call are in JSON, with ms/frame and invocation counts; conditional
|
||
overlays absent from the fixture have zero calls, not a measured zero execution cost.
|
||
|
||
Five calls warm each render configuration before 30 measured calls. Simulation warms five
|
||
full 120-tick periods on a disposable scene, then rebuilds the same seeded fixture and
|
||
measures 240 consecutive ticks. Every direct `simTick` callee is discovered from its source;
|
||
JSON records per-call and amortized per-tick costs and invocation counts, including periodic
|
||
functions. Inactive paths (e.g. load wake, saplings, flow tracking) have zero calls and null
|
||
per-call cost. Unattributed time includes tick bookkeeping and clock/wrapper overhead.
|
||
Awake/hot chunk counts are recorded at the start/end of simulation and real runs.
|
||
|
||
Real-loop tests independently rebuild each scene for zooms 2 and 6 in both requested looks,
|
||
regardless of `--zooms`. `--seconds` applies to each of these twelve default runs. Raw samples
|
||
retain frame interval (rAF timestamps, including waiting/scheduling), CPU frame work, total
|
||
simulation work per frame and number of ticks. p50/p95/p99 use interpolated sorted samples;
|
||
five warm-up frames are excluded. Zero-tick frames correctly have zero simulation time.
|
||
JSON also records the smooth renderer's adaptive resolution scale: compare it as well as time.
|
||
Comparisons require identical options, print all timing changes, and mark >10% regressions;
|
||
tiny timings and tails can be noisy, and a zero baseline has no meaningful percent change.
|
||
|
||
Performance runs launch Chromium with
|
||
`--use-angle=vulkan --enable-features=Vulkan,VulkanFromANGLE --enable-gpu --ignore-gpu-blocklist --enable-gpu-rasterization --enable-unsafe-webgpu`.
|
||
They record the unmasked GPU/renderer, cores, browser version, viewport/world size,
|
||
`crossOriginIsolated`, `navigator.gpu`, source hash and elapsed time. Real mode refuses software
|
||
renderers; both modes require isolation. Chromium needs access to the host GPU devices; a filesystem
|
||
sandbox that hides `/dev/dri`/NVIDIA devices must be lifted for this measurement. The browser
|
||
tests' SwiftShader numbers (and explicit `--gpu soft` profiler runs) are not a hardware performance baseline.
|
||
|
||
The existing in-game Performance stats toggle shows smoothed CPU world/light/overlay times
|
||
and movement/heat/rest per simulation tick. Detailed renderer clocks are off by default
|
||
(`PERF_MARKS = false`); the raw frame hook (`PERF_FRAME`) is also off outside the profiler.
|
||
`tools/server.py` shares the isolation/no-cache headers with `tools/serve.py`, and supplies a
|
||
free-port background server with cleanup for measurement tools.
|
||
- `tools/experiments/factory_cost.js`: what running factories cost the simulation.
|
||
- `tools/experiments/flow.js`: belt and screw conveyor throughput in grains a second.
|
||
- `tools/experiments/worldgen.js`: material counts and timings of world generation.
|