44r0n7andClaude Opus 5.5
1f8a89db6a
Player text says Screw conveyor; code uses screw (BP_SCREW, piece id screw, screwIntake...). No compatibility code for old saves or blueprints. Suite outputs match before and after apart from names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
135 lines
7.1 KiB
Markdown
135 lines
7.1 KiB
Markdown
# Testing
|
||
|
||
There are three layers. Use all three.
|
||
|
||
1. **Headless suites** (`test/t_*.js`) run the simulation in Node with no browser. They are fast
|
||
and deterministic, and they are where most checks live.
|
||
2. **Browser tests** (`test/play.py`, `test/t_smooth.py`) drive the real page in headless
|
||
Chromium with real key, mouse and gamepad input. They check the UI, saving, the smooth look,
|
||
and that the page throws no errors.
|
||
3. **Screenshots** (`tools/shot.py`) show what something looks like. Open the PNG and look at it.
|
||
|
||
Run everything with `python3 test/run_all.py --browser`. It exits non-zero if anything failed and
|
||
prints the failing checks.
|
||
|
||
## Headless suites
|
||
|
||
Each suite is a plain Node script:
|
||
|
||
```js
|
||
// What this suite covers, in a sentence or two.
|
||
const { load, suite, flatWorld, rect, count, ticks } = require('./harness');
|
||
const { run } = load(); // loads src/sim/ (the files in src/manifest.json)
|
||
const { check, done } = suite();
|
||
|
||
flatWorld(run, 240, 160, 100); // 240x160 grains, sky above row 100, stone below
|
||
run('money = 5000; bot.x = 60; bot.y = 40;');
|
||
run('for (let bx = 20; bx < 30; bx++) placeBlock(BP_BELT, 1, bx, 20, true);'); // true: free, no reach check
|
||
rect(run, 84, 60, 95, 79, run('SAND'));
|
||
ticks(run, 300);
|
||
check('sand rides the belt off its end', count(run, run('SAND'), 120, 80, 239, 99) > 100, count(run, run('SAND')));
|
||
done(); // prints ALL PASS or N FAILED, sets the exit code
|
||
```
|
||
|
||
- `run(code)` evaluates JavaScript inside the simulation and returns the result. Every top-level
|
||
name in `src/sim/` is available. Return plain values (numbers, strings, `JSON.stringify(...)`).
|
||
- Harness helpers:
|
||
- `flatWorld(run, w, h, ground)`;
|
||
- `put(run, x, y, m)`;
|
||
- `rect(run, x0, y0, x1, y1, m)`;
|
||
- `count(run, m, x0, y0, x1, y1)`;
|
||
- `maxTemp(run, ...)`;
|
||
- `ticks(run, n)`;
|
||
- `dump(run, x0, y0, x1, y1, withTemp)` for an ASCII picture of an area.
|
||
- `check(name, ok, detail)`: name the behaviour the player would notice ("a pump moves water
|
||
over a wall"). Put the measured numbers in `detail`, so a failure shows how far off it was.
|
||
- Coordinates are in grains. Block coordinates (for `placeBlock`, `placeMachineAt`) are grains
|
||
divided by 4.
|
||
- Errors name the real file and line (`src/sim/pipes.js:42`).
|
||
- A new suite named `test/t_<topic>.js` is picked up by `run_all.py` automatically.
|
||
|
||
**Determinism.** The simulation's randomness comes from seeded generators. The same code and the
|
||
same test give the same output every run, apart from lines with timings in ms. To prove a
|
||
refactor changed nothing, save every suite's output before and after, and diff them:
|
||
|
||
```sh
|
||
for t in test/t_*.js; do node $t > /tmp/before/$(basename $t).txt; done # before the change
|
||
for t in test/t_*.js; do node $t > /tmp/after/$(basename $t).txt; done # after
|
||
diff -r /tmp/before /tmp/after
|
||
```
|
||
|
||
## Browser tests
|
||
|
||
- They need the `.venv` from README.md → Develop; plain `python3` switches to it on its own.
|
||
- `test/play.py` (about 2.5 minutes) builds a world through the title screen and plays it with real
|
||
input. It checks the tools, building, copy/cut/paste, undo, the build menu, filters, pipes,
|
||
the notebook and market, saving and continuing, exporting and importing a world file, the
|
||
gamepad and world sizes. Screenshots go to
|
||
`shots/play/`. Pixel look by default; `python3 test/play.py shots/play smooth` uses the smooth
|
||
look.
|
||
- `test/t_smooth.py` checks that the WebGL2 smooth look compiles and draws, that it falls back
|
||
to pixels cleanly, and that the thermal lens works.
|
||
- Run them one at a time. Two headless browsers at once can starve each other, and
|
||
`t_smooth.py`'s half-resolution check then fails. `run_all.py` runs them one after the other.
|
||
- Rebuild first (`python3 build.py`). They load `dist/index.html`. `run_all.py --browser`
|
||
rebuilds for you.
|
||
|
||
## Screenshots: tools/shot.py
|
||
|
||
```sh
|
||
python3 tools/shot.py shots/silo.png --setup tools/scenes/clear-sky.js \
|
||
--js "$(cat tools/scenes/stone-uses.js)" --ticks 240 --zoom 7 --no-light
|
||
```
|
||
|
||
- It starts a new world, pauses it at once, runs your setup code in the page, runs `--ticks`
|
||
simulation steps, then saves the picture. Because the world is paused otherwise, the same
|
||
options always give the same picture.
|
||
- Setup code can use every game global, plus `SX`, `SY` (the bot's starting position).
|
||
`tools/scenes/clear-sky.js` clears a flat work area and sets `FLOOR`, `BX`, `BY` and `CAM`. Set
|
||
`CAM = [x, y]` to frame the shot.
|
||
- The HUD and the cursor previews are hidden unless you pass `--hud` or `--previews`.
|
||
- `--look smooth` for the smooth look. Test in the pixel look unless you are checking graphics.
|
||
- `--freeze` draws the last frame at a fixed time and frame count, so animations are caught at
|
||
the same moment. Two shots of the same scene then match pixel for pixel, and a before / after
|
||
pair differs only where something really changed. A quick comparison with Pillow:
|
||
|
||
```python
|
||
from PIL import Image, ImageChops
|
||
a, b = (Image.open(p).convert('RGB') for p in ('before.png', 'after.png'))
|
||
print(ImageChops.difference(a, b).getbbox()) # None: identical
|
||
```
|
||
|
||
To compare against an older version, build it somewhere else (for example from a git
|
||
worktree) and pass that page with `--page`.
|
||
- Page errors are printed with their source file and line, and the exit status is 1.
|
||
- Look at the picture. A shot that "ran without errors" can still show the wrong thing.
|
||
|
||
## Pitfalls
|
||
|
||
- **WebGL in headless Chromium.** There is no GPU, so the smooth look needs SwiftShader:
|
||
launch with `--use-gl=angle --use-angle=swiftshader --enable-unsafe-swiftshader`. Also set
|
||
`window.__allowSoftwareGL = true` before the page loads (`add_init_script`); otherwise the game
|
||
sees software rendering and switches to the pixel look. `tools/shot.py` does both.
|
||
- **Top-level `const` and `let` are not `window` properties.** In the page, `window.SM` is
|
||
`undefined` even though `SM` exists. Write `typeof SM !== 'undefined'`, or use the bare name.
|
||
Functions declared with `function` are on `window`.
|
||
- **Playwright calls a string that evaluates to a function.** `page.evaluate("x = function () {}")`
|
||
assigns the function and then calls it. Wrap statements in an arrow function:
|
||
`page.evaluate("() => { ... }")`.
|
||
- **Starting a world takes a moment.** `startNewWorld(seed, w, h)` generates on a timer. Wait
|
||
for `GAME.running === true`. Then remove the welcome card if it is in the way:
|
||
`document.querySelectorAll('.card, .toast').forEach(e => e.remove())`.
|
||
- **Real-time play is not deterministic.** In the browser the simulation runs as fast as frames
|
||
arrive. For repeatable results, pause (`GAME.paused = true`) and call `simTick()` yourself, as
|
||
`tools/shot.py` does.
|
||
- **World sizes** offered by the game: 2048×1024, 3072×1280, 4096×1536 grains.
|
||
- **Rebuild after every source change** before any browser test or screenshot. They all load
|
||
`dist/index.html`.
|
||
|
||
## Measuring
|
||
|
||
- `tools/perf.py`: renderer and tick times in the browser.
|
||
- `tools/experiments/factory_cost.js`: what running factories cost the simulation.
|
||
- `tools/experiments/flow.js`: belt and screw conveyor throughput in grains a second.
|
||
- `tools/experiments/worldgen.js`: material counts and timings of world generation.
|