deepgrain-foundry/docs/TESTING.md

Testing

There are three layers. Use all three.

1. Headless suites (test/t_*.js) run the simulation in Node with no browser. They are fast and deterministic, and they are where most checks live.
2. Browser tests (test/play.py, test/t_smooth.py) drive the real page in headless Chromium with real key, mouse and gamepad input. They check the UI, saving, the smooth look, and that the page throws no errors.
3. Screenshots (tools/shot.py) show what something looks like. Open the PNG and look at it.

Run everything with python3 test/run_all.py --browser. It exits non-zero if anything failed and prints the failing checks.

Headless suites

Each suite is a plain Node script:

// What this suite covers, in a sentence or two.
const { load, suite, flatWorld, rect, count, ticks } = require('./harness');
const { run } = load();                 // loads src/sim/ (the files in src/manifest.json)
const { check, done } = suite();

flatWorld(run, 240, 160, 100);          // 240x160 grains, sky above row 100, stone below
run('money = 5000; bot.x = 60; bot.y = 40;');
run('for (let bx = 20; bx < 30; bx++) placeBlock(BP_BELT, 1, bx, 20, true);'); // true: free, no reach check
rect(run, 84, 60, 95, 79, run('SAND'));
ticks(run, 300);
check('sand rides the belt off its end', count(run, run('SAND'), 120, 80, 239, 99) > 100, count(run, run('SAND')));
done();                                 // prints ALL PASS or N FAILED, sets the exit code

• run(code) evaluates JavaScript inside the simulation and returns the result. Every top-level name in src/sim/ is available. Return plain values (numbers, strings, JSON.stringify(...)).
• Harness helpers:
  • flatWorld(run, w, h, ground);
  • put(run, x, y, m);
  • rect(run, x0, y0, x1, y1, m);
  • count(run, m, x0, y0, x1, y1);
  • maxTemp(run, ...);
  • ticks(run, n);
  • dump(run, x0, y0, x1, y1, withTemp) for an ASCII picture of an area.
• check(name, ok, detail): name the behaviour the player would notice ("a pump moves water over a wall"). Put the measured numbers in detail, so a failure shows how far off it was.
• Coordinates are in grains. Block coordinates (for placeBlock, placeMachineAt) are grains divided by 4.
• Errors name the real file and line (src/sim/pipes.js:42).
• A new suite named test/t_<topic>.js is picked up by run_all.py automatically.

Determinism. The simulation's randomness comes from seeded generators. The same code and the same test give the same output every run, apart from lines with timings in ms. To prove a refactor changed nothing, save every suite's output before and after, and diff them:

for t in test/t_*.js; do node $t > /tmp/before/$(basename $t).txt; done   # before the change
for t in test/t_*.js; do node $t > /tmp/after/$(basename $t).txt; done    # after
diff -r /tmp/before /tmp/after

Browser tests

• They need the .venv from README.md → Develop; plain python3 switches to it on its own.
• test/play.py (about 2.5 minutes) builds a world through the title screen and plays it with real input. It checks the tools, building, copy/cut/paste, undo, the build menu, filters, pipes, the notebook and market, saving and continuing, exporting and importing a world file, the gamepad and world sizes. Screenshots go to shots/play/. Pixel look by default; python3 test/play.py shots/play smooth uses the smooth look.
• test/t_smooth.py checks that the WebGL2 smooth look compiles and draws, that it falls back to pixels cleanly, and that the thermal lens works.
• Run them one at a time. Two headless browsers at once can starve each other, and t_smooth.py's half-resolution check then fails. run_all.py runs them one after the other.
• Rebuild first (python3 build.py). They load dist/index.html. run_all.py --browser rebuilds for you.

Screenshots: tools/shot.py

python3 tools/shot.py shots/silo.png --setup tools/scenes/clear-sky.js \
    --js "$(cat tools/scenes/stone-uses.js)" --ticks 240 --zoom 7 --no-light

• It starts a new world, pauses it at once, runs your setup code in the page, runs --ticks simulation steps, then saves the picture. Because the world is paused otherwise, the same options always give the same picture.

• Setup code can use every game global, plus SX, SY (the bot's starting position). tools/scenes/clear-sky.js clears a flat work area and sets FLOOR, BX, BY and CAM. Set CAM = [x, y] to frame the shot.

• The HUD and the cursor previews are hidden unless you pass --hud or --previews.

• --look smooth for the smooth look. Test in the pixel look unless you are checking graphics.

• --freeze draws the last frame at a fixed time and frame count, so animations are caught at the same moment. Two shots of the same scene then match pixel for pixel, and a before / after pair differs only where something really changed. A quick comparison with Pillow:

  from PIL import Image, ImageChops
  a, b = (Image.open(p).convert('RGB') for p in ('before.png', 'after.png'))
  print(ImageChops.difference(a, b).getbbox())   # None: identical
  

  To compare against an older version, build it somewhere else (for example from a git worktree) and pass that page with --page.

• Page errors are printed with their source file and line, and the exit status is 1.

• Look at the picture. A shot that "ran without errors" can still show the wrong thing.

Pitfalls

• WebGL in headless Chromium. The browser tests use SwiftShader for portable graphics checks: launch with --use-gl=angle --use-angle=swiftshader --enable-unsafe-swiftshader. Also set window.__allowSoftwareGL = true before the page loads (add_init_script); otherwise the game sees software rendering and switches to the pixel look. tools/shot.py does both.
• Top-level const and let are not window properties. In the page, window.SM is undefined even though SM exists. Write typeof SM !== 'undefined', or use the bare name. Functions declared with function are on window.
• Playwright calls a string that evaluates to a function. page.evaluate("x = function () {}") assigns the function and then calls it. Wrap statements in an arrow function: page.evaluate("() => { ... }").
• Starting a world takes a moment. startNewWorld(seed, w, h) generates on a timer. Wait for GAME.running === true. Then remove the welcome card if it is in the way: document.querySelectorAll('.card, .toast').forEach(e => e.remove()).
• Real-time play is not deterministic. In the browser the simulation runs as fast as frames arrive. For repeatable results, pause (GAME.paused = true) and call simTick() yourself, as tools/shot.py does.
• World sizes offered by the game: 2048×1024, 3072×1280, 4096×1536 grains.
• Rebuild after every source change before any browser test or screenshot. They all load dist/index.html.

Measuring

• tools/perf.py: the phase 0 CPU/frame profiler, served over HTTP with COOP/COEP isolation. Rebuilds automatically. Run on an otherwise idle machine, without another browser test:

  python3 tools/perf.py --json tools/perf-baseline.json
  python3 tools/perf.py --json /tmp/perf-now.json --compare tools/perf-baseline.json
  python3 tools/perf.py --viewport 1920x1080 --world 2048x1024 --zooms 12,6,4,3,2 \
      --looks pixel,smooth --scenes settled,factory,heat --seconds 10 --json /tmp/perf.json
  python3 tools/test_perf.py
  

  The default matrix is three scenes × two looks × five zooms × lighting full/off. The world seed is 424242; fog is off so all visible grains are shaded. Settings otherwise use game defaults (including glow and the minimap), with autosave, camera lead and shake disabled. settled runs 600 ticks; factory reuses the furnace/transport layout from experiments/factory_cost.js, adding screws, a sand filter and full silos for sustained sand and ore flow; heat places water over magma beside a rock floor. The latter two settle for another 60 ticks. Each rebuild resets the power-rebuild scheduler, weather random stream and all statistics to fresh-page values; fixture hashes are checked by test/t_perf.js. No save/load snapshot is used: it would quantize heat and land flyers.

  World is the whole drawWorld CPU call. Light is the lighting work inside that call; Prep sets up the view/masks/joins; Colour fills CPU grain buffers (including ghosts and flyers); Upload submits the pixel buffer or smooth textures; Blit submits drawing, including pixel light/glow composition or the smooth shader and Canvas2D copy. These are CPU wall times, including driver stalls; they are not GPU timestamp measurements. In pixel look, the glow upload is part of Blit. The separate computeLight_ms times an unconditional light call even when the row has lighting off. World includes Light: do not add them together. Overlay totals and each individual draw call are in JSON, with ms/frame and invocation counts; conditional overlays absent from the fixture have zero calls, not a measured zero execution cost.

  Five calls warm each render configuration before 30 measured calls. Simulation warms five full 120-tick periods on a disposable scene, then rebuilds the same seeded fixture and measures 240 consecutive ticks. Every direct simTick callee is discovered from its source; JSON records per-call and amortized per-tick costs and invocation counts, including periodic functions. Inactive paths (e.g. load wake, saplings, flow tracking) have zero calls and null per-call cost. Unattributed time includes tick bookkeeping and clock/wrapper overhead. Awake/hot chunk counts are recorded at the start/end of simulation and real runs.

  Real-loop tests independently rebuild each scene for zooms 2 and 6 in both requested looks, regardless of --zooms. --seconds applies to each of these twelve default runs. Raw samples retain frame interval (rAF timestamps, including waiting/scheduling), CPU frame work, total simulation work per frame and number of ticks. p50/p95/p99 use interpolated sorted samples; five warm-up frames are excluded. Zero-tick frames correctly have zero simulation time. JSON also records the smooth renderer's adaptive resolution scale: compare it as well as time. Comparisons require identical options, print all timing changes, and mark >10% regressions; tiny timings and tails can be noisy, and a zero baseline has no meaningful percent change.

  Performance runs launch Chromium with --use-angle=vulkan --enable-features=Vulkan --enable-gpu --ignore-gpu-blocklist --enable-gpu-rasterization. They record the unmasked GPU/renderer, cores, browser version, viewport/world size, crossOriginIsolated, navigator.gpu, source hash and elapsed time. They refuse software renderers or missing isolation. Chromium needs access to the host GPU devices; a filesystem sandbox that hides /dev/dri/NVIDIA devices must be lifted for this measurement. The browser tests' SwiftShader numbers are not a performance baseline.

  The existing in-game Performance stats toggle shows smoothed CPU world/light/overlay times and movement/heat/rest per simulation tick. Detailed renderer clocks are off by default (PERF_MARKS = false); the raw frame hook (PERF_FRAME) is also off outside the profiler. tools/server.py shares the isolation/no-cache headers with tools/serve.py, and supplies a free-port background server with cleanup for measurement tools.

• tools/experiments/factory_cost.js: what running factories cost the simulation.

• tools/experiments/flow.js: belt and screw conveyor throughput in grains a second.

• tools/experiments/worldgen.js: material counts and timings of world generation.