deepgrain-foundry/docs/TESTING.md

Testing

There are three layers. Use all three.

1. Headless suites (test/t_*.js) run the simulation in Node with no browser. They are fast and deterministic, and they are where most checks live.
2. Browser tests (test/play.py, test/t_smooth.py) drive the real page in headless Chromium with real key, mouse and gamepad input. They check the UI, saving, the smooth look, and that the page throws no errors.
3. Screenshots (tools/shot.py) show what something looks like. Open the PNG and look at it.

Run everything with python3 test/run_all.py --browser. It exits non-zero if anything failed and prints the failing checks.

Headless suites

Each suite is a plain Node script:

// What this suite covers, in a sentence or two.
const { load, suite, flatWorld, rect, count, ticks } = require('./harness');
const { run } = load();                 // loads src/sim/ (the files in src/manifest.json)
const { check, done } = suite();

flatWorld(run, 240, 160, 100);          // 240x160 grains, sky above row 100, stone below
run('money = 5000; bot.x = 60; bot.y = 40;');
run('for (let bx = 20; bx < 30; bx++) placeBlock(BP_BELT, 1, bx, 20, true);'); // true: free, no reach check
rect(run, 84, 60, 95, 79, run('SAND'));
ticks(run, 300);
check('sand rides the belt off its end', count(run, run('SAND'), 120, 80, 239, 99) > 100, count(run, run('SAND')));
done();                                 // prints ALL PASS or N FAILED, sets the exit code

• run(code) evaluates JavaScript inside the simulation and returns the result. Every top-level name in src/sim/ is available. Return plain values (numbers, strings, JSON.stringify(...)).
• Harness helpers:
  • flatWorld(run, w, h, ground);
  • put(run, x, y, m);
  • rect(run, x0, y0, x1, y1, m);
  • count(run, m, x0, y0, x1, y1);
  • maxTemp(run, ...);
  • ticks(run, n);
  • dump(run, x0, y0, x1, y1, withTemp) for an ASCII picture of an area.
• check(name, ok, detail): name the behaviour the player would notice ("a pump moves water over a wall"). Put the measured numbers in detail, so a failure shows how far off it was.
• Coordinates are in grains. Block coordinates (for placeBlock, placeMachineAt) are grains divided by 4.
• Errors name the real file and line (src/sim/pipes.js:42).
• A new suite named test/t_<topic>.js is picked up by run_all.py automatically.

Determinism. The simulation's randomness comes from seeded generators. The same code and the same test give the same output every run, apart from lines with timings in ms. To prove a refactor changed nothing, save every suite's output before and after, and diff them:

for t in test/t_*.js; do node $t > /tmp/before/$(basename $t).txt; done   # before the change
for t in test/t_*.js; do node $t > /tmp/after/$(basename $t).txt; done    # after
diff -r /tmp/before /tmp/after

Browser tests

• They need the .venv from README.md → Develop; plain python3 switches to it on its own.
• test/play.py (about 2.5 minutes) builds a world through the title screen and plays it with real input. It checks the tools, building, copy/cut/paste, undo, the build menu, filters, pipes, the notebook and market, saving and continuing, exporting and importing a world file, the gamepad and world sizes. Screenshots go to shots/play/. Pixel look by default; python3 test/play.py shots/play smooth uses the smooth look.
• test/t_smooth.py checks that the WebGPU smooth look compiles and draws, compares legacy at half resolution, and falls back to pixels cleanly, and that the thermal lens works.
• Run them one at a time. Two headless browsers at once can starve each other, and t_smooth.py's half-resolution check then fails. run_all.py runs them one after the other.
• Rebuild first (python3 build.py); run_all.py --browser rebuilds for you. All browser tools use tools/browser.py and tools/server.py: a free-port background HTTP server serves the built page with COOP/COEP headers and shuts down after the browser. No file:// loads.
• Choose Chromium graphics with --gpu real|soft or DEEPGRAIN_GPU=real|soft (the command line wins). This works for the suite runner, individual browser tests, screenshots and profiler. Real mode uses ANGLE Vulkan on the host GPU; soft uses SwiftShader and sets window.__allowSoftwareGL before load. The explicit software-fallback test still uses soft mode with that override off. Firefox uses its own default graphics backend. The default is real, including the profiler and caps.py. No colour tolerances were changed. The play test holds the slow-save fixture open during its screenshot, waits for tool/selection UI updates, disables camera lead after testing it, and gives the gamepad drill a known reachable target; these checks do not depend on GPU speed or earlier digging.
• python3 tools/caps.py prints isolation, actual WebGPU adapter info and the GL renderer for Chromium and installed Firefox. Real mode requires a hardware Chromium adapter.
• test/t_caps.py checks missing, null and rejected WebGPU adapters, both title notices, warning mode and the future REQUIRE_GPU_AND_THREADS startup gate. Its forced notice screenshot is shots/t88-p1-notice.png.
• Install Firefox with .venv/bin/python -m playwright install firefox. python3 test/firefox_smoke.py checks isolation and the WebGPU notice, starts a pixel world, runs three seconds and checks page errors. run_all.py --browser includes it and prints a skip note if Firefox is absent; an installed browser that fails to launch is a failure.
• Linux WebGPU currently needs Firefox Nightly; the notice points to about:config → dom.webgpu.enabled. Sources: Mozilla's experimental features and GPUWeb implementation status.

Screenshots: tools/shot.py

python3 tools/shot.py shots/silo.png --setup tools/scenes/clear-sky.js \
    --js "$(cat tools/scenes/stone-uses.js)" --ticks 240 --zoom 7 --no-light

• It starts a new world, pauses it at once, runs your setup code in the page, runs --ticks simulation steps, then saves the picture. Because the world is paused otherwise, the same options always give the same picture.

• Setup code can use every game global, plus SX, SY (the bot's starting position). tools/scenes/clear-sky.js clears a flat work area and sets FLOOR, BX, BY and CAM. Set CAM = [x, y] to frame the shot.

• The HUD and the cursor previews are hidden unless you pass --hud or --previews.

• --renderer webgpu|legacy selects the renderer for either look (default WebGPU). WebGPU screenshots fail clearly when no device is available; Legacy smooth uses WebGL2; WebGPU smooth uses WGSL.

• test/t_gpu.py compares the rendered composite against Legacy for ordinary cells, built pieces, ghosts, fog and all thermal palettes, and checks world replacement, resize, timing and device loss. World-pixel tests use worldPixelContext() to read both stacked canvases. The helper presents the retained GPU result again before copying, because canvas textures expire after presentation (WebGPU specification, Mozilla implementation notes). Tests that change static noiseB in place set GPU.world = null before drawing to upload their new fixture.

• --look smooth for the smooth look. Test in the pixel look unless you are checking graphics.

• --freeze draws the last frame at a fixed time and frame count, so animations are caught at the same moment. Two shots of the same scene then match pixel for pixel, and a before / after pair differs only where something really changed. A quick comparison with Pillow:

  from PIL import Image, ImageChops
  a, b = (Image.open(p).convert('RGB') for p in ('before.png', 'after.png'))
  print(ImageChops.difference(a, b).getbbox())   # None: identical
  

  To compare against an older version, build it somewhere else (for example from a git worktree) and pass that page with --page.

• Page errors are printed with their source file and line, and the exit status is 1.

• Look at the picture. A shot that "ran without errors" can still show the wrong thing.

Pitfalls

• GPU access in headless Chromium. Real mode needs access to GPU devices and driver caches; a filesystem sandbox can silently force software rendering. Use tools/caps.py to check. Soft mode is available for machines without a GPU. Both modes load isolated HTTP pages.
• Top-level const and let are not window properties. In the page, window.SM is undefined even though SM exists. Write typeof SM !== 'undefined', or use the bare name. Functions declared with function are on window.
• Playwright calls a string that evaluates to a function. page.evaluate("x = function () {}") assigns the function and then calls it. Wrap statements in an arrow function: page.evaluate("() => { ... }").
• Starting a world takes a moment. startNewWorld(seed, w, h) generates on a timer. Wait for GAME.running === true. Then remove the welcome card if it is in the way: document.querySelectorAll('.card, .toast').forEach(e => e.remove()).
• Real-time play is not deterministic. In the browser the simulation runs as fast as frames arrive. For repeatable results, pause (GAME.paused = true) and call simTick() yourself, as tools/shot.py does.
• World sizes offered by the game: 2048×1024, 3072×1280, 4096×1536 grains.
• Rebuild after every source change before any browser test or screenshot. They all load dist/index.html.

Measuring

• tools/perf.py: the phase 0 CPU/frame profiler, served over HTTP with COOP/COEP isolation. Rebuilds automatically. Run on an otherwise idle machine, without another browser test:

  python3 tools/perf.py --json tools/perf-baseline.json
  python3 tools/perf.py --json /tmp/perf-now.json --compare tools/perf-baseline.json
  python3 tools/perf.py --viewport 1920x1080 --world 2048x1024 --zooms 12,6,4,3,2 \
      --looks pixel,smooth --scenes settled,factory,heat --seconds 10 --json /tmp/perf.json
  python3 tools/test_perf.py
  

  The default matrix is three scenes × two looks × five zooms × lighting full/off. The world seed is 424242; fog is off so all visible grains are shaded. Settings otherwise use game defaults (including glow and the minimap), with autosave, camera lead and shake disabled. settled runs 600 ticks; factory reuses the furnace/transport layout from experiments/factory_cost.js, adding screws, a sand filter and full silos for sustained sand and ore flow; heat places water over magma beside a rock floor. The latter two settle for another 60 ticks. Each rebuild resets the power-rebuild scheduler, weather random stream and all statistics to fresh-page values; fixture hashes are checked by test/t_perf.js. No save/load snapshot is used: it would quantize heat and land flyers.

  World is the whole drawWorld CPU call. Light is the lighting work inside that call; Prep sets up the view/masks/joins; Colour fills CPU grain buffers (including ghosts and flyers); Upload submits the pixel buffer or smooth textures; Blit submits drawing, including pixel light/glow composition or the smooth shader and Canvas2D copy. These are CPU wall times, including driver stalls; they are not GPU timestamp measurements. In pixel look, the glow upload is part of Blit. The separate computeLight_ms times an unconditional light call even when the row has lighting off. World includes Light: do not add them together. Overlay totals and each individual draw call are in JSON, with ms/frame and invocation counts; conditional overlays absent from the fixture have zero calls, not a measured zero execution cost.

  Five calls warm each render configuration before 30 measured calls. Simulation warms five full 120-tick periods on a disposable scene, then rebuilds the same seeded fixture and measures 240 consecutive ticks. Every direct simTick callee is discovered from its source; JSON records per-call and amortized per-tick costs and invocation counts, including periodic functions. Inactive paths (e.g. load wake, saplings, flow tracking) have zero calls and null per-call cost. Unattributed time includes tick bookkeeping and clock/wrapper overhead. Awake/hot chunk counts are recorded at the start/end of simulation and real runs.

  Real-loop tests independently rebuild each scene for zooms 2 and 6 in both requested looks, regardless of --zooms. --seconds applies to each of these twelve default runs. Raw samples retain frame interval (rAF timestamps, including waiting/scheduling), CPU frame work, total simulation work per frame and number of ticks. p50/p95/p99 use interpolated sorted samples; five warm-up frames are excluded. Zero-tick frames correctly have zero simulation time. JSON also records the smooth renderer's adaptive resolution scale: compare it as well as time. Comparisons accept subsets of the baseline scenes, looks and zooms, with identical world/viewport dimensions and run duration, print all timing changes, and mark >10% regressions; tiny timings and tails can be noisy, and a zero baseline has no meaningful percent change.

  Performance runs launch Chromium with --use-angle=vulkan --enable-features=Vulkan,VulkanFromANGLE --enable-gpu --ignore-gpu-blocklist --enable-gpu-rasterization --enable-unsafe-webgpu. They record the unmasked GPU/renderer, cores, browser version, viewport/world size, crossOriginIsolated, navigator.gpu, source hash and elapsed time. Real mode refuses software renderers; both modes require isolation. Chromium needs access to the host GPU devices; a filesystem sandbox that hides /dev/dri/NVIDIA devices must be lifted for this measurement. The browser tests' SwiftShader numbers (and explicit --gpu soft profiler runs) are not a hardware performance baseline.

  The existing in-game Performance stats toggle shows smoothed CPU world/light/overlay times and movement/heat/rest per simulation tick. Detailed renderer clocks are off by default (PERF_MARKS = false); the raw frame hook (PERF_FRAME) is also off outside the profiler. tools/server.py shares the isolation/no-cache headers with tools/serve.py, and supplies a free-port background server with cleanup for measurement tools.

• tools/experiments/factory_cost.js: what running factories cost the simulation.

• tools/experiments/flow.js: belt and screw conveyor throughput in grains a second.

• tools/experiments/worldgen.js: material counts and timings of world generation.

WebGPU Phase 2a: Colour measures the CPU override fill (including loose objects and ghosts). frame_ms is the complete paused-frame CPU call (including overlays and UI). Upload measures strided visible-array writes and the override/light uploads, and Blit measures GPU command submission. override_cells records the selected CPU cells; GPU timing is a completed timestamp sample for that configuration across colour, bloom and drawing when timestamp-query is available. The timestamp excludes uploads and the CPU lighting pass. Lighting remains on the CPU. The GPU canvas draws the world; the transparent game canvas holds the existing overlays.

The comparison fixture is tools/scenes/t88-renderer.js. Capture with --freeze --zoom 12 (and 6 or 2), --viewport 1920x1080, and each --renderer. It includes terrain, water, fire, magma, fog, a tree, flat/slope belts, screws, pipe, mold, bar and blueprint ghost.

WebGPU Phase 2c: both looks use WebGPU. test/t_gpu.py also reads all four smooth input textures back and compares them against the legacy CPU buffers at zoom 12/6/2 with fog on/off: classes, sky flags, field bits and geometry tags must match exactly; RGB can differ by one. The smooth shader retains the legacy nine-tap glow filter and samples field AO at mip level 2.2; three render passes generate field levels 1-3, including non-power-of-two view sizes. Only smooth allocates/uploads the back/tree/field overrides and a separate presence mask. tools/perf.py refuses a WebGPU fallback in either look and records WebGPU scale as 1. The screenshot tool invalidates the shared parallax horizon after scene setup changes surfY, so renderer startup timing cannot change the hills in a comparison. Mip pass references: Chrome's porting guide and WebGPU Fundamentals.