sysadmin-chronicles/docs/VM_BUILD_SYSTEM.md

VM Build System

Overview

VM provisioning uses a modular driver + profile pattern. One driver script handles the full build pipeline; per-VM profile files declare what makes each machine distinct. Adding a new VM means writing one profile file — no changes to the driver.

Structure

tools/vm/
  build-vm.sh              # Driver — sources a profile and runs the build pipeline
  build-workstation.sh     # Wrapper → build-vm.sh profiles/workstation.sh
  build-web-server.sh      # Wrapper → build-vm.sh profiles/web-server.sh
  build-build-machine.sh   # Wrapper → build-vm.sh profiles/build-machine.sh
  profiles/
    workstation.sh         # sc-workstation / ares    — XFCE desktop (Debian)
    web-server.sh          # sc-web-server  / hermes  — nginx app server (Debian)
    build-machine.sh       # sc-build-machine / vulcan — build toolchain (Arch)
  lib/
    common.sh              # Shared libvirt helpers (pool, domain, seed ISO, wait-for-IP)

Invocation

# By wrapper (backwards-compatible)
./build-workstation.sh [--dry-run] [--force]
./build-web-server.sh  [--dry-run] [--force]
./build-build-machine.sh [--dry-run] [--force]

# By driver directly — profile name (no extension) or explicit path
./build-vm.sh workstation [--dry-run] [--force]
./build-vm.sh profiles/web-server.sh --force

--dry-run skips all libvirt/qemu-img calls and prints what would run. --force destroys and recreates a domain that already exists.

Profile Contract

A profile is a bash file sourced by build-vm.sh. It must set these variables:

Variable	Example	Description
DOMAIN	sc-web-server	libvirt domain name
HOSTNAME	hermes	Guest hostname
RAM_MB	512	Memory in MB
VCPUS	1	vCPU count
DISK_SIZE	8G	qcow2 overlay size
GRAPHICS	vnc	vnc, spice, spice-qxl, or none
BASE_URL	https://...	URL to download base cloud image from
BASE_IMAGE	$SC_BASE_DIR/...	Local path to cache the base image

It must also define generate_user_data() — a function that prints the complete cloud-init #cloud-config YAML to stdout. The driver calls this function and writes the output to the seed ISO. The following variables are available when the function runs (set by the driver after sourcing the profile):

Variable	Value
PUBKEY	Contents of ${SC_SSH_KEY}.pub
GAME_HOST_IP	${SC_GAME_HOST_IP:-10.42.0.1}
POOL_DIR	Resolved libvirt pool path
DISK_PATH	$POOL_DIR/${DOMAIN}.qcow2
SEED_ISO	$SC_SEED_DIR/${DOMAIN}-seed.iso

Profile-specific variables (e.g. HUD_URL, SAGE_URL, PRIVKEY_INDENT) are set in the profile before generate_user_data is defined and are available inside it.

Writing a New Profile

1. Copy profiles/web-server.sh as a starting point.
2. Set the 8 required variables.
3. Write generate_user_data() with the cloud-init YAML for the new machine.
4. Run ./build-vm.sh profiles/my-new-vm.sh --dry-run to validate.
5. Run without --dry-run to build.

No changes to the driver or any other file are needed.

Build Pipeline (driver)

1. Parse --dry-run / --force flags
2. Resolve and source the profile file
3. Validate required variables and generate_user_data function exist
4. Source lib/common.sh (sets SC_* env, exposes helpers)
5. Run ensure_vm_tooling (checks virsh, qemu-img, virt-install, SSH keys, pool/network)
6. If domain exists and --force not set: exit cleanly
7. download_if_missing — fetch base image if not cached
8. Call generate_user_data → write to tmpdir, build NoCloud seed ISO
9. destroy_domain — remove existing domain if present
10. create_backing_disk — qcow2 overlay over the base image
11. build_import_domain — virt-install --import, enable autostart
12. wait_for_agent_ip — poll QEMU guest agent for IP (up to 300 s)
13. Cleanup tmpdir on exit (trap)

Environment Variables

Variable	Default	Description
SC_GAME_HOST_IP	10.42.0.1	Host machine IP on the game network
SC_SSH_KEY	~/.ssh/sc_host_key	SSH key pair used for all host→guest connections
SC_BASE_DIR	See common.sh	Where base cloud images are cached
SC_SEED_DIR	See common.sh	Where cloud-init seed ISOs are written
SC_POOL_NAME	sc-images	libvirt storage pool
SC_NETWORK_NAME	sc-internal	libvirt network
LIBVIRT_DEFAULT_URI	qemu:///system	Override to qemu:///session for user-mode libvirt
SC_WORKSTATION_GRAPHICS	spice	Override workstation graphics backend

Current VMs

Profile	Domain	Hostname	OS	RAM	vCPUs	Disk	Graphics
workstation.sh	sc-workstation	ares	Debian 12	2048 MB	2	20 G	SPICE
web-server.sh	sc-web-server	hermes	Debian 12	512 MB	1	8 G	VNC
build-machine.sh	sc-build-machine	vulcan	Arch Linux	768 MB	2	10 G	VNC

Hostname Resolution

All VMs resolve internal hostnames via static /etc/hosts. There is no DNS server on the game network — this matches how small company networks often work before a proper internal DNS is set up.

Each VM only has entries for the hosts it needs to reach:

• ares (workstation): knows hermes, vulcan, portal.axiomworks.internal, sage.axiomworks.internal
• hermes: knows portal.axiomworks.internal
• vulcan: knows hermes (deploy target), portal.axiomworks.internal

The .axiomworks.internal domain is fictional but realistic — real companies use private suffixes like .internal or .corp for their infrastructure.

Networking Notes

• All VMs attach to the sc-internal libvirt network
• The host machine (10.42.0.1) serves the game portal (:3000) and Sage KB (/sage/)
• Fixed IPs used in /etc/hosts across VMs: hermes=10.42.0.40, vulcan=10.42.0.24
• These must match the DHCP reservations configured in network-sc-internal.xml
• IPv6 disabled on all VMs (sysctl) — not needed, reduces noise

Performance Tuning

All VMs share a common sysctl baseline applied via /etc/sysctl.d/:

Setting	Value	Rationale
vm.swappiness	10	Prefer RAM; swap only under real pressure
vm.vfs_cache_pressure	50	Keep inode cache warm longer
vm.dirty_ratio	15–25	Batch writes; vulcan higher for build workloads
IPv6 disabled	—	Removes unnecessary network overhead

All VMs have a swap file (512 MB – 1 GB depending on role) created at first boot.

DHCP Reservations and MAC Addresses

Fixed IPs are set via DHCP reservations in network-sc-internal.xml and the live libvirt network. The reservations reference MAC addresses, which virt-install generates fresh on every --force rebuild. After any rebuild, the old reservation is stale and the VM will get a random IP from the pool.

After a --force rebuild, update the reservations:

# 1. Get the new MAC
virsh domiflist sc-web-server   # (or sc-workstation, sc-build-machine)

# 2. Remove the old reservation (use the old MAC from network-sc-internal.xml)
sudo virsh net-update sc-internal delete ip-dhcp-host \
  "<host mac='OLD_MAC' name='hermes' ip='10.42.0.40'/>" --live --config

# 3. Add the new one
sudo virsh net-update sc-internal add ip-dhcp-host \
  "<host mac='NEW_MAC' name='hermes' ip='10.42.0.40'/>" --live --config

# 4. Update network-sc-internal.xml to match

The VM will pick up the reserved IP on its next DHCP renewal (or reboot).

Current reservations

VM	Domain	Hostname	MAC	IP
Workstation	sc-workstation	ares	52:54:00:bd:aa:29	10.42.0.36
Web server	sc-web-server	hermes	52:54:00:49:9b:64	10.42.0.40
Build machine	sc-build-machine	vulcan	52:54:00:5e:9f:b9	10.42.0.24