chore: bootstrap lean sysadmin-chronicles repo

Import the runnable game code, content, docs, scripts, and repo guidance while leaving local agent state, dependency installs, build output, and backup copies out of the published tree.
This commit is contained in:
2026-05-02 11:49:07 -04:00
commit 0265afa054
252 changed files with 37574 additions and 0 deletions
+29
View File
@@ -0,0 +1,29 @@
{
"categories": [
{
"id": "access",
"label": "Access & Authentication",
"articles": ["ssh-keys", "ssh-access-controls"]
},
{
"id": "web",
"label": "Web Services",
"articles": ["nginx-config"]
},
{
"id": "storage",
"label": "Storage & Logs",
"articles": ["disk-logs"]
},
{
"id": "sysadmin",
"label": "System Administration",
"articles": ["file-permissions", "cron-jobs", "time-sync"]
},
{
"id": "packages",
"label": "Package Management",
"articles": ["package-management"]
}
]
}
+40
View File
@@ -0,0 +1,40 @@
{
"id": "cron-jobs",
"title": "Cron Jobs & Scheduled Tasks",
"category": "sysadmin",
"tags": ["cron", "crontab", "schedule", "backup", "automation"],
"updated": "2025-12-01",
"summary": "Cron syntax, user vs system crons, and common failure modes.",
"sections": [
{
"heading": "Cron Syntax",
"body": "<p>A crontab entry has five time fields followed by the command:</p>",
"code": "# ┌─── minute (059)\n# │ ┌─── hour (023)\n# │ │ ┌─── day of month (131)\n# │ │ │ ┌─── month (1–12)\n# │ │ │ │ ┌─── day of week (07, 0 and 7 are Sunday)\n# │ │ │ │ │\n * * * * * /path/to/command\n\n# Examples:\n0 2 * * * /usr/local/bin/backup.sh # 2am every day\n*/15 * * * * /usr/local/bin/check.sh # every 15 minutes\n0 0 1 * * /usr/local/bin/monthly.sh # midnight on the 1st"
},
{
"heading": "User Crontabs",
"body": "<p>Each user can have their own crontab. Commands run as that user.</p>",
"code": "crontab -e # edit your crontab\ncrontab -l # list your crontab\ncrontab -l -u alice # list alice's crontab (root only)\ncrontab -r # delete your crontab (dangerous—no confirmation)"
},
{
"heading": "System Cron Directories",
"body": "<p>Scripts dropped into these directories run at the corresponding interval without needing a crontab entry:</p>",
"code": "/etc/cron.daily/\n/etc/cron.weekly/\n/etc/cron.monthly/\n/etc/cron.hourly/\n\n# Scripts here must be executable and owned by root.\n# They must NOT have a file extension—run-parts ignores files with dots in the name."
},
{
"heading": "Ownership and the PATH Problem",
"body": "<p>Two common failure modes:</p><p><strong>Wrong owner:</strong> A cron script in <code>/etc/cron.daily/</code> must be owned by root. If it is owned by another user, run-parts may skip it.</p><p><strong>Missing PATH:</strong> Cron does not source <code>.bashrc</code> or <code>.profile</code>. Commands that work interactively may fail in cron because the PATH only contains <code>/usr/bin:/bin</code>. Always use full paths in cron scripts, or set PATH explicitly at the top of the script.</p>",
"code": "#!/bin/bash\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n..."
},
{
"heading": "Checking If a Cron Ran",
"body": "",
"code": "# Check syslog or the cron-specific log\ngrep CRON /var/log/syslog | tail -20\ncat /var/log/cron.log # if separate cron log is configured\n\n# Check journald\njournalctl -u cron --since \"1 hour ago\""
},
{
"heading": "Capturing Cron Output",
"body": "<p>By default, cron mails output to the user. On servers with no mail configured, errors disappear silently. Redirect to a log file instead:</p>",
"code": "0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1"
}
]
}
+43
View File
@@ -0,0 +1,43 @@
{
"id": "disk-logs",
"title": "Disk Space & Log Rotation",
"category": "storage",
"tags": ["disk", "df", "du", "logs", "logrotate", "cleanup"],
"updated": "2025-08-22",
"summary": "Finding what is filling the disk and keeping logs from growing unbounded.",
"sections": [
{
"heading": "Checking Disk Usage",
"body": "<p><code>df</code> shows you how full each filesystem is. <code>du</code> tells you where the space went.</p>",
"code": "df -h # human-readable filesystem summary\ndf -h /var/log # check a specific mount\n\ndu -sh /var/log/* # top-level breakdown of /var/log\ndu -sh /var/* | sort -rh # sort by size, largest first\ndu -sh /var/log/*.log # sizes of individual log files"
},
{
"heading": "Finding Large Files",
"body": "<p>When du does not point at an obvious culprit:</p>",
"code": "# Files over 100MB anywhere on the system\nfind / -xdev -size +100M -type f 2>/dev/null\n\n# Files in /var that have grown recently\nfind /var -xdev -mtime -1 -size +10M -type f 2>/dev/null"
},
{
"heading": "Emergency Cleanup",
"body": "<p>If disk is at 100% and a service is failing because of it:</p>",
"code": "# Truncate a log file without deleting it (safe for running processes)\ntruncate -s 0 /var/log/nginx/access.log\n\n# Remove old compressed logs (the .gz files are already rotated)\nrm /var/log/nginx/*.gz\n\n# Clear journald logs older than 2 days\njournalctl --vacuum-time=2d"
},
{
"heading": "logrotate Basics",
"body": "<p>logrotate is the standard tool for rotating and compressing logs on a schedule. It is usually run daily from cron. Config files live in <code>/etc/logrotate.d/</code>—one file per service.</p>"
},
{
"heading": "Writing a logrotate Config",
"body": "<p>Example for an nginx access log:</p>",
"code": "/var/log/nginx/access.log {\n daily\n rotate 14\n compress\n delaycompress\n missingok\n notifempty\n sharedscripts\n postrotate\n /bin/kill -USR1 $(cat /run/nginx.pid 2>/dev/null) 2>/dev/null || true\n endscript\n}"
},
{
"heading": "Testing logrotate",
"body": "<p>Run logrotate manually in debug mode to verify a config without actually rotating anything:</p>",
"code": "logrotate -d /etc/logrotate.d/nginx\n\n# To force a rotation right now (useful for testing):\nlogrotate -f /etc/logrotate.d/nginx"
},
{
"heading": "Key logrotate Directives",
"body": "<table><tr><th>Directive</th><th>Meaning</th></tr><tr><td><code>daily/weekly/monthly</code></td><td>Rotation frequency</td></tr><tr><td><code>rotate N</code></td><td>Keep N old copies</td></tr><tr><td><code>compress</code></td><td>gzip old files</td></tr><tr><td><code>delaycompress</code></td><td>Skip compressing the most recent rotation (useful when the app still has it open)</td></tr><tr><td><code>missingok</code></td><td>Do not error if the log file does not exist</td></tr><tr><td><code>notifempty</code></td><td>Skip rotation if the file is empty</td></tr><tr><td><code>size 100M</code></td><td>Rotate when file exceeds this size instead of on schedule</td></tr></table>"
}
]
}
@@ -0,0 +1,37 @@
{
"id": "file-permissions",
"title": "File Ownership & Permissions",
"category": "sysadmin",
"tags": ["chown", "chmod", "permissions", "ownership", "ls"],
"updated": "2025-10-07",
"summary": "Understanding and fixing file ownership and permission bits.",
"sections": [
{
"heading": "Reading the Permission String",
"body": "<p>Run <code>ls -l</code> to see permissions. The first column looks like <code>-rwxr-xr--</code>.</p><ul><li>First character: <code>-</code> file, <code>d</code> directory, <code>l</code> symlink</li><li>Next three: owner read/write/execute</li><li>Next three: group read/write/execute</li><li>Last three: others read/write/execute</li></ul><p><code>r</code>=4, <code>w</code>=2, <code>x</code>=1. Add them up for octal notation: <code>rwx</code>=7, <code>rw-</code>=6, <code>r--</code>=4.</p>"
},
{
"heading": "chown — Changing Ownership",
"body": "<p>Change the owner and/or group of a file or directory.</p>",
"code": "chown user file # change owner only\nchown user:group file # change owner and group\nchown :group file # change group only\n\n# Recursive — change everything under a directory\nchown -R user:group /path/to/dir"
},
{
"heading": "chmod — Changing Permissions",
"body": "",
"code": "chmod 644 file.txt # rw-r--r-- (typical for files)\nchmod 755 /usr/local/bin/app # rwxr-xr-x (typical for executables)\nchmod 700 ~/.ssh # rwx------ (private directory)\nchmod 600 ~/.ssh/authorized_keys # rw------- (private file)\n\n# Recursive\nchmod -R 755 /var/www/html\n\n# Symbolic form (add execute for owner only)\nchmod u+x script.sh"
},
{
"heading": "Common Patterns",
"body": "<table><tr><th>Mode</th><th>Numeric</th><th>Typical use</th></tr><tr><td><code>rw-r--r--</code></td><td>644</td><td>Regular files, config files</td></tr><tr><td><code>rwxr-xr-x</code></td><td>755</td><td>Directories, executables</td></tr><tr><td><code>rwx------</code></td><td>700</td><td>Private directories (e.g. ~/.ssh)</td></tr><tr><td><code>rw-------</code></td><td>600</td><td>Private files (e.g. private keys, authorized_keys)</td></tr><tr><td><code>rwxrwxr-x</code></td><td>775</td><td>Shared directories where the group needs write access</td></tr></table>"
},
{
"heading": "Checking Who Owns What",
"body": "",
"code": "ls -la /var/www/html # list with ownership\nstat file.txt # detailed file metadata\nfind /path -user root # find files owned by root\nfind /path -not -user deploy # find files NOT owned by deploy"
},
{
"heading": "A Note on Recursive chown",
"body": "<p>When you run <code>chown -R</code>, it changes <em>everything</em> under the path—including files and subdirectories that may have intentionally different ownership. Know what you are targeting before running it on a live system. Check with <code>ls -laR</code> or <code>find</code> first.</p>"
}
]
}
+38
View File
@@ -0,0 +1,38 @@
{
"id": "nginx-config",
"title": "nginx Configuration",
"category": "web",
"tags": ["nginx", "config", "syntax", "reload", "vhost"],
"updated": "2025-09-18",
"summary": "nginx config structure, common syntax errors, and safe reload procedure.",
"sections": [
{
"heading": "Config File Layout",
"body": "<p>nginx uses a block-based config syntax. The main file is <code>/etc/nginx/nginx.conf</code>. Site configs live in <code>/etc/nginx/sites-available/</code> and are symlinked into <code>/etc/nginx/sites-enabled/</code> to activate them.</p><p>Every block opens with <code>{</code> and closes with <code>}</code>. Every directive ends with <code>;</code>. Missing either one will fail the syntax check.</p>"
},
{
"heading": "Testing Config Before Reloading",
"body": "<p>Always test before reloading. A bad config will prevent nginx from reloading, but it will <em>not</em> take down the running process—the old config stays live.</p>",
"code": "nginx -t\n# or\nnginx -T # prints the full parsed config"
},
{
"heading": "Reloading vs Restarting",
"body": "<p>Use reload, not restart. Reload applies the new config without dropping existing connections.</p>",
"code": "systemctl reload nginx\n\n# Only use restart if you have to—it drops active connections.\nsystemctl restart nginx"
},
{
"heading": "Common Syntax Errors",
"body": "<ul><li>Missing semicolon at the end of a directive</li><li>Missing closing brace <code>}</code> on a block</li><li>Typo in a directive name (nginx will report \"unknown directive\")</li><li>Referencing a cert file or log path that does not exist</li><li>Duplicate <code>listen</code> directives on the same port across multiple vhosts without <code>default_server</code> resolution</li></ul><p>The error message from <code>nginx -t</code> includes the file name and line number. Read it.</p>"
},
{
"heading": "Useful Log Paths",
"body": "<p>Default paths on Debian/Ubuntu:</p>",
"code": "/var/log/nginx/error.log\n/var/log/nginx/access.log\n\n# Per-vhost logs are usually defined in the server block:\naccess_log /var/log/nginx/mysite.access.log;\nerror_log /var/log/nginx/mysite.error.log;"
},
{
"heading": "Quick Vhost Template",
"body": "<p>Minimal working vhost for a static site:</p>",
"code": "server {\n listen 80;\n server_name example.internal;\n\n root /var/www/example;\n index index.html;\n\n location / {\n try_files $uri $uri/ =404;\n }\n\n access_log /var/log/nginx/example.access.log;\n error_log /var/log/nginx/example.error.log;\n}"
}
]
}
@@ -0,0 +1,49 @@
{
"id": "package-management",
"title": "Package Management & Version Pinning",
"category": "packages",
"tags": ["apt", "pacman", "packages", "pinning", "rollback", "IgnorePkg"],
"updated": "2026-01-08",
"summary": "Installing, rolling back, and pinning packages on Debian and Arch Linux.",
"sections": [
{
"heading": "Debian / Ubuntu (apt)",
"body": "<p>Most commands need root.</p>",
"code": "apt update # refresh package list\napt install nginx # install\napt remove nginx # remove (keep config)\napt purge nginx # remove + delete config\napt list --installed # list installed packages\napt show nginx # info about a package\ndpkg -l | grep nginx # alternative listing"
},
{
"heading": "Listing Available Versions (Debian)",
"body": "",
"code": "apt-cache policy nginx\n# Shows installed version, candidate version, and all available versions by priority"
},
{
"heading": "Installing a Specific Version (Debian)",
"body": "",
"code": "apt install nginx=1.22.1-9\n# Use apt-cache policy to find the exact version string first"
},
{
"heading": "Pinning a Package (Debian)",
"body": "<p>Pinning prevents apt from upgrading a specific package. Create or edit <code>/etc/apt/preferences.d/</code>:</p>",
"code": "# /etc/apt/preferences.d/nginx-pin\nPackage: nginx\nPin: version 1.22.1-9\nPin-Priority: 1001\n\n# Priority > 1000 = keep this version even if newer is available\n# After creating the file:\napt-mark hold nginx # belt-and-suspenders hold\napt-cache policy nginx # verify the pin took effect"
},
{
"heading": "Arch Linux (pacman)",
"body": "",
"code": "pacman -Syu # update all\npacman -S nginx # install\npacman -R nginx # remove\npacman -Rs nginx # remove + unneeded deps\npacman -Q | grep nginx # list installed\npacman -Qi nginx # info about installed package"
},
{
"heading": "Rolling Back a Package (Arch)",
"body": "<p>Arch keeps a package cache in <code>/var/cache/pacman/pkg/</code>. If the current package broke something:</p>",
"code": "ls /var/cache/pacman/pkg/nginx*\n# Find the version you want, then:\npacman -U /var/cache/pacman/pkg/nginx-1.24.0-1-x86_64.pkg.tar.zst"
},
{
"heading": "Preventing Upgrades (Arch — IgnorePkg)",
"body": "<p>After rolling back, prevent the package from upgrading on the next <code>pacman -Syu</code>:</p>",
"code": "# /etc/pacman.conf\n[options]\n...\nIgnorePkg = nginx\n\n# Verify:\npacman -Syu\n# Should print: warning: nginx: ignoring package upgrade (1.24.0-1 => 1.25.x-y)"
},
{
"heading": "When to Pin vs When to Fix",
"body": "<p>Pinning is a stop-gap, not a solution. Document why you pinned it and set a reminder to revisit. A pinned package stops receiving security updates. If the upstream bug is fixed in a newer minor version, upgrade to that instead of staying pinned indefinitely.</p>"
}
]
}
@@ -0,0 +1,39 @@
{
"id": "ssh-access-controls",
"title": "SSH Server Access Controls",
"category": "access",
"tags": ["ssh", "sshd_config", "AllowUsers", "AllowGroups", "security", "hardening"],
"updated": "2025-10-29",
"summary": "Restricting who can SSH in using sshd_config directives.",
"sections": [
{
"heading": "The Config File",
"body": "<p>SSH server configuration lives in <code>/etc/ssh/sshd_config</code>. Drop-in overrides can go in <code>/etc/ssh/sshd_config.d/*.conf</code>.</p><p><strong>Always test your config before reloading:</strong></p>",
"code": "sshd -t\n# If it prints nothing and exits 0, the config is valid.\nsystemctl reload ssh"
},
{
"heading": "AllowUsers and AllowGroups",
"body": "<p>These are whitelist directives. If either is set, only matching users or group members can log in. If neither is set, all users may try.</p>",
"code": "# Only these users may log in\nAllowUsers alice bob deploy\n\n# Only members of these groups may log in\nAllowGroups sshusers ops\n\n# Combining: user must match AllowUsers AND (if AllowGroups is set) be in an allowed group\n# These are independent filters—if both are set, a user must satisfy both."
},
{
"heading": "DenyUsers and DenyGroups",
"body": "<p>Blacklist alternatives. <code>DenyUsers</code> and <code>DenyGroups</code> are checked before Allow rules.</p><p>Prefer <code>AllowUsers</code>/<code>AllowGroups</code> over Deny lists—it is safer to enumerate who <em>can</em> in rather than who cannot.</p>"
},
{
"heading": "Other Common Restrictions",
"body": "",
"code": "# Disable root login entirely (recommended)\nPermitRootLogin no\n\n# Disable password authentication (once keys are working)\nPasswordAuthentication no\n\n# Change the listening port (minor obscurity, not real security)\nPort 2222\n\n# Restrict to specific network interface\nListenAddress 10.42.0.1\n\n# Idle session timeout (seconds × count before disconnect)\nClientAliveInterval 300\nClientAliveCountMax 2"
},
{
"heading": "Match Blocks",
"body": "<p>You can apply different rules to specific users, groups, or source addresses:</p>",
"code": "# Allow password auth only from the management network\nMatch Address 10.42.0.0/24\n PasswordAuthentication yes\n\n# Give one user a restricted shell\nMatch User backup-agent\n ForceCommand /usr/local/bin/backup-only\n AllowTcpForwarding no"
},
{
"heading": "Checking Who Has Access",
"body": "<p>There is no built-in command to list all users who currently satisfy the access rules. Check manually:</p>",
"code": "# Current AllowUsers/AllowGroups settings\ngrep -iE '(AllowUsers|AllowGroups|DenyUsers|DenyGroups)' /etc/ssh/sshd_config\n\n# Members of a group\ngetent group sshusers\n\n# All users with a valid shell (can SSH in if no restrictions)\ngrep -v '/nologin\\|/false' /etc/passwd"
}
]
}
+38
View File
@@ -0,0 +1,38 @@
{
"id": "ssh-keys",
"title": "SSH Key Authentication",
"category": "access",
"tags": ["ssh", "authorized_keys", "keys", "permissions"],
"updated": "2025-11-03",
"summary": "How SSH key auth works and how to set it up correctly.",
"sections": [
{
"heading": "How It Works",
"body": "<p>SSH key authentication replaces passwords with a cryptographic key pair. The <strong>private key</strong> stays on your machine. The <strong>public key</strong> goes into <code>~/.ssh/authorized_keys</code> on the target host. When you connect, the server checks whether your private key corresponds to one of the public keys it trusts.</p><p>There is no password transmitted. Either the key matches or the connection fails.</p>"
},
{
"heading": "Generating a Key Pair",
"body": "<p>Use <code>ed25519</code> unless something forces you onto RSA. It is smaller and more secure.</p>",
"code": "ssh-keygen -t ed25519 -C \"your-comment-here\"\n# Accept the default path (~/.ssh/id_ed25519) or specify one.\n# Passphrase is optional but recommended for keys that leave your machine."
},
{
"heading": "Installing the Public Key",
"body": "<p>Copy the public key to the remote host:</p>",
"code": "# Option 1 — if password auth is still working\nssh-copy-id -i ~/.ssh/id_ed25519.pub user@host\n\n# Option 2 — manually\ncat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys"
},
{
"heading": "File and Directory Permissions",
"body": "<p>This is the most common reason key auth fails. SSH will silently reject keys if the permissions are too open.</p>",
"code": "chmod 700 ~/.ssh\nchmod 600 ~/.ssh/authorized_keys\nchown -R youruser:youruser ~/.ssh"
},
{
"heading": "Troubleshooting",
"body": "<p>Run <code>ssh -v user@host</code> for verbose output. The auth failure reason is usually in the first 20 lines.</p><p>Common causes:</p><ul><li><code>authorized_keys</code> file has wrong permissions (see above)</li><li><code>~/.ssh</code> directory is world-writable</li><li><code>authorized_keys</code> file does not exist</li><li>The file exists but is empty or the key was pasted with a line break in the middle</li><li><code>sshd_config</code> has <code>PubkeyAuthentication no</code></li></ul>"
},
{
"heading": "Checking the sshd Config",
"body": "<p>Relevant lines in <code>/etc/ssh/sshd_config</code>:</p>",
"code": "PubkeyAuthentication yes\nAuthorizedKeysFile .ssh/authorized_keys\n\n# After editing sshd_config, test before reloading:\nsshd -t\nsystemctl reload ssh"
}
]
}
+44
View File
@@ -0,0 +1,44 @@
{
"id": "time-sync",
"title": "System Time & NTP",
"category": "sysadmin",
"tags": ["ntp", "time", "timedatectl", "timesyncd", "chrony", "drift"],
"updated": "2025-07-14",
"summary": "Keeping system clocks accurate and diagnosing time drift.",
"sections": [
{
"heading": "Why System Time Matters",
"body": "<p>Clocks that drift cause more problems than you expect: SSL certificate validation failures, log timestamps that do not correlate across machines, cron jobs that fire at the wrong time, authentication tokens that expire prematurely, and package signature checks that fail.</p><p>On a server, time should be correct to within a second. Most NTP implementations keep it within milliseconds.</p>"
},
{
"heading": "Checking Current Time Status",
"body": "",
"code": "timedatectl\n# Shows: local time, UTC time, timezone, NTP sync status, RTC time\n\ntimedatectl show\n# Machine-readable version of the same"
},
{
"heading": "systemd-timesyncd",
"body": "<p>Most Debian/Ubuntu systems ship with <code>systemd-timesyncd</code> as the default NTP client. It is a lightweight SNTP implementation—adequate for most servers.</p>",
"code": "# Enable and start\nsystemctl enable --now systemd-timesyncd\n\n# Check sync status\ntimedatectl timesync-status\n\n# Force a resync\nsystemctl restart systemd-timesyncd\n\n# Config file (NTP servers, fallback)\ncat /etc/systemd/timesyncd.conf"
},
{
"heading": "NTP Server Configuration",
"body": "<p>The default NTP servers are usually fine. If you need to change them—for example, to use an internal NTP server:</p>",
"code": "# /etc/systemd/timesyncd.conf\n[Time]\nNTP=ntp.internal.example.com\nFallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org"
},
{
"heading": "chrony (alternative)",
"body": "<p>chrony is a more capable NTP implementation. It handles intermittent network connections and large initial offsets better than timesyncd. On systems where accuracy matters:</p>",
"code": "apt install chrony\nsystemctl enable --now chrony\n\nchronyc tracking # current sync status\nchronyc sources -v # configured time sources and their offsets"
},
{
"heading": "Diagnosing Time Problems",
"body": "",
"code": "# Is NTP enabled?\ntimedatectl | grep NTP\n\n# Is timesyncd active?\nsystemctl status systemd-timesyncd\n\n# Did a sync happen recently?\njournalctl -u systemd-timesyncd --since \"1 hour ago\"\n\n# What is the current offset?\ntimedatectl timesync-status | grep Offset"
},
{
"heading": "Setting Timezone",
"body": "",
"code": "timedatectl list-timezones | grep Europe\ntimedatectl set-timezone Europe/London"
}
]
}