{
  "id": "disk-logs",
  "title": "Disk Space & Log Rotation",
  "category": "storage",
  "tags": ["disk", "df", "du", "logs", "logrotate", "cleanup"],
  "updated": "2025-08-22",
  "summary": "Finding what is filling the disk and keeping logs from growing unbounded.",
  "sections": [
    {
      "heading": "Checking Disk Usage",
      "body": "<p><code>df</code> shows you how full each filesystem is. <code>du</code> tells you where the space went.</p>",
      "code": "df -h                        # human-readable filesystem summary\ndf -h /var/log               # check a specific mount\n\ndu -sh /var/log/*            # top-level breakdown of /var/log\ndu -sh /var/* | sort -rh     # sort by size, largest first\ndu -sh /var/log/*.log        # sizes of individual log files"
    },
    {
      "heading": "Finding Large Files",
      "body": "<p>When du does not point at an obvious culprit:</p>",
      "code": "# Files over 100MB anywhere on the system\nfind / -xdev -size +100M -type f 2>/dev/null\n\n# Files in /var that have grown recently\nfind /var -xdev -mtime -1 -size +10M -type f 2>/dev/null"
    },
    {
      "heading": "Emergency Cleanup",
      "body": "<p>If disk is at 100% and a service is failing because of it:</p>",
      "code": "# Truncate a log file without deleting it (safe for running processes)\ntruncate -s 0 /var/log/nginx/access.log\n\n# Remove old compressed logs (the .gz files are already rotated)\nrm /var/log/nginx/*.gz\n\n# Clear journald logs older than 2 days\njournalctl --vacuum-time=2d"
    },
    {
      "heading": "logrotate Basics",
      "body": "<p>logrotate is the standard tool for rotating and compressing logs on a schedule. It is usually run daily from cron. Config files live in <code>/etc/logrotate.d/</code>—one file per service.</p>"
    },
    {
      "heading": "Writing a logrotate Config",
      "body": "<p>Example for an nginx access log:</p>",
      "code": "/var/log/nginx/access.log {\n    daily\n    rotate 14\n    compress\n    delaycompress\n    missingok\n    notifempty\n    sharedscripts\n    postrotate\n        /bin/kill -USR1 $(cat /run/nginx.pid 2>/dev/null) 2>/dev/null || true\n    endscript\n}"
    },
    {
      "heading": "Testing logrotate",
      "body": "<p>Run logrotate manually in debug mode to verify a config without actually rotating anything:</p>",
      "code": "logrotate -d /etc/logrotate.d/nginx\n\n# To force a rotation right now (useful for testing):\nlogrotate -f /etc/logrotate.d/nginx"
    },
    {
      "heading": "Key logrotate Directives",
      "body": "<table><tr><th>Directive</th><th>Meaning</th></tr><tr><td><code>daily/weekly/monthly</code></td><td>Rotation frequency</td></tr><tr><td><code>rotate N</code></td><td>Keep N old copies</td></tr><tr><td><code>compress</code></td><td>gzip old files</td></tr><tr><td><code>delaycompress</code></td><td>Skip compressing the most recent rotation (useful when the app still has it open)</td></tr><tr><td><code>missingok</code></td><td>Do not error if the log file does not exist</td></tr><tr><td><code>notifempty</code></td><td>Skip rotation if the file is empty</td></tr><tr><td><code>size 100M</code></td><td>Rotate when file exceeds this size instead of on schedule</td></tr></table>"
    }
  ]
}