Control log growth and verify disk space recovery

Database writes can fail even when no new application files have appeared on disk. A common cause is an error log that keeps repeating the same messages until it fills the system partition. Controlling log growth means managing writes, rotation, and retention separately—and verifying that space has actually been reclaimed. This guide uses Debian 12 to cover the system journal and file logs from short-running tasks, without changing database files or the existing storage layout.

CloudSFP dedicated servers come with two directly attached 1TB NVMe drives and no hardware RAID. Their combined capacity is not a measure of the free space available on the filesystem that holds the logs; log budgets must be based on the actual mount point. All capacity limits below are examples, not service defaults.

1. Locate usage and distinguish disk blocks from inodes

After payment, check your credentials in the console and connect over SSH. If the server has multiple addresses, use the first one. Start with these read-only checks:

df -hT / /var/log
df -i / /var/log
findmnt -T /var/log
sudo du -xhd1 /var/log
sudo journalctl --disk-usage

df reports filesystem usage, while du measures files visible within a directory. Their totals do not have to match. The -x option prevents traversal into other filesystems, reducing the risk of including separately mounted volumes in the results. If /var/log is not a separate partition, it shares space on the root partition with system files.

If free space is still available but inodes are nearly exhausted, look for large numbers of small files rather than just the largest logs. Record the filesystem, available space, inode usage, and fastest-growing directories before deciding what to address. Do not write diagnostic records to a partition that is already nearly full.

Do not delete the entire /var/log directory or truncate incident logs before preserving them. Before cleanup, save evidence for the relevant time range and confirm that the target is not database data, a transaction log, or a backup file.

2. Set a separate capacity budget for journald

Persistent journal files are usually stored in /var/log/journal. Managing limits through a drop-in configuration makes changes easier to audit and undo than overwriting the main configuration. The following example sets a journal capacity budget and reserves free space on the filesystem:

sudo install -d -m 0755 /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/60-capacity.conf >/dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=4G
MaxFileSec=1day
EOF

sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
sudo journalctl --disk-usage
sudo systemctl status systemd-journald --no-pager

SystemMaxUse controls the journal's storage budget, while SystemKeepFree requires space to be kept free on the same filesystem. Actual behavior also depends on the space currently available. These settings do not limit files written directly by applications. MaxFileSec sets the maximum time span covered by an active file; it does not mean that only one day of logs is retained.

If another program also writes the same messages to text logs, limiting the journal does not remove that duplicate copy. Check the logging pipeline and manage each destination separately. Do not disable the system logging service without first checking how it is used.

Rotate before cleanup, then check the results

sudo journalctl --rotate
sudo journalctl --vacuum-size=1G
sudo journalctl --disk-usage
df -h /var/log

Vacuuming deletes only archived files, not active files, so the final usage is not guaranteed to match the specified value exactly. Restarting journald is part of a configuration change: check the current service status beforehand, then verify that no new errors have appeared and that application logs are still reaching the journal.

3. Assign clear responsibility for file log rotation

First establish which program writes the logs, whether it keeps the file open for long periods, and whether an existing rule already covers the path. Do not manage the same file with both the application's built-in rotation and logrotate. Avoid adding broad patterns that duplicate existing system rules.

Install the inspection tools:

sudo apt-get update
sudo apt-get install logrotate lsof

Suppose a short-running task opens /var/log/local-batch/run.log each time it starts and closes the file when it finishes. You can add the following to /etc/logrotate.d/local-batch:

/var/log/local-batch/run.log {
    daily
    maxsize 100M
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640
}

The directory and original log file must already exist, with ownership appropriate for the task's account. When no owner is specified, create 0640 keeps the original file's owner and group. If an unprivileged account can write to the directory, assess the su directive and directory permissions separately rather than copying this rule unchanged.

sudo logrotate -d /etc/logrotate.conf
systemctl list-timers --all --no-pager

The -d option only shows a dry run; it does not rotate files. Check how often the timer or cron job actually runs: maxsize is checked only when logrotate is invoked, not enforced as a real-time hard limit. Likewise, rotate 7 retains seven archives; it does not necessarily mean seven days.

Long-running processes need a log-reopening mechanism supported by the application. After a file is renamed, the process may continue writing to the old file. Do not guess which signal to send or use copytruncate by default; copying and truncating leave a window in which log entries can be lost.

4. Test rotation and permissions in an isolated directory

Do not test a new rule by forcing rotation against the global configuration. This test uses a separate directory and state file and does not touch production logs:

umask 077
test_dir=$(mktemp -d)
printf '%0200d\n' $(seq 1 20) >"$test_dir/run.log"

cat >"$test_dir/rules" <<EOF
"$test_dir/run.log" {
    size 1k
    rotate 2
    create 0640
}
EOF

logrotate -d -s "$test_dir/state" "$test_dir/rules"
logrotate -s "$test_dir/state" "$test_dir/rules"
ls -l "$test_dir"
printf 'after rotation\n' >>"$test_dir/run.log"
cat "$test_dir/run.log"

You should see an archived file, a newly created active file, and the expected permissions. The final write only proves that reopening the path works; it does not prove that a long-running application will switch files automatically. Production validation must also have the real task generate an identifiable entry and confirm that it appears in the new active file.

Keep the test directory while reviewing the results, then remove it once they are confirmed. Production rules should continue to use the system's existing state file. Do not run manual invocations and scheduled jobs at the same time, or manage the same log with two state files on an ongoing basis.

5. Verify space recovery and address repeated errors

If deleting files does not increase free space, first check for files that processes still hold open:

sudo lsof +L1
df -h /var/log
df -i /var/log
sudo du -xhd1 /var/log

lsof +L1 lists open files with a link count below one. Assess each entry using its path, size, and process; do not assume that every result is a log. Once the target is confirmed, reopen the log as documented by the application, restart the relevant service if necessary, and check df again. Do not force space recovery by terminating unidentified processes or writing to their file descriptors.

Keep at least four validation results: new entries reach the active file, the archive count follows the rule, the account can still write, and filesystem free space is not continuing to fall abnormally. If error messages still repeat in large batches, fix the source of failed retries, invalid requests, or debug output. Rotation is not a substitute for resolving the underlying fault.

A capacity budget also cannot guarantee a fixed span of log history. Logs grow faster during peaks, so the same budget causes older records to be discarded sooner. Evidence that needs long-term retention requires a separate design for transfer, access control, and backups, with actual read-back verification. Local log rotation only limits local storage usage; it does not serve as a backup.

Frequently asked questions