Validate data disk mounts before starting services

When a data disk fails to mount, the application may not report an error immediately. The intended mount point, /srv/data, remains an ordinary directory on the system disk, and the service may keep writing there until the system disk runs out of space. Once the mount is restored, those files are hidden beneath it, leaving two sets of data that can be difficult to tell apart. Before putting the server into production, check the filesystem behind the path—not just whether the directory exists.

1. Identify the data disk and define the scope of changes

This guide uses Ubuntu Server 24.04 LTS and a data partition with an existing ext4 filesystem. It does not cover partitioning or formatting. Run the commands on the server over SSH. CloudSFP connection credentials are available in the console after payment; if multiple addresses are listed, use the first one.

The standard configuration has two directly attached 1TB NVMe drives, with no hardware RAID. Device numbers do not identify which drive is the system disk or the data disk, nor do they indicate any redundancy. If software RAID or another storage layer is already in use, mount the filesystem on that layer rather than bypassing it to mount a member drive.

lsblk -o NAME,PATH,SIZE,TYPE,FSTYPE,UUID,MOUNTPOINTS
sudo blkid
findmnt -T /
findmnt -T /srv

Identify the target partition using its capacity, filesystem type, existing mount points and installation records, then record its UUID. A UUID is more stable than an enumerated device name, but it is not inherently unique: cloning a filesystem can also copy its UUID. Resolve any duplicate UUIDs before adding an entry to the boot configuration.

Make the following changes before putting the data disk into production. On a server already running workloads, stop writes first and verify the backups and recovery procedure. Do not unmount an active filesystem just to run a test.

2. Test the mount before adding it to fstab

Create an empty mount point and check whether it already contains any files, including hidden files. If the directory is not empty, establish where the files came from before proceeding. Do not delete or overwrite them.

sudo mkdir -p /srv/data
sudo find /srv/data -mindepth 1 -maxdepth 1 -print
read -r -p 'Filesystem UUID: ' DATA_UUID
sudo blkid -U "$DATA_UUID"
sudo mount -t ext4 -o ro "UUID=$DATA_UUID" /srv/data
findmnt -M /srv/data -o TARGET,SOURCE,FSTYPE,OPTIONS,UUID

Proceed with the mount only if the blkid output matches the recorded disk identity. The initial read-only mount is for inspecting the contents, not for repairing a damaged filesystem. Here, findmnt -M checks the exact mount point. By contrast, findmnt -T finds the filesystem containing a path and may return the root filesystem if the data disk is not mounted. A successful exit status from that command alone does not prove that the intended mount is in place.

After confirming the source and contents, unmount the read-only mount. Back up /etc/fstab before editing it, and do not add a duplicate entry for the same mount point. Replace the UUID placeholder in this example with the actual value you have just verified:

UUID=<已核对的文件系统UUID> /srv/data ext4 defaults,nofail,x-systemd.device-timeout=15s 0 2

This entry allows the system to continue booting when the disk is missing. The 15-second device timeout is a setting chosen for this example, not a delivery or recovery commitment. If the workload requires a missing disk to prevent the system from booting, design a separate boot policy rather than copying nofail.

sudo umount /srv/data
sudo cp -a /etc/fstab /etc/fstab.before-data-mount
sudoedit /etc/fstab
sudo findmnt --verify --verbose
sudo systemctl daemon-reload
sudo mount /srv/data
findmnt -M /srv/data -o TARGET,SOURCE,FSTYPE,OPTIONS,UUID

If validation fails, correct the configuration before attempting the mount. Mount only the target path; avoid mounting all filesystems in one command, which can obscure other configuration problems. Passing the syntax check does not guarantee that the device is available. The actual mount result still needs to be verified.

3. Tie service startup to the mount

nofail controls system boot behavior; it does not protect the application. The service that uses the data directory must declare a mount dependency. In the following example, app.service stands for an existing application unit. Replace it with the actual unit name and confirm that the unit really starts the process that writes data.

Run sudo systemctl edit app.service and add:

[Unit]
RequiresMountsFor=/srv/data
AssertPathIsMountPoint=/srv/data

RequiresMountsFor brings up the mounts needed to access this path and establishes the startup order. The assertion rejects a state in which the directory exists but is not a mount point. Neither setting can identify the wrong disk or replace the application's data consistency checks, so verifying the UUID and its source remains a separate step.

sudo systemctl daemon-reload
sudo systemctl cat app.service
sudo systemctl show app.service -p Requires -p After

Check that the override has taken effect and that the application's actual data directory is under /srv/data. If a startup script redirects the path elsewhere, the application may write to the wrong location even with correct dependencies. Scheduled jobs, manually run scripts and other services do not automatically inherit these constraints; check each of them separately.

Create application subdirectories only after the mount succeeds, then set ownership and permissions for the actual service account. Do not give every account write access to the entire data disk, and do not put credentials in the mount configuration.

4. Test a missing mount and recovery before production writes

Run this test before there is any production data or active writing. Stop the application first, then stop the data mount unit. Verify the unit name for /srv/data with systemd-escape --path --suffix=mount /srv/data; do not guess unit names manually for complex paths.

sudo systemctl stop app.service
sudo systemctl stop srv-data.mount
sudo systemctl mask --runtime srv-data.mount
sudo systemctl start app.service
sudo systemctl status app.service srv-data.mount --no-pager

Temporarily masking the mount unit simulates an unavailable mount and prevents the application startup from automatically mounting it again. The expected result is a failed start with no application process running. If unmounting reports that the filesystem is busy, identify what is using it. Do not bypass this check with a forced or lazy unmount.

At this point, check whether any new files have appeared in the underlying /srv/data directory. After the test, remove the mask first, then restore the mount:

sudo systemctl unmask --runtime srv-data.mount
sudo systemctl start srv-data.mount
findmnt -M /srv/data -o TARGET,SOURCE,FSTYPE,OPTIONS,UUID
sudo systemctl reset-failed app.service
sudo systemctl start app.service

Recovery is complete only when the source matches and the application's health checks pass. Before finishing acceptance testing, confirm that the temporary mask has been removed. Startup dependencies primarily protect the startup phase; they are not a detector for disk failures during operation. Application error handling and monitoring are still required.

5. Keep a verifiable deployment record

Record the disk identity, UUID, mount point, filesystem type, fstab changes, service override and missing-disk test results. Schedule reboot testing before putting the workload into production, and confirm that the mount completes before the service starts. A single successful manual mount is not a substitute for boot validation.

If system disk usage looks abnormal after mounting the data disk, first check for old files hidden beneath the mount. Stop the relevant writes and arrange an inspection. Do not unmount the filesystem on a live system and immediately delete files, and do not blindly merge the underlying directory with the data disk directory.

The final acceptance criteria are: the source is correct, the path is genuinely mounted, the service dependency is effective, no writes go to the wrong location when the disk is unavailable, and health checks pass after recovery. Mount configuration provides neither redundancy nor backups; both must be designed and validated separately.

Frequently asked questions