ECC error monitoring: validate the path from kernel to alert

When a database process exits unexpectedly and memory errors appear in the kernel log at the same time, restarting the application restores the process but does not explain the cause of the failure. ECC can correct some memory errors, but it does not automatically preserve events, assess trends, or collect evidence for a support ticket. Acceptance checks for a dedicated server should verify the entire chain: hardware reporting, kernel reception, event storage, monitoring detection, and alert delivery.

CloudSFP dedicated servers with AMD EPYC 7543P use DDR4-3200 ECC memory and are available in two memory configurations: 128GB and 256GB. The following procedure uses Debian 12 as an example. Run the commands on the host operating system, not inside a guest. Different kernels and firmware may expose different information. Record missing data sources separately; do not treat them as zero errors.

1. Distinguish errors from monitoring gaps first

Correctable errors are usually labeled CE, meaning the reported error has been corrected. Uncorrectable errors are usually labeled UE, meaning the error cannot be repaired by the correction mechanism involved. The actual impact also depends on the error location, how the kernel handles it, and the affected process. These abbreviations alone do not establish that application data is intact.

For CE events, focus on counter increases, recurring locations, and the workload at the time of the event. An isolated event and repeated events at the same location warrant different priorities. For UE events, prioritize data protection, check for terminated processes and system anomalies, review existing backups, and stop adding load to the affected workload.

“No records” is meaningful only after the collection chain has been verified as operational. An empty EDAC directory, a stopped collection service, or an unreadable event database is a monitoring gap—not evidence of healthy memory.

ECC does not replace data validation, storage redundancy, or backups. Incorrect application results and corrupted files must be correlated with memory errors using timestamps and application evidence. A causal relationship cannot simply be assumed.

2. Verify which data sources the kernel actually provides

First record the boot ID and kernel version, then inspect the memory controller directory and kernel events. These commands only read the current state; they do not load drivers or change hardware settings.

uname -r
cat /proc/sys/kernel/random/boot_id
ls -ld /sys/devices/system/edac/mc
find /sys/devices/system/edac/mc -maxdepth 2 -type f \
  \( -name ce_count -o -name ue_count -o -name mc_name \) \
  -print -exec cat {} \;
sudo journalctl -k -b --no-pager \
  --grep='EDAC|MCE|Hardware Error|memory error'

In this directory, mc_name identifies the controller, while ce_count and ue_count are cumulative counters for that controller. If the first reading is nonzero, save it as a baseline rather than treating the entire historical count as a new alert.

If the directory does not exist or contains no controller entries, preserve the command output and check the kernel log for driver initialization messages. Driver support, firmware settings, and the platform's reporting path can all affect visibility. Repeatedly loading unfamiliar modules to probe a production server is not recommended.

MCE or Hardware Error messages in the kernel log do not necessarily indicate a memory fault; the event may originate in another hardware component. Preserve the full context. Do not classify an event based on a single line containing “error.”

3. Set up rasdaemon logging and counter baselines

On Debian 12, rasdaemon can be installed from the distribution's packages. Save the existing service configuration before making changes. After installation, check the service status, startup arguments, and actual running process.

sudo apt-get update
sudo apt-get install rasdaemon
sudo systemctl enable --now rasdaemon
systemctl status rasdaemon --no-pager
systemctl cat rasdaemon
ps -C rasdaemon -o pid,args
sudo ras-mc-ctl --status
sudo ras-mc-ctl --summary
sudo ras-mc-ctl --errors

--status is only one part of the check, not a complete health assessment. Also inspect the service log for event subscription failures, permission errors, or database write problems.

sudo journalctl -u rasdaemon -b --no-pager

A running collector does not guarantee persistent records

Check whether the startup arguments enable -r or --record, and confirm that the installed package includes database support. If the service is running without persistent recording, restarting the collector can leave a gap in the evidence. Determine the database location from the actual package configuration rather than assuming the path used by another distribution.

Each sample should include at least the machine ID, boot ID, sampling time, controller name, and CE and UE counts. Compare counter increases only within the same boot ID and data source. After a system reboot, module reload, or counter decrease, establish a new baseline instead of calculating a negative delta and ignoring it.

rasdaemon and EDAC may not cover exactly the same data. Before comparing them, check the event types, collection start point, and controller. Do not expect the totals in both sets of records to match mechanically.

4. Make alerts convey risk rather than pile up messages

Separate monitoring into three categories: collection-chain failures, new CE events, and new UE events. Collection-chain failures include a stopped service, failed readings, and database query errors. Alert on these independently; do not replace an error condition with a numeric zero.

Notifications for new CE events should include the current delta, controller, and original event. If the records provide location fields, repeated events can be grouped by location, but there is no single “safe count” suitable for every machine. Event rate, concentration, and application symptoms should jointly determine whether to escalate.

New UE events require higher priority. Preserve the current state, then check application and data integrity. Do not use stress testing to further confirm the fault, and do not reboot the machine merely to clear the counters.

The alert chain can be validated with offline samples: create before-and-after counter inputs and separately test zero delta, a CE increase, a UE increase, a failed reading, and a changed boot ID. Clearly label test messages as drills. Feed them only into the monitoring parser, not into the real hardware event database.

Do not use error injection to validate notifications on a production host. A successful notification test proves only that parsing and delivery work. Whether the kernel can receive hardware events must still be verified separately using driver status, event sources, and actual records.

5. Preserve evidence that supports a hardware investigation

When errors recur, first export the kernel log for the current boot, the rasdaemon service log, and the event query results, then save a counter snapshot. Restrict access to the evidence directory. Before attaching files to a support ticket, check for addresses, account details, or sensitive application information.

Organize the support ticket in this order:

• Machine ID, operating system and kernel versions, and boot ID.
• Time of first detection with an explicit time zone, and whether new errors continue to appear.
• Original CE and UE events and before-and-after counts, not just alert screenshots.
• Controller and location fields from the reports, with missing fields explicitly noted.
• Application symptoms during the same period, plus actions already taken and their results.

Do not independently convert DIMM labels or addresses in events into physical slot numbers; platform mappings can differ. Leave hardware localization to support staff using the machine information. The customer renting the server is responsible for protecting the operating system, application data, and backups.

The final deliverable from these checks is not an “all green” screenshot. It is a verifiable baseline, a clear statement of data-source coverage, a tested alert chain, and complete incident evidence. Reconfirm these conditions after a kernel update or machine reboot so that failed collection is not mistaken for the disappearance of errors.

Frequently asked questions