Fix for PMM Missing Disk Stats with Linux Kernels 4.19+
We find PMM (Percona Monitoring and Management) invaluable here at Shattered Silicon. But, as with all software, every once in a while, we encounter a bug. One that has been particularly annoying is to do with PMM missing disk stats on more recent Linux kernels. On more recent Linux distributions that ship with kernel versions 4.19+, disk I/O statistics were not showing up in PMM because the PMM client’s node_exporter didn’t understand the new, extended format of /proc/diskstats. This is because of a kernel patch in 4.19 that added tracking for discard I/O requests. Some Linux distributions that are based on older kernels also suffer from this bug. This is due to their backporting the kernel feature patch into the kernel they ship.
While the patch is desirable as it provides useful additional information, it does break prometheus node_exporter’s diskstats parser. PMM 1.x (which is what we happen to prefer), is based on an older version of node_exporter that hasn’t had the above patch backported to it. Unfortunately, Percona have moved on to developing PMM 2.x, That means that the fix is less likely to be forthcoming from upstream. We backported the node_exporter patch for fixing /proc/diskstats parsing on 4.19+ kernels. We also updated our real source RPMs for PMM Client with the fixed versions (as of 1.17.4-10), which can be built on x86-64 and aarch64. Happy PMM-ing with diskstats on newer kernels. Do you need help with performance tuning and optimisation of your MySQL and MariaDB databases? Give our MySQL consultants call, we can almost certainly help.
Fixes in pmm-client 1.17.4-10:
– Backport prometheus diskstats fix
– Backport fix for delegating to Go’s standard reflect version
– Fix systemd service file update bug that kept appending “textfile” collectors, resulting in very large error logs