Skip to content

Commit ffb2b62

Browse files
committed
fix(disk-io): emit per-disk I/O as gauges instead of continuous counters
The per-disk perfdata used the continuous-counter UOM ('c') and emitted raw cumulative /proc/diskstats values (read_bytes, busy_time, ...), which forces Grafana to non_negative_difference() over millions of points and breaks aggregation. Emit ready-to-graph gauges instead, computed from the deltas the plugin already keeps in SQLite: read/write bytes per second (current and period average), total throughput, and disk busy percentage (%util). See #320. This changes the per-disk metric names, so it is a breaking change for existing dashboards. The Grafana "Bytes per Second" panel now plots the gauges directly (the obsolete non_negative_difference step is removed) and the cumulative "Bytes" / "Times" panels are replaced by a disk "Busy" (%util) panel. Update README and CHANGELOG.
1 parent f0d65f7 commit ffb2b62

4 files changed

Lines changed: 84 additions & 266 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Monitoring Plugins:
1919
Monitoring Plugins:
2020

2121
* disk-io: iowait is now reported as saturated CPU cores (e.g. `1.02 cores`) instead of a percentage that could exceed 100% on multi-core hosts, and gets its own Grafana panel; the alerting thresholds keep their meaning and defaults, so no reconfiguration is needed
22+
* disk-io: per-disk performance data now reports read/write throughput per second and disk busy percentage as ready-to-graph values instead of raw cumulative counters; re-import the disk-io Grafana dashboard after updating ([#320](https://github.com/Linuxfabrik/monitoring-plugins/issues/320))
2223

2324
### Fixed
2425

‎check-plugins/disk-io/README.md‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ Checks disk I/O bandwidth over time and alerts on sustained saturation, not shor
77

88
On Linux, the check also monitors how much I/O wait the system accumulates (CPU time spent waiting for I/O). The machine-wide iowait reported by the kernel is normalized to a per-core basis and shown as *saturated cores*: `1.0 cores` means the equivalent of one CPU core spent the whole interval waiting for I/O, `2.0 cores` means two, and so on. This is the same mental model as the load average. Reporting per-core instead of as a percentage of the whole machine is deliberate: it lets you deploy one identical threshold to an entire fleet. `--iowait-warning` and `--iowait-critical` are expressed in percent of a single core (default 80/90), so `90` means "alert when 1 core is stuck 90% in I/O wait", and that means the same thing on a 4-core VM and a 64-core VM, even when the core count changes over time. Like bandwidth alerts, iowait alerts require COUNT consecutive threshold violations.
99

10-
Perfdata is emitted for each disk (busy_time, read_bytes, read_time, write_bytes, write_time) and for iowait, so you can graph trends. On Linux the check focuses on block devices with a mounted filesystem by default; use `--include-unmounted` to also include raw, unmounted devices such as multipath SAN volumes. On Windows it uses psutil's disk counters. Optionally, `--top` lists the processes that generated the most I/O traffic (read/write totals) to help identify offenders.
10+
Perfdata is emitted for each disk (read/write throughput per second and disk busy percentage) and for iowait, so you can graph trends. On Linux the check focuses on block devices with a mounted filesystem by default; use `--include-unmounted` to also include raw, unmounted devices such as multipath SAN volumes. On Windows it uses psutil's disk counters. Optionally, `--top` lists the processes that generated the most I/O traffic (read/write totals) to help identify offenders.
1111

1212
This check is cross-platform and works on Linux, Windows, and all psutil-supported systems. The check stores its short trend state locally in an SQLite DB to evaluate sustained load across runs.
1313

@@ -67,8 +67,8 @@ fleet-wide threshold works on every host regardless of its core count:
6767
--iowait-warning and --iowait-critical are given in percent of one core
6868
(default 80/90, i.e. alert near one saturated core). Like bandwidth alerts,
6969
iowait alerts require COUNT consecutive threshold violations. Perfdata is
70-
emitted for each disk (busy_time, read_bytes, read_time, write_bytes,
71-
write_time) and for iowait, so you can graph trends. On Linux the check
70+
emitted for each disk (read/write throughput per second and disk busy
71+
percentage) and for iowait, so you can graph trends. On Linux the check
7272
focuses on block devices with a mounted filesystem by default; use `--include-
7373
unmounted` to also include raw, unmounted devices such as multipath SAN
7474
volumes. On Windows it uses psutil's disk counters. Optionally, `--top` lists
@@ -200,15 +200,17 @@ Global:
200200
|----|----|----|
201201
| iowait | Number | Machine-wide I/O wait normalized to saturated CPU cores (`1.0` = one core fully waiting for I/O). Linux only. |
202202

203-
Per matched disk, where `<disk>` is the block device name:
203+
Per matched disk, where `<disk>` is the block device name. The `1` suffix is the latest interval (like a "load1"), the `15` suffix is the average over the last `--count` runs (like a "load15"):
204204

205205
| Name | Type | Description |
206206
|----|----|----|
207-
| `<disk>`\_busy_time | Continuous Counter | Time spent doing actual I/Os (in milliseconds). |
208-
| `<disk>`\_read_bytes | Continuous Counter | Number of bytes read. |
209-
| `<disk>`\_read_time | Continuous Counter | Time spent reading from disk (in milliseconds). |
210-
| `<disk>`\_write_bytes | Continuous Counter | Number of bytes written. |
211-
| `<disk>`\_write_time | Continuous Counter | Time spent writing to disk (in milliseconds). |
207+
| `<disk>`\_busy_percent | Percentage | Share of wall-clock time the device was busy with I/O over the last interval (iostat's %util). |
208+
| `<disk>`\_read_bytes_per_second1 | Bytes | Bytes read per second over the latest interval. |
209+
| `<disk>`\_read_bytes_per_second15 | Bytes | Bytes read per second, averaged over the last `--count` runs. |
210+
| `<disk>`\_throughput1 | Bytes | Read plus write bytes per second over the latest interval. |
211+
| `<disk>`\_throughput15 | Bytes | Read plus write bytes per second, averaged over the last `--count` runs. |
212+
| `<disk>`\_write_bytes_per_second1 | Bytes | Bytes written per second over the latest interval. |
213+
| `<disk>`\_write_bytes_per_second15 | Bytes | Bytes written per second, averaged over the last `--count` runs. |
212214

213215

214216
## Troubleshooting

‎check-plugins/disk-io/disk-io‎

Lines changed: 66 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ except ImportError:
3434

3535

3636
__author__ = 'Linuxfabrik GmbH, Zurich/Switzerland'
37-
__version__ = '2026061801'
37+
__version__ = '2026061802'
3838

3939
DESCRIPTION = """Checks disk I/O bandwidth over time and alerts on sustained saturation, not
4040
short spikes. The check records per-disk read/write counters and then derives current (R1/W1)
@@ -50,7 +50,7 @@ host regardless of its core count: --iowait-warning and --iowait-critical are gi
5050
of one core (default 80/90, i.e. alert near one saturated core). Like bandwidth alerts, iowait
5151
alerts require COUNT consecutive threshold violations.
5252
53-
Perfdata is emitted for each disk (busy_time, read_bytes, read_time, write_bytes, write_time)
53+
Perfdata is emitted for each disk (read/write throughput per second and disk busy percentage)
5454
and for iowait, so you can graph trends. On Linux the check focuses on block devices with a
5555
mounted filesystem by default; use `--include-unmounted` to also include raw, unmounted devices
5656
such as multipath SAN volumes. On Windows it uses psutil's disk counters. Optionally, `--top`
@@ -610,7 +610,7 @@ def main():
610610
lib.base.oao('Waiting for more data.', state)
611611

612612
# calculate current rates (like "load1")
613-
timediff, read_bytes_per_second1, write_bytes_per_second1, bandwidth1 = (
613+
timediff1, read_bytes_per_second1, write_bytes_per_second1, bandwidth1 = (
614614
get_rate(
615615
data[0]['timestamp'],
616616
data[1]['timestamp'],
@@ -620,7 +620,7 @@ def main():
620620
data[1]['write_bytes'],
621621
)
622622
)
623-
if timediff <= 0: # often happens after a reboot
623+
if timediff1 <= 0: # often happens after a reboot
624624
lib.db_sqlite.close(conn)
625625
lib.base.oao('Waiting for more data.', state)
626626

@@ -644,7 +644,7 @@ def main():
644644
if len(data) != args.COUNT: # not enough data yet
645645
continue
646646

647-
timediff, read_bytes_per_second15, write_bytes_per_second15, bandwidth15 = (
647+
timediff15, read_bytes_per_second15, write_bytes_per_second15, bandwidth15 = (
648648
get_rate(
649649
data[0]['timestamp'],
650650
data[args.COUNT - 1]['timestamp'],
@@ -654,7 +654,7 @@ def main():
654654
data[args.COUNT - 1]['write_bytes'],
655655
)
656656
)
657-
if timediff <= 0: # often happens after a reboot
657+
if timediff15 <= 0: # often happens after a reboot
658658
lib.db_sqlite.close(conn)
659659
lib.base.oao('Waiting for more data.', state)
660660

@@ -682,55 +682,66 @@ def main():
682682
}
683683
)
684684

685-
# perfdata
686-
try:
687-
perfdata += lib.base.get_perfdata(
688-
f'{bd}_busy_time',
689-
data[0]['busy_time'],
690-
uom='c',
691-
warn=None,
692-
crit=None,
693-
_min=0,
694-
_max=None,
695-
)
696-
perfdata += lib.base.get_perfdata(
697-
f'{bd}_read_bytes',
698-
data[0]['read_bytes'],
699-
uom='c',
700-
warn=None,
701-
crit=None,
702-
_min=0,
703-
_max=None,
704-
)
705-
perfdata += lib.base.get_perfdata(
706-
f'{bd}_read_time',
707-
data[0]['read_time'],
708-
uom='c',
709-
warn=None,
710-
crit=None,
711-
_min=0,
712-
_max=None,
713-
)
714-
perfdata += lib.base.get_perfdata(
715-
f'{bd}_write_bytes',
716-
data[0]['write_bytes'],
717-
uom='c',
718-
warn=None,
719-
crit=None,
720-
_min=0,
721-
_max=None,
722-
)
723-
perfdata += lib.base.get_perfdata(
724-
f'{bd}_write_time',
725-
data[0]['write_time'],
726-
uom='c',
727-
warn=None,
728-
crit=None,
729-
_min=0,
730-
_max=None,
731-
)
732-
except (KeyError, TypeError):
733-
pass
685+
# disk utilization (iostat's %util): share of wall-clock time the
686+
# device was busy with I/O over the last interval. busy_time is a
687+
# cumulative millisecond counter, so derive the value from the delta
688+
# here rather than emitting the raw counter (see CONTRIBUTING: no
689+
# continuous counters in perfdata).
690+
busy_delta = data[0]['busy_time'] - data[1]['busy_time']
691+
busy_percent = round(
692+
max(0.0, min(100.0, busy_delta / (timediff1 * 1000) * 100)),
693+
1,
694+
)
695+
696+
# perfdata: emit per-second rates and utilization as gauges, never the
697+
# raw cumulative counters. "throughput" is read + write per second; its
698+
# _15 average carries the per-disk alerting threshold (a percentage of
699+
# the observed maximum bandwidth).
700+
perfdata += lib.base.get_perfdata(
701+
f'{bd}_read_bytes_per_second1',
702+
read_bytes_per_second1,
703+
uom='B',
704+
_min=0,
705+
)
706+
perfdata += lib.base.get_perfdata(
707+
f'{bd}_read_bytes_per_second15',
708+
read_bytes_per_second15,
709+
uom='B',
710+
_min=0,
711+
)
712+
perfdata += lib.base.get_perfdata(
713+
f'{bd}_write_bytes_per_second1',
714+
write_bytes_per_second1,
715+
uom='B',
716+
_min=0,
717+
)
718+
perfdata += lib.base.get_perfdata(
719+
f'{bd}_write_bytes_per_second15',
720+
write_bytes_per_second15,
721+
uom='B',
722+
_min=0,
723+
)
724+
perfdata += lib.base.get_perfdata(
725+
f'{bd}_throughput1',
726+
bandwidth1,
727+
uom='B',
728+
_min=0,
729+
)
730+
perfdata += lib.base.get_perfdata(
731+
f'{bd}_throughput15',
732+
bandwidth15,
733+
uom='B',
734+
warn=int(bandwidth_max * args.WARN / 100),
735+
crit=int(bandwidth_max * args.CRIT / 100),
736+
_min=0,
737+
)
738+
perfdata += lib.base.get_perfdata(
739+
f'{bd}_busy_percent',
740+
busy_percent,
741+
uom='%',
742+
_min=0,
743+
_max=100,
744+
)
734745

735746
# check iowait alerting (only if enough consecutive values collected)
736747
iowait_state = STATE_OK

0 commit comments

Comments
 (0)