You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ffb2b62
Browse filesBrowse the repository at this point in the historyBrowse files
fix(disk-io): emit per-disk I/O as gauges instead of continuous counters
The per-disk perfdata used the continuous-counter UOM ('c') and emitted
raw cumulative /proc/diskstats values (read_bytes, busy_time, ...), which
forces Grafana to non_negative_difference() over millions of points and
breaks aggregation. Emit ready-to-graph gauges instead, computed from the
deltas the plugin already keeps in SQLite: read/write bytes per second
(current and period average), total throughput, and disk busy percentage
(%util). See #320.
This changes the per-disk metric names, so it is a breaking change for
existing dashboards. The Grafana "Bytes per Second" panel now plots the
gauges directly (the obsolete non_negative_difference step is removed) and
the cumulative "Bytes" / "Times" panels are replaced by a disk "Busy"
(%util) panel. Update README and CHANGELOG.
Copy file name to clipboardExpand all lines: CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,6 +19,7 @@ Monitoring Plugins:
19
19
Monitoring Plugins:
20
20
21
21
* disk-io: iowait is now reported as saturated CPU cores (e.g. `1.02 cores`) instead of a percentage that could exceed 100% on multi-core hosts, and gets its own Grafana panel; the alerting thresholds keep their meaning and defaults, so no reconfiguration is needed
22
+
* disk-io: per-disk performance data now reports read/write throughput per second and disk busy percentage as ready-to-graph values instead of raw cumulative counters; re-import the disk-io Grafana dashboard after updating ([#320](https://github.com/Linuxfabrik/monitoring-plugins/issues/320))
Copy file name to clipboardExpand all lines: check-plugins/disk-io/README.md
+11-9Lines changed: 11 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@ Checks disk I/O bandwidth over time and alerts on sustained saturation, not shor
7
7
8
8
On Linux, the check also monitors how much I/O wait the system accumulates (CPU time spent waiting for I/O). The machine-wide iowait reported by the kernel is normalized to a per-core basis and shown as *saturated cores*: `1.0 cores` means the equivalent of one CPU core spent the whole interval waiting for I/O, `2.0 cores` means two, and so on. This is the same mental model as the load average. Reporting per-core instead of as a percentage of the whole machine is deliberate: it lets you deploy one identical threshold to an entire fleet. `--iowait-warning` and `--iowait-critical` are expressed in percent of a single core (default 80/90), so `90` means "alert when 1 core is stuck 90% in I/O wait", and that means the same thing on a 4-core VM and a 64-core VM, even when the core count changes over time. Like bandwidth alerts, iowait alerts require COUNT consecutive threshold violations.
9
9
10
-
Perfdata is emitted for each disk (busy_time, read_bytes, read_time, write_bytes, write_time) and for iowait, so you can graph trends. On Linux the check focuses on block devices with a mounted filesystem by default; use `--include-unmounted` to also include raw, unmounted devices such as multipath SAN volumes. On Windows it uses psutil's disk counters. Optionally, `--top` lists the processes that generated the most I/O traffic (read/write totals) to help identify offenders.
10
+
Perfdata is emitted for each disk (read/write throughput per second and disk busy percentage) and for iowait, so you can graph trends. On Linux the check focuses on block devices with a mounted filesystem by default; use `--include-unmounted` to also include raw, unmounted devices such as multipath SAN volumes. On Windows it uses psutil's disk counters. Optionally, `--top` lists the processes that generated the most I/O traffic (read/write totals) to help identify offenders.
11
11
12
12
This check is cross-platform and works on Linux, Windows, and all psutil-supported systems. The check stores its short trend state locally in an SQLite DB to evaluate sustained load across runs.
13
13
@@ -67,8 +67,8 @@ fleet-wide threshold works on every host regardless of its core count:
67
67
--iowait-warning and --iowait-critical are given in percent of one core
68
68
(default 80/90, i.e. alert near one saturated core). Like bandwidth alerts,
69
69
iowait alerts require COUNT consecutive threshold violations. Perfdata is
70
-
emitted for each disk (busy_time, read_bytes, read_time, write_bytes,
71
-
write_time) and for iowait, so you can graph trends. On Linux the check
70
+
emitted for each disk (read/write throughput per second and disk busy
71
+
percentage) and for iowait, so you can graph trends. On Linux the check
72
72
focuses on block devices with a mounted filesystem by default; use `--include-
73
73
unmounted` to also include raw, unmounted devices such as multipath SAN
74
74
volumes. On Windows it uses psutil's disk counters. Optionally, `--top` lists
@@ -200,15 +200,17 @@ Global:
200
200
|----|----|----|
201
201
| iowait | Number | Machine-wide I/O wait normalized to saturated CPU cores (`1.0` = one core fully waiting for I/O). Linux only. |
202
202
203
-
Per matched disk, where `<disk>` is the block device name:
203
+
Per matched disk, where `<disk>` is the block device name. The `1` suffix is the latest interval (like a "load1"), the `15` suffix is the average over the last `--count` runs (like a "load15"):
204
204
205
205
| Name | Type | Description |
206
206
|----|----|----|
207
-
|`<disk>`\_busy_time | Continuous Counter | Time spent doing actual I/Os (in milliseconds). |
208
-
|`<disk>`\_read_bytes | Continuous Counter | Number of bytes read. |
209
-
|`<disk>`\_read_time | Continuous Counter | Time spent reading from disk (in milliseconds). |
210
-
|`<disk>`\_write_bytes | Continuous Counter | Number of bytes written. |
211
-
|`<disk>`\_write_time | Continuous Counter | Time spent writing to disk (in milliseconds). |
207
+
|`<disk>`\_busy_percent | Percentage | Share of wall-clock time the device was busy with I/O over the last interval (iostat's %util). |
208
+
|`<disk>`\_read_bytes_per_second1 | Bytes | Bytes read per second over the latest interval. |
209
+
|`<disk>`\_read_bytes_per_second15 | Bytes | Bytes read per second, averaged over the last `--count` runs. |
210
+
|`<disk>`\_throughput1 | Bytes | Read plus write bytes per second over the latest interval. |
211
+
|`<disk>`\_throughput15 | Bytes | Read plus write bytes per second, averaged over the last `--count` runs. |
212
+
|`<disk>`\_write_bytes_per_second1 | Bytes | Bytes written per second over the latest interval. |
213
+
|`<disk>`\_write_bytes_per_second15 | Bytes | Bytes written per second, averaged over the last `--count` runs. |
0 commit comments