Repeated disk reads during retransformation contributing to iowait spikes

[alexandergunnarson]

Describe the bug

I've been trying to figure out why our OTEL-instrumented JVMs keep experiencing massively high iowait, and I suspect the OTEL Java agent is a primary cause.

Below is partial output from running sudo /usr/share/bcc/tools/filetop on an EC2 instance of mine:

There are other class files this happens to, but Throwable.class is the most common. I'd expect

Steps to reproduce

Start a JVM with the OTEL Java agent, with probably any real-world .jar. Observe via sudo /usr/share/bcc/tools/filetop that there are repeated reads from class files.

Expected behavior

I would expect that the class bytes are fully cached in memory upon first read. I suppose using Caffeine or similar with configurable eviction strategy is fine, but really I'd like zero unnecessary disk reads.

Actual behavior

There are repeated reads from class files, contributing to massive iowait spikes.

Javaagent or library instrumentation version

v2.9.0

Environment

JDK: 24ea (perhaps the EA is part of the problem?)
OS: Amazon Linux, within Docker

Additional context

No response

[laurit]

Are you seeing this only when your application starts?

[alexandergunnarson]

No, it's ongoing throughout the lifetime of the application, and has huge spikes at times. (Why there are spikes, I'm not sure.) sudo /usr/share/bcc/tools/filetop — and bpftrace, though the signal is weaker because it's complicated to ensure that reported reads are actually persistent disk reads and not ramdisk reads — both show that java.lang.Throwable is by far the most common one to get read (sometimes thousands of times a second), but there are others. When I turn off all agents, our application has essentially zero disk reads. When I turn on the OTEL agent exclusively, there are thousands per second.

[alexandergunnarson]

For now the way I've gotten around it is to mount both our uberjar and the JDK to a ramdisk. Now there are essentially zero disk reads and the iowait + corresponding service lockups are gone.

[laurit]

Is your application constantly defining new classes? Have you tried to profile your application or capture stack traces that would show what it is doing and which code path might trigger the disk access?