Stop monitors racing their own subscription when the channel closes

[slominskir-coding-agent]

Fixes the intermittent CaGetConcurrencyTest failure in CI on main (run 53: "Monitors that never reported connected expected:<0> but was:<75>").

A cloud Claude session diagnosed the failure and wrote this commit (its session is linked in the commit trailer). I reviewed it, checked its claims, and applied it unchanged except for setting the author to the repository's bot.

Cause

Since #45, a /caget and a monitor of the same PV share one CAJ channel.

• What drops the circuit: CAJ queues a monitor's EVENT_ADD until a flush, but sends EVENT_CANCEL at once (EventCancelRequest.getPriority() is SEND_IMMEDIATELY_PRIORITY; I checked the bytecode of jca 2.4.10). A cancel that closely follows its add can therefore reach the IOC first. The IOC rejects it and drops the whole circuit, with every channel on that IOC.
• Why ChannelMonitor made that likely: it never removed its connection listener or cleared its subscription on close(). On a channel a /caget kept open, a closed monitor could still subscribe when its connection callback ran late, and subscriptions piled up. When the last /caget then destroyed the channel, CAJ cancelled every subscription still on it, including that fresh one.
• Why one drop became 75 failures: when the drop landed while a monitor was closing, destroyChannel threw IllegalStateException. close() didn't catch it, so it escaped removePv's computeIfPresent and left the dead monitor mapped. Every later client of the PV was then told it was disconnected.

The CI run's server log matches: the IOC logged CAS: forcing disconnect, then epics2web logged transport closed and Unable to destroy channel channel2, followed by 76 Unable to handle client message lines.

Changes

ChannelMonitor.close() now:

• stops the monitor subscribing after close;
• removes its connection listener from the possibly shared channel;
• clears its own subscription once the first update shows the IOC has it, waiting at most 3 s;
• reports IllegalStateException as IOException, so removePv always unmaps the monitor.

MonitorEndpoint now logs the message of an IllegalStateException, which its log call used to drop.

Waiting inside computeIfPresent: the first update arrives on a CAJ thread, and nothing on that path takes monitorMap locks. Connection events go through callbackExecutor, and the closing monitor has no listeners left. The wait happens only while the channel is connected with an unconfirmed subscription, which is normally milliseconds.

Checks

Run locally with Tomcat CPU-limited (docker update --cpus) to make the race likelier:

Build	CPU	Runs of CaGetConcurrencyTest	Failures	IOC forcing disconnect
main (61761e6)	1	10	0	0
main	0.5	15	0	1
main	0.3	15	0	1
this PR	1	10	0	0
this PR	0.3	30	0	0

• What this does and doesn't show: the trigger (a forced disconnect) appears about once in 15 runs on main, and never in 30 with this PR. The 75-monitor failure itself didn't reproduce locally, so this PR's CI runs are the real confirmation.
• Other checks:
  • The full integration suite passed twice at normal CPU, with timings unchanged (CaGetConcurrencyTest 3.0 s).
  • ./gradlew spotlessCheck test passes.
• Still logged: User destroyed channel (7 in the patched runs). The handoff found these come from CAJ destroying a shared channel while a get is pending, before and after this change. They don't drop the circuit.

🤖 Generated with Claude Code