Skip to content

Read response streams on a thread of their own, not in their callback - #747

Open
hjohn wants to merge 1 commit into
mainfrom
enhancement/connectors-signal-only-callbacks
Open

hjohn wants to merge 1 commit into
mainfrom
enhancement/connectors-signal-only-callbacks

Conversation

@hjohn

@hjohn hjohn commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Merge this before AxonIQ/AxonFramework#5251. That change makes streams reject reads from their callback, which FlowControlledResponseSender, IncomingQueryInvoker and SpringCloudQueryBusConnector do today. This change works with and without it.

  • CallbackHandOff in axoniq-distributed-messaging returns a stream callback that only schedules the work. Signals that arrive before the work starts share one run, and it runs on a virtual thread, or on a given Executor.
  • The three classes above register it instead of reading from the stream and writing the responses inside the callback.
  • Tests that expected the drain to run synchronously wait for it.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Build status (run 44130930369, commit 80c854a)

JDK 21 - SUCCESS - 33 modules (0 cached, 33 built) in 22:34

Per-module details
ModuleCacheTime
✅ axoniq-frameworkbuilt00:01
✅ axoniq-test-loggingbuilt00:01
✅ axoniq-framework-parentbuilt00:03
✅ axoniq-framework-bombuilt00:00
✅ axoniq-postgresqlbuilt00:47
✅ axoniq-distributed-messagingbuilt00:08
✅ axoniq-testcontainerbuilt00:36
✅ axon-server-connectorbuilt01:52
✅ axoniq-springcloudbuilt00:14
✅ axoniq-dead-letterbuilt00:24
✅ axoniq-event-streamingbuilt00:13
✅ axoniq-multi-tenancybuilt00:17
✅ axoniq-workflow-parentbuilt00:00
✅ axoniq-workflow-enginebuilt00:28
✅ axoniq-tracing-micrometerbuilt00:04
✅ axoniq-workflow-dslbuilt00:01
✅ axoniq-workflow-testbuilt00:09
✅ axoniq-framework-springbuilt00:00
✅ axoniq-spring-boot-autoconfigurebuilt01:34
✅ axoniq-spring-boot-starterbuilt00:00
✅ axoniq-integrationtestsbuilt00:00
✅ axoniq-integrationtests-supportbuilt00:01
✅ axoniq-integrationtests-connectorbuilt00:00
✅ axoniq-integrationtests-conformancebuilt04:41
✅ axoniq-integrationtests-tracingbuilt00:09
✅ axoniq-integrationtests-dead-letterbuilt00:03
✅ axoniq-legacybuilt02:48
✅ axoniq-integrationtests-legacybuilt00:06
✅ axoniq-integrationtests-workflowbuilt00:33
✅ axoniq-legacy-testbuilt00:07
✅ axoniq-message-transformationbuilt00:05
✅ axoniq-workflow-dsl-kotlinbuilt00:13
✅ axoniq-docs-samplesbuilt00:06

JDK 25 - SUCCESS - 33 modules (0 cached, 33 built) in 23:42

Per-module details
ModuleCacheTime
✅ axoniq-frameworkbuilt00:01
✅ axoniq-test-loggingbuilt00:01
✅ axoniq-framework-parentbuilt00:05
✅ axoniq-framework-bombuilt00:00
✅ axoniq-postgresqlbuilt00:59
✅ axoniq-distributed-messagingbuilt00:09
✅ axoniq-testcontainerbuilt00:42
✅ axon-server-connectorbuilt01:58
✅ axoniq-springcloudbuilt00:15
✅ axoniq-dead-letterbuilt00:28
✅ axoniq-event-streamingbuilt00:15
✅ axoniq-multi-tenancybuilt00:19
✅ axoniq-workflow-parentbuilt00:00
✅ axoniq-workflow-enginebuilt00:34
✅ axoniq-tracing-micrometerbuilt00:04
✅ axoniq-workflow-dslbuilt00:02
✅ axoniq-workflow-testbuilt00:10
✅ axoniq-framework-springbuilt00:00
✅ axoniq-spring-boot-autoconfigurebuilt01:44
✅ axoniq-spring-boot-starterbuilt00:00
✅ axoniq-integrationtestsbuilt00:00
✅ axoniq-integrationtests-supportbuilt00:01
✅ axoniq-integrationtests-connectorbuilt00:00
✅ axoniq-integrationtests-conformancebuilt04:30
✅ axoniq-integrationtests-tracingbuilt00:09
✅ axoniq-integrationtests-dead-letterbuilt00:03
✅ axoniq-legacybuilt03:01
✅ axoniq-integrationtests-legacybuilt00:07
✅ axoniq-integrationtests-workflowbuilt00:33
✅ axoniq-legacy-testbuilt00:08
✅ axoniq-message-transformationbuilt00:06
✅ axoniq-workflow-dsl-kotlinbuilt00:15
✅ axoniq-docs-samplesbuilt00:04

@smcvb smcvb added Priority 2: Should High priority. Ideally, these issues are part of the release they’re assigned to. Type: Enhancement Use to signal an issue enhances an already existing feature of the project. labels Nov 9, 2026
@smcvb smcvb added this to the Release 5.5.0 milestone Nov 9, 2026
@hjohn
hjohn force-pushed the enhancement/connectors-signal-only-callbacks branch 2 times, most recently from 7abf16d to b60a259 Compare November 16, 2026 10:30
@hjohn
hjohn marked this pull request as ready for review November 16, 2026 10:32
@hjohn
hjohn requested a review from a team as a code owner November 16, 2026 10:32
@hjohn
hjohn requested review from hatzlj, schananas and smcvb and removed request for a team November 16, 2026 10:32
@hjohn
hjohn force-pushed the enhancement/connectors-signal-only-callbacks branch from b60a259 to 2c6afe0 Compare November 16, 2026 13:47

@smcvb smcvb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Frankly a bit in doubt whether to request changes yes or no. The added util makes sense to me, although I can imagine it's better situated in Axon Framework. But more importantly, the LLM-concern raised on the CallbackHandOff#handOff method is something we should look into.

* @throws NullPointerException if {@code task} is {@code null}
*/
public static Runnable handOff(Runnable task) {
return handOff(Thread::startVirtualThread, task);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential helpful LLM-based comment I am leaving as is:

Curious: how does this behave on JDK 21 with the synchronized drains? IncomingQueryInvoker.drain() and SpringCloudQueryBusConnector.drain(source) are synchronized and do blocking I/O (sink.response(...)) inside the monitor. On JDK 21 a virtual thread that blocks inside synchronized pins its carrier, so a slow SSE client could occupy carrier threads, which is what we're trying to avoid for the producer threads. Either use a ReentrantLock in those drains, or tell me why pinning is acceptable here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed it to use a ReentrantLock, but I think in this case it wouldn't matter that much. The nasty cases are when a thread is pinned, but it is waiting for another thread that needs a pinned resource.

@@ -368,21 +371,18 @@ private synchronized void drain() {
return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another LLM-pointer I am leaving as is:

Same point as on CallbackHandOff: drain() is synchronized and writes to the sink (blocking I/O) while running on a virtual thread, which pins its carrier on JDK 21.

* them draining the same source would take entries out from under each other.
*/
private void drain(Source source) {
synchronized (source) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same synchronized question as on the IncomingQueryInvoker applies here.

try {
task.run();
} catch (RuntimeException e) {
logger.error("A task handling a signal from a message stream failed.", e);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing of this class dictates it's used by a MessageStream, right? If you agree, can we adjust the logging such to not refer to "message stream" exactly?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, adjusted

* Returns as soon as the query has been handed to the local handler. Responses are written to the {@code sink} as
* the handler produces them, which for a handler that answers asynchronously is after this method returned.
* Returns as soon as the query has been handed to the local handler. The responses are written to the
* {@code sink} on another thread, as and when the handler's stream has them, so after this method returned.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this new sentence at all. Care to reword it? 🙏

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjusted :)

@hjohn
hjohn force-pushed the enhancement/connectors-signal-only-callbacks branch from 2c6afe0 to 63ac5c3 Compare November 17, 2026 16:49
The connectors read from a response stream inside its callback, which runs
on the thread of whoever produced the response. That thread is shared with
other streams, so the read can stall them, and Axon Framework is going to
reject it.

The callback now only schedules the drain, through the new CallbackHandOff,
which combines signals and runs the work on a virtual thread. The drain that
followed registering the callback is gone, as a stream calls back at once when
it already has something. This also removes a lock-order inversion between the
connector's lock and the stream's lock. Tests that expected the drain to run
synchronously now wait for it.
@hjohn
hjohn force-pushed the enhancement/connectors-signal-only-callbacks branch from 63ac5c3 to 80c854a Compare November 17, 2026 16:53
@hjohn

hjohn commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Frankly a bit in doubt whether to request changes yes or no. The added util makes sense to me, although I can imagine it's better situated in Axon Framework.

I didn't want to make another cross dependency for merge ordering... frankly I was considering not even making a utility at all. It is marked @Internal regardless so it can be moved easily later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Priority 2: Should High priority. Ideally, these issues are part of the release they’re assigned to. Type: Enhancement Use to signal an issue enhances an already existing feature of the project.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants