Skip to content

Reject reads from a stream's callback while it handles a progress signal - #5251

Open
hjohn wants to merge 1 commit into
mainfrom
enhancement/reject-reads-from-signal-callbacks
Open

hjohn wants to merge 1 commit into
mainfrom
enhancement/reject-reads-from-signal-callbacks

Conversation

@hjohn

@hjohn hjohn commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Merge AxonIQ/axoniq-framework#747 first. The Axoniq connectors read from a stream inside its callback and break with this change; that PR fixes them and works with and without this one.

  • AbstractMessageStream marks a thread while it handles a progress signal, and throws an IllegalStateException when next(), peek() or hasNextAvailable() is called on any such stream on that thread. The exception names the read, the stream, and the stream and callback being run. The setCallback documentation now says a callback only signals, and must be fast.
  • With debug logging enabled for AbstractMessageStream, a callback that takes 100 ms or more to handle a signal is logged.
  • MessageStreamUtils.reduce and FluxUtils.of read entries that arrive later on an Executor or Scheduler (virtual threads and boundedElastic by default), and have overloads that take one.
  • MessageStream.reduce(identity, accumulator, Executor) is new. With the DirectExecutor, all entries are read on the calling thread, which blocks while the stream has nothing available (30 seconds without progress, or an interrupt, fails the result and closes the stream). It is rejected on a thread that is handling a progress signal.
  • MergedMessageStream passes the signals of the streams it merges on without reading from them.
  • Tests that read inside a callback, or expected synchronous delivery, now read on their own thread or wait.

@hjohn hjohn self-assigned this Nov 9, 2026
@hjohn hjohn added Priority 2: Should High priority. Ideally, these issues are part of the release they’re assigned to. Type: Enhancement Use to signal an issue enhances an already existing feature of the project. labels Nov 9, 2026
@smcvb smcvb added this to the Release 5.5.0 milestone Nov 9, 2026
@hjohn
hjohn force-pushed the enhancement/reject-reads-from-signal-callbacks branch 2 times, most recently from 6996e65 to a6d098f Compare November 9, 2026 13:32
@hjohn
hjohn force-pushed the enhancement/reject-reads-from-signal-callbacks branch from a6d098f to 6bc5bc1 Compare November 16, 2026 14:09
@hjohn
hjohn marked this pull request as ready for review November 17, 2026 05:27
@hjohn
hjohn requested a review from a team as a code owner November 17, 2026 05:27
@hjohn
hjohn requested review from MateuszNaKodach, schananas and smcvb and removed request for a team November 17, 2026 05:27

@smcvb smcvb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am pretty sure I am aligned with the changes in the AbstractMessageStream. But I am not yet with the changes on the MessageStreamUtils.

BiFunction<@Nullable R, ? super MessageStream.Entry<M>, @Nullable R> accumulator) {
var reducer = new Reducer<>(source, identity, accumulator);
source.setCallback(reducer::process);
return reduce(source, identity, accumulator, Thread::startVirtualThread);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't the default without adjusting behavior be to replace this of the DirectExecutor, or something akin?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current behavior of main is to reduce as much as possible on the caller thread, and if the stream blocks, wait for the callback. It then continues the reduce on the callback thread, which is what is illegal now. If you make the direct executor the default, that would make this call always blocking, which I think would be a behavioral change.

So by default, two (or more threads depending on where the callback came from) could already be involved:

Doing a reduce against an Axon Server DCB stream will usually not block as there is no data available, and so you get an incomplete future (the way it was intended). When data comes in, it would have continued on the callback thread (so not the caller thread) but now it hops one thread further to actually read that data instead of doing it on the callback thread. Making the direct executor the default would mean it always blocks even for Axon Server.

Doing a reduce against a Postgres/JPA or Axon Server aggregate stream blocks the caller until the database call has completed (IO) -- if for whatever reason it needs to do another call, you may get a callback and it would also thread hop, but none of those currently do this (they are all iterator based streams that fetch more data immediately when hasNextAvailable is called), so it basically blocks until the entire sourcing completes and returns you a completed future.

/**
* The time a blocking reduction waits for its source to report progress, before it gives up.
*/
private static final Duration IDLE_TIMEOUT = Duration.ofSeconds(30);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I already anticipate the requests to adjust this hard timeout which I'd hope isn't necessarily something we provide at all on this level. See my other comment further down in this class.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can remove it or make it adjustable -- it's not a maximum on the reduction time though, it is a maximum time we'll wait for more data to arrive. For sourcings it would be hard to hit this time out, but not impossible of course (a 30 second database or GC hiccup could trigger it, but a reduction that is so large it just takes 10 minutes to process wouldn't hit it).

When reducing a stream (which are unbounded) you may get lucky and the time out would abort it eventually -- otherwise it would keep running forever (this is a different problem).

second.setCallback(delegating);
delegating.run();

/*

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super duper nit: it feels a bit out of style for AF to do /*...*/ styled comments inside a method block.

Objects.requireNonNull(accumulator, "The accumulator parameter must not be null.");
Objects.requireNonNull(executor, "The executor parameter must not be null.");

if (executor instanceof DirectExecutor) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not simply invoke the DirectExecutor? It already ensure it's done right away instead of async, right?

@hjohn hjohn Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the problem I'm trying to solve here is still that I don't want to run things on the callback thread. So technically, the DirectExecutor is illegal. If we were to use it in the same way as say the virtual thread executor, we would get the exception from AbstractMessageStream.

So what I've done here (and perhaps we shouldn't do it in that way) is that if the executor provided is the direct one, that we run everything on the thread that called reduce (which is not the same thing as the callback thread). So you get this:

  • reduce with virtual thread: reduce caller thread doesn't block and gets an incomplete future -- any reads after a callback run on the virtual thread provided
  • reduce with direct executor: caller thread is "repurposed" as the reading thread; in order to do that we make the call blocking and unblock it when a callback comes in; all reads are done on the thread that calls reduce, and you get a completed future after it returns -- the direct executor isn't actually used, it is just a marker to get this behavior (but it does match what the UoW uses, which can be important for how we do Expose the executor to read message streams on from a ProcessingContext #5301 )

These aren't equivalent. If we just used the direct executor the way we use any other executor, it would normally disallow it as you'd be directly using the callback thread.

I see two ways to resolve this:

  1. Make a different reduce call (blockingReduce or reduce with a blocking flag). A blockingReduce wouldn't have to return a future which favors making it a different method.
  2. Make a new kind of executor that we can distinguish and give it the meaning discussed above (blocks the reduce calling thread and does all reads on that thread) -- this ThreadBoundExecutor or whatever you want to call it would never be used, it would just be a marker.

I'm open to suggestions here on how to solve this :)

MessageStream<M> source,
@Nullable R identity,
BiFunction<@Nullable R, ? super MessageStream.Entry<M>, @Nullable R> accumulator,
Duration idleTimeout

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not so sure about the timeout behavior. Is that really a choice the MessageStream should make? What if somebody truly intends to return a MessageStream result that takes 2 minutes purely because of the big reduction it's doing? Or, are you deflecting that with the response "then don't use the DirectExecutor"? If so, why not clarify on the docs that it's dangerous to call long running operations without a executor that can be timed-out/shutdown?

* read from a specific thread, such as one backed by a thread-bound transaction, by passing an executor that runs
* its tasks on that thread.
* <p>
* When the given {@code executor} is the {@link org.axonframework.common.DirectExecutor}, all entries are read on

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not so sure about referring to the DirectExecutor specifically here. It's more about people spoofing the Executor to be something that runs ASAP, right? That's not DirectExecutor at all; users can derive that themselves. As pointed out on the MessageStreamUtils, I am not so sure about doing it this way.

Wouldn't simply warning the users to use an actual executor for potentially long running (reduce) operations is wise?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is actually more tied to how I wanted to resolve the "next" problem in #5301 -- reduce is not really called that often by users, but it is called by our own code, specifically sourcing will use reduce to convert a stream of events to an entity. That runs in a UoW, and I want to adjust those reduce calls to pass in an executor supplied by the UoW. I then used the direct executor to distinguish whether we are basically running in Spring/JDBC world (forced same thread, uses direct executor) and Axon Server world (UoW uses a non-direct executor).

Another option is to expose the forced same thread execution flag instead of an executor, and have all code that uses reduce from within a UoW check that flag and call a different blocking reduce method (see other comment).

private static final ThreadLocal<AbstractMessageStream<?>> HANDLING_SIGNAL = new ThreadLocal<>();

/**
* The time a callback may take to handle a progress signal before it is logged, when debug logging is enabled.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: actually mentioning the value in ms or seconds would be nice.

*
* @return {@code true} if the current thread is running the callback of a stream because of a progress signal
*/
static boolean isHandlingSignal() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who is using this method?

A callback runs on the thread of whoever signalled, which is usually shared
with other streams, so reading from a stream there can stall all of them.
While a thread handles a progress signal, AbstractMessageStream now throws an
IllegalStateException when next(), peek() or hasNextAvailable() is called on
any stream on that thread. The exception names the read, the stream, and the
stream and callback being run.

reduce and FluxUtils.of used to read in the callback. They now hand the read
to an Executor or Scheduler (virtual threads and boundedElastic by default).
MessageStream.reduce has an overload that takes the Executor. With the
DirectExecutor it reads everything on the calling thread, blocking while the
stream has nothing available, for streams that must be read from that thread.
MergedMessageStream passes the signals of the streams it merges on without
reading from them, and the stream test suite reads on a consumer thread where
it used to read in the callback.

With debug logging enabled for AbstractMessageStream, a callback that takes
100 ms or more to handle a signal is logged, naming the callback and the
stream.
@hjohn
hjohn force-pushed the enhancement/reject-reads-from-signal-callbacks branch from 6bc5bc1 to 78e8acf Compare November 17, 2026 16:54

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Priority 2: Should High priority. Ideally, these issues are part of the release they’re assigned to. Type: Enhancement Use to signal an issue enhances an already existing feature of the project.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants