Repository navigation
Conversation
6996e65 to
a6d098f
Compare
a6d098f to
6bc5bc1
Compare
smcvb
left a comment
There was a problem hiding this comment.
I am pretty sure I am aligned with the changes in the AbstractMessageStream. But I am not yet with the changes on the MessageStreamUtils.
| BiFunction<@Nullable R, ? super MessageStream.Entry<M>, @Nullable R> accumulator) { | ||
| var reducer = new Reducer<>(source, identity, accumulator); | ||
| source.setCallback(reducer::process); | ||
| return reduce(source, identity, accumulator, Thread::startVirtualThread); |
There was a problem hiding this comment.
Wouldn't the default without adjusting behavior be to replace this of the DirectExecutor, or something akin?
There was a problem hiding this comment.
The current behavior of main is to reduce as much as possible on the caller thread, and if the stream blocks, wait for the callback. It then continues the reduce on the callback thread, which is what is illegal now. If you make the direct executor the default, that would make this call always blocking, which I think would be a behavioral change.
So by default, two (or more threads depending on where the callback came from) could already be involved:
Doing a reduce against an Axon Server DCB stream will usually not block as there is no data available, and so you get an incomplete future (the way it was intended). When data comes in, it would have continued on the callback thread (so not the caller thread) but now it hops one thread further to actually read that data instead of doing it on the callback thread. Making the direct executor the default would mean it always blocks even for Axon Server.
Doing a reduce against a Postgres/JPA or Axon Server aggregate stream blocks the caller until the database call has completed (IO) -- if for whatever reason it needs to do another call, you may get a callback and it would also thread hop, but none of those currently do this (they are all iterator based streams that fetch more data immediately when hasNextAvailable is called), so it basically blocks until the entire sourcing completes and returns you a completed future.
| /** | ||
| * The time a blocking reduction waits for its source to report progress, before it gives up. | ||
| */ | ||
| private static final Duration IDLE_TIMEOUT = Duration.ofSeconds(30); |
There was a problem hiding this comment.
I already anticipate the requests to adjust this hard timeout which I'd hope isn't necessarily something we provide at all on this level. See my other comment further down in this class.
There was a problem hiding this comment.
I can remove it or make it adjustable -- it's not a maximum on the reduction time though, it is a maximum time we'll wait for more data to arrive. For sourcings it would be hard to hit this time out, but not impossible of course (a 30 second database or GC hiccup could trigger it, but a reduction that is so large it just takes 10 minutes to process wouldn't hit it).
When reducing a stream (which are unbounded) you may get lucky and the time out would abort it eventually -- otherwise it would keep running forever (this is a different problem).
| second.setCallback(delegating); | ||
| delegating.run(); | ||
|
|
||
| /* |
There was a problem hiding this comment.
Super duper nit: it feels a bit out of style for AF to do /*...*/ styled comments inside a method block.
| Objects.requireNonNull(accumulator, "The accumulator parameter must not be null."); | ||
| Objects.requireNonNull(executor, "The executor parameter must not be null."); | ||
|
|
||
| if (executor instanceof DirectExecutor) { |
There was a problem hiding this comment.
Why not simply invoke the DirectExecutor? It already ensure it's done right away instead of async, right?
There was a problem hiding this comment.
So the problem I'm trying to solve here is still that I don't want to run things on the callback thread. So technically, the DirectExecutor is illegal. If we were to use it in the same way as say the virtual thread executor, we would get the exception from AbstractMessageStream.
So what I've done here (and perhaps we shouldn't do it in that way) is that if the executor provided is the direct one, that we run everything on the thread that called reduce (which is not the same thing as the callback thread). So you get this:
- reduce with virtual thread: reduce caller thread doesn't block and gets an incomplete future -- any reads after a callback run on the virtual thread provided
- reduce with direct executor: caller thread is "repurposed" as the reading thread; in order to do that we make the call blocking and unblock it when a callback comes in; all reads are done on the thread that calls
reduce, and you get a completed future after it returns -- the direct executor isn't actually used, it is just a marker to get this behavior (but it does match what the UoW uses, which can be important for how we do Expose the executor to read message streams on from a ProcessingContext #5301 )
These aren't equivalent. If we just used the direct executor the way we use any other executor, it would normally disallow it as you'd be directly using the callback thread.
I see two ways to resolve this:
- Make a different
reducecall (blockingReduceorreducewith a blocking flag). AblockingReducewouldn't have to return a future which favors making it a different method. - Make a new kind of executor that we can distinguish and give it the meaning discussed above (blocks the reduce calling thread and does all reads on that thread) -- this
ThreadBoundExecutoror whatever you want to call it would never be used, it would just be a marker.
I'm open to suggestions here on how to solve this :)
| MessageStream<M> source, | ||
| @Nullable R identity, | ||
| BiFunction<@Nullable R, ? super MessageStream.Entry<M>, @Nullable R> accumulator, | ||
| Duration idleTimeout |
There was a problem hiding this comment.
Not so sure about the timeout behavior. Is that really a choice the MessageStream should make? What if somebody truly intends to return a MessageStream result that takes 2 minutes purely because of the big reduction it's doing? Or, are you deflecting that with the response "then don't use the DirectExecutor"? If so, why not clarify on the docs that it's dangerous to call long running operations without a executor that can be timed-out/shutdown?
| * read from a specific thread, such as one backed by a thread-bound transaction, by passing an executor that runs | ||
| * its tasks on that thread. | ||
| * <p> | ||
| * When the given {@code executor} is the {@link org.axonframework.common.DirectExecutor}, all entries are read on |
There was a problem hiding this comment.
Not so sure about referring to the DirectExecutor specifically here. It's more about people spoofing the Executor to be something that runs ASAP, right? That's not DirectExecutor at all; users can derive that themselves. As pointed out on the MessageStreamUtils, I am not so sure about doing it this way.
Wouldn't simply warning the users to use an actual executor for potentially long running (reduce) operations is wise?
There was a problem hiding this comment.
It is actually more tied to how I wanted to resolve the "next" problem in #5301 -- reduce is not really called that often by users, but it is called by our own code, specifically sourcing will use reduce to convert a stream of events to an entity. That runs in a UoW, and I want to adjust those reduce calls to pass in an executor supplied by the UoW. I then used the direct executor to distinguish whether we are basically running in Spring/JDBC world (forced same thread, uses direct executor) and Axon Server world (UoW uses a non-direct executor).
Another option is to expose the forced same thread execution flag instead of an executor, and have all code that uses reduce from within a UoW check that flag and call a different blocking reduce method (see other comment).
| private static final ThreadLocal<AbstractMessageStream<?>> HANDLING_SIGNAL = new ThreadLocal<>(); | ||
|
|
||
| /** | ||
| * The time a callback may take to handle a progress signal before it is logged, when debug logging is enabled. |
There was a problem hiding this comment.
Nit: actually mentioning the value in ms or seconds would be nice.
| * | ||
| * @return {@code true} if the current thread is running the callback of a stream because of a progress signal | ||
| */ | ||
| static boolean isHandlingSignal() { |
A callback runs on the thread of whoever signalled, which is usually shared with other streams, so reading from a stream there can stall all of them. While a thread handles a progress signal, AbstractMessageStream now throws an IllegalStateException when next(), peek() or hasNextAvailable() is called on any stream on that thread. The exception names the read, the stream, and the stream and callback being run. reduce and FluxUtils.of used to read in the callback. They now hand the read to an Executor or Scheduler (virtual threads and boundedElastic by default). MessageStream.reduce has an overload that takes the Executor. With the DirectExecutor it reads everything on the calling thread, blocking while the stream has nothing available, for streams that must be read from that thread. MergedMessageStream passes the signals of the streams it merges on without reading from them, and the stream test suite reads on a consumer thread where it used to read in the callback. With debug logging enabled for AbstractMessageStream, a callback that takes 100 ms or more to handle a signal is logged, naming the callback and the stream.
6bc5bc1 to
78e8acf
Compare
Merge AxonIQ/axoniq-framework#747 first. The Axoniq connectors read from a stream inside its callback and break with this change; that PR fixes them and works with and without this one.
AbstractMessageStreammarks a thread while it handles a progress signal, and throws anIllegalStateExceptionwhennext(),peek()orhasNextAvailable()is called on any such stream on that thread. The exception names the read, the stream, and the stream and callback being run. ThesetCallbackdocumentation now says a callback only signals, and must be fast.AbstractMessageStream, a callback that takes 100 ms or more to handle a signal is logged.MessageStreamUtils.reduceandFluxUtils.ofread entries that arrive later on anExecutororScheduler(virtual threads andboundedElasticby default), and have overloads that take one.MessageStream.reduce(identity, accumulator, Executor)is new. With theDirectExecutor, all entries are read on the calling thread, which blocks while the stream has nothing available (30 seconds without progress, or an interrupt, fails the result and closes the stream). It is rejected on a thread that is handling a progress signal.MergedMessageStreampasses the signals of the streams it merges on without reading from them.