reduce time of lock holding by flush #2009
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
apps/framework-cli/src/infrastructure/processes/kafka_clickhouse_sync.rs
Show resolved
Hide resolved
| @@ -120,6 +121,59 @@ impl<C: ClickHouseClientTrait + 'static> Inserter<C> { | |||
| Ok(()) | |||
| } | |||
|
|
|||
| async fn get_front_batch(&self) -> Option<Batch> { | |||
There was a problem hiding this comment.
I think this removes the at least once delivery no?
Can you explain why it doesn't?
The thought is that if another process claims the queue in parallel and that batch goes through then if the next batch gets committed then the intermediary batch won't get replayed.
There was a problem hiding this comment.
i think there's only one spawned task doing the flush
if that insert failed, the batch gets returned to the queue, and consumed by that same flushing process
There was a problem hiding this comment.
Feels like if we merge this we need to a test that shows we don't drop data since that's the current implementation just got
There was a problem hiding this comment.
Could be a manual test on your laptop where you send a bunch of things kill the process, restart and see records not get dropped
|
What should we do about this? |
|
I do think it's worth merging. I'll tweak the code to simulate failures to test. |
No description provided.