.Net MEVD: Decide and align error behavior when upserts fail

[roji]

Currently, in the Weaviate provider, if any errors occur when upserting records, those upserts silently fail and no exception is thrown (link). Since non-batch Upsert delegates to UpsertBatch, the same behavior applies for regular Upsert - errors are silently swallowed.

We should decide on a proper error handling behavior and apply it across all implementations. Specifically, in some databases, some upserts may succeed while others fail; in others, batches should be atomic and an error should cause the entire batch to fail. We should in any case throw an exception if any upsert in the batch failed, to make sure the user is aware of the failure.

[dmytrostruk]

We should in any case throw an exception if any upsert in the batch failed, to make sure the user is aware of the failure

This is tricky, because even if we throw an exception if at least one record upsert failed, the others will be upserted successfully. User should know which record IDs were stored successfully in order to either delete them and try to re-upsert the entire batch again or re-upsert only failed ones. We can pass this info to the user through some exception property.

But it's already possible to identify such a case today even without an exception. User knows the number of records to upsert, and user gets a collection of upserted IDs in response. By checking records.Count == ids.Count you can understand if all records where upserted and it's possible to identify which failed, although without additional error details.

On one side, the exception will provide user awareness and could hold more details about an error for each record, but on the other side, it will be always required to wrap upsert method in try/catch block to handle it properly. Taking into account that UpsertBatchAsync returns IAsyncEnumerable, it becomes harder to use that method, since you can't yield in try/catch block and so on.

It's probably worth to evaluate this API and its cases from user perspective to see which solution will be more user-friendly.