Add fallback from shard awareness in case of overloaded coordinator shards

[psarna]

This issue is about implementing a mechanism for falling back to picking least busy connections, even if we know the owner shard, in case we detect that a particular coordinator shard is severely overloaded. The idea is already implemented and explained here: scylladb/gocql#86

The whole discussion is worth reading, but specifically, in comment scylladb/gocql#86 (comment) @vladzcloudius requests implementing a similar behavior in the Java driver, so that he can conveniently test it with cassandra-stress. Hence, this issue :)

/cc @avelanarius does your team have spare cycles for implementing such a change, potentially based on existing PR for gocql linked above?

[vladzcloudius]

@DoronArazii Please, assign somebody to work on this.

[DoronArazii]

Thanks for raising this up @vladzcloudius

@mykaul please consult with @avelanarius regard this enhancement.

[mykaul]

Thanks for raising this up @vladzcloudius

@mykaul please consult with @avelanarius regard this enhancement.

Sure. I'll let @avelanarius asses feasibility and priority.

[tarzanek]

based on how many people use shard aware java driver (and to how many I told to not use it in case of shard overload) ... guys ... this is a P1 imho
@mykaul @avelanarius it's low hanging fruit that will save us from lots of issues, let's get it fixed

[avelanarius]

What would be the test scenario? A single overloaded shard (for example due to compaction)? Or someone else would do the actual benchmarking after we implemented it?

[tarzanek]

@avelanarius yes, the compaction test would be good
ideal test would be more clients fighting over one shard ... this way their requests would be routed away if they hit the limit of it

[tarzanek]

ref scylladb/scylladb#8996

[vladzcloudius]

@avelanarius yes, the compaction test would be good ideal test would be more clients fighting over one shard ... this way their requests would be routed away if they hit the limit of it

Simply overload a single shard with some CPU hogging work (some tight loop C application) and run a cassandra-stress read test with RF=3.

Without a patch a corresponding amount of requests will have a higher latency due to queuing on that shard - foreground reads would go as high as you would allow: if you set threads=N then on that shard it will go as high as N.

With the fix the driver is expected to start steering reads to other shards after some threshold (AFAIR half of the max_requests_per_connection which is controlled by maxPending argument of cassandra-stress AFAIR).

[avelanarius]

An update on the issue: @Gor027 will start with reproducing the issue (in a clean way!) - a very similar scenario @vladzcloudius described - starting a X shard node, overloading one shard and testing sending queries that touch data in the overloaded shard. Next, test how well it performs if we always send the queries to the correct (overloaded) shard, always send data to the wrong (non-overloaded) shard. I think it'll be useful to do a "semi-deep" dive analyzing what actually happens in Scylla.