Add toxiproxy resiliency testing
#39
Conversation
mrattle
force-pushed
the
mrattle/toxiproxy_resiliency_testing
branch
from
3a764af
to
d3d0c0a
Compare
mrattle
changed the title
toxiproxy resiliency testing
mrattle
force-pushed
the
mrattle/toxiproxy_resiliency_testing
branch
5 times, most recently
from
245dd54
to
e03c466
Compare
mrattle
force-pushed
the
mrattle/toxiproxy_resiliency_testing
branch
from
307682e
to
6d93b6f
Compare
tests/resiliency_tests.rs
Outdated
| let values = vec!["value1", "value2", "value3"]; | ||
| let kv: Vec<(&str, &str)> = keys.clone().into_iter().zip(values).collect(); | ||
|
|
||
| let mut clean_client = rt.block_on(async { |
There was a problem hiding this comment.
Why do you need both a clean client and a toxic client? The toxic client should work fine as long as you don't use down ?
There was a problem hiding this comment.
Using the toxic client for everything until you apply a toxic condition is fine, but in order to do the checks after using an actual toxic (i.e. a closure using with_down() or with_limit_data()) you need a clean client. The toxic client is still in a compromised state at that point and trying to use it for further actions leads to out-of-bounds errors like this:
thread 'tests::test_set_multi_errors_on_downstream_with_toxic_client_via_limit_data' panicked at /Users/mark/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bytes-1.7.1/src/bytes_mut.rs:394:9:
split_to out of bounds: 8 <= 1
|
|
||
| let (proxies, toxic_local_addr) = create_proxies_and_configs(); | ||
|
|
||
| let toxic_local_url = "tcp://".to_string() + &toxic_local_addr; |
There was a problem hiding this comment.
There is a lot of boiler plate code here, can we turn it into some helper methods?
There was a problem hiding this comment.
Good callout, I'll clean this up 👍
There was a problem hiding this comment.
Did a refactoring pass on this. Reconfigured everything so that it's making proper use of environment variables, methods make more sense in the context that they're being use in (e.g. no more unnecessary vecs / proxies[0] since we're only ever making one proxy per test) tests should be more parsable and it should be easier to add more tests in the future.
mrattle
force-pushed
the
mrattle/toxiproxy_resiliency_testing
branch
from
83c1582
to
efbb364
Compare
TL;DR - What are you trying to accomplish?
This PR introduces
Toxiproxyresiliency testing to test potential errors between the client and server at the crate level.Details - How are you making this change? What are the effects of this change?
In these tests a
clean_clientrefers to a client that runs without any toxics applied to it, i.e. it performs as you would expect a normal client to perform. Atoxic_clientrefers to a client that is running through theToxiproxyservice to apply various toxics used to simulate different network issues that can be encountered in normal client/server communication.with_down()method is used to test the behaviour of the crate if the memcached server happens to be down / cannot be contacted.with_limit_data()method is used to test behaviour when the connection fromclient->server(i.e.upstream) is interrupted and an incomplete request / command string is sent to the server, and when the connection fromserver->client(i.e.downstream) is interrupted and an incomplete response is sent back from the server.