Unable to integrate tower retry with tonic because http::Request is not Clone

[vitarb]

Hello community, I'm looking for some guidance.

I'm trying to integrate retry layer from tower into my tonic setup in order to enable gRPC retries for some common retryable errors.

My initial setup is almost identical to this example with retry policy like this one.

The issue I'm facing is that in order to work correctly, one must implement tower's policy trait including clone_request function. If you don't implement it and return None (as documentation suggests in the case when Clone is not implemented for the request) then whole retry logic will not be executed. Now the problem is that under the hood tonic uses http::Request, which is NOT Clone.

One one hand I see reasoning behind why http::Request should not be clone, but on the other it makes whole retry mechanism unusable.

Is there any work-around for this issue or are there any improvements we can make in tonic or tower in order to eliminate it in the first place (maybe put request behind an Arc?)?

[davidpdrsn]

Making the requests themselves Clone is hard due to streaming.

There are tricks you can pull like buffering the http_body::Body as it's being generated and then "replay" it when retrying. That is what linkerd-proxy does. It does introduce quite a bit of complexity though and isn't something we can build into tonic.

It's instead recommended to handle retries at a higher level in your application than at the tower level. Exactly how to do that depends on your app.

[vitarb]

Thanks, I was hoping that we could stay consistent across the ecosystem and be able to put retry functionality into the middleware layer (which tower is), like for example in go which allows adding an interceptor that handles retry logic.
Can client generator be customizable so that non-streaming APIs can have this type of retry integration?

[davidpdrsn]

Can client generator be customizable so that non-streaming APIs can have this type of retry integration?

No I don't think so. http2 is always streaming whether you want to or not. In gRPC, unary requests are streams that happen to only contain one message.

[vitarb]

I agree, having to clone http request sounds like a big hustle.
Can you explain why is it a bad idea to store request behind an Arc, making it clonable without having to clone it?

[davidpdrsn]

Can you explain why is it a bad idea to store request behind an Arc, making it clonable without having to clone it?

Prost requires owned buffers for encoding. There has been some discussion about supporting Arcs but don't anyone has worked on it yet.

It's also hard to say how supporting Request and Arc<Request> would impact tonics design.

[vitarb]

Got it, should we keep this conversation open and try to find a potential long term solution, or would you prefer to shelve it?
Personally I think that aiming for feature parity with other languages, such as go, which has retry interceptors, should be important.

[davidpdrsn]

I would love for tonic to support tower::retry out of the box. I just don't know how to make it work 😞 But we should keep the issue open.

EDIT: I wasn't trying to dismiss the idea of retrys, I was just trying to state why we haven't supported it thus far.

For reference linkerd-proxy's implementation of a cloneable body type is here https://github.com/linkerd/linkerd2-proxy/blob/main/linkerd/http-retry/src/lib.rs. I don't know enough about the details to know if we could copy/paste that into tonic. Maybe @hawkw has more insight.

[LucioFranco]

Yeah, I would like this and I have some ideas on how to accomplish it. Maybe it will make it into the next release but not sure. Ill try to be optimistic :D

[fmassot]

I'm interested in the retry feature for our project as we heavily use tonic there (and thanks a ton for what you do :)).
@LucioFranco what did you have in mind exactly? We may give it a try so it could be nice to be aware of a good design.

[LucioFranco]

There is a tracking issue in tower here tower-rs/tower#682

[stan-irl]

im trying to write some auth middleware that listens for UNAUTHENTICATED and refreshes the token then retries the request. this is also not possible while a request cant be cloned which is unfortunate because its a pretty common usecase.

Is there a quick any dirty way to clone a request even if its not performant and doesnt support stream? I've been trying for hours to clone the body as bytes but I cant get anything going:

async fn clone_request(req: &Request<BoxBody>) -> Request<BoxBody> {
    let bytes = hyper::body::to_bytes(req.body_mut()).await.unwrap();
    let body = hyper::Body::from(bytes);
    let box_body = BoxBody::new(body);
    Request::builder()
        .method(req.method())
        .uri(req.uri())
        .version(req.version())
        .body(box_body)
        .unwrap()
}

throws error:

error[E0271]: type mismatch resolving `<Body as HttpBody>::Error == Status`
  --> src/components/grpc/src/middleware/auth.rs:88:33
   |
88 |     let box_body = BoxBody::new(body);
   |                    ------------ ^^^^ expected `Status`, found `Error`
   |                    |
   |                    required by a bound introduced by this call
   |
note: required by a bound in `http_body::combinators::box_body::UnsyncBoxBody::<D, E>::new`
  --> ~/.cargo/registry/src/github.com-1ecc6299db9ec823/http-body-0.4.5/src/combinators/box_body.rs:82:27
   |
82 |         B: Body<Data = D, Error = E> + Send + 'static,
   |                           ^^^^^^^^^ required by this bound in `UnsyncBoxBody::<D, E>::new`

[fraillt]

Since this issue was created 1 thing has changed. In http crate, version "1" request object become cloneable. So this fixes half of the problem.
However, I don't think that it's possible to make tonic request cloneable, because by design it uses http2 which is streaming protocol. So no one should expect a tonic request to be cloneable.
However, I think retry middleware in tower crate is a right place to solve this issue.
I have create PR to retry middleware in order to make tonic requests work with it.