How to proxy HTTP/2 cleartext (h2c) request?

[banool]

Some prior context, but I'll describe the situation fully here: #3956.

I'm building a proxy using Hyper. We run it in GCP Cloud Run.

I'm building my client like this, so we only support HTTP/2 requests:

pub fn build_http_client() -> HyperClient {
    Client::builder(TokioExecutor::new())
        .http2_only(true)
        .build::<_, axum::body::Body>(hyper_tls::HttpsConnector::new())

I run a HTTP/2-only server like this:

nghttpd -v 8354 --no-tls -d ~/Downloads

$ curl -v --http2-prior-knowledge http://localhost:8354/blah.pdf
> GET /blah.pdf HTTP/2
* Request completely sent off
< HTTP/2 200

I then configure my proxy like this:

upstreams:
  - path: endpoint1
    url: http://127.0.0.1:8354
  - path: endpoint2
    url: https://myapi.com

We can confirm that https://myapi.com supports HTTP/2 requests:

$ curl -v https://myapi.com/v1
* IPv6: (none)
* IPv4: 34.111.219.107
*   Trying 34.111.219.107:443...
* Connected to myapi.com (34.111.219.107) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://api.testnet.aptoslabs.com/v1
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: api.testnet.aptoslabs.com]
* [HTTP/2] [1] [:path: /v1]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /v1 HTTP/2
>
* Request completely sent off
< HTTP/2 200
happy response

Now I run my proxy.

I send a request to my proxy to hit endpoint1 locally like this:

$ curl -v --http2-prior-knowledge http://localhost:9898/endpoint1/blah
> GET /blah HTTP/2
> Host: localhost:9898
* Request completely sent off
< HTTP/2 200

Perfect, this is working great. Now let's hit endpoint2:

$ curl -v --http2-prior-knowledge http://localhost:9898/endpoint2/blah
> GET /v1 HTTP/2
* Request completely sent off
< HTTP/2 500
client error (SendRequest)

Ruh roh. This is what we see from the proxy (abridged, full logs here):

{"target":"hyper_util::client::legacy::connect::http","message":"connecting to 34.111.219.107:443"}
{"target":"hyper_util::client::legacy::connect::http","message":"connected to 34.111.219.107:443"}
{"target":"h2::proto::connection","message":"Connection::poll; connection error","error":"GoAway(b\"\", FRAME_SIZE_ERROR, Library)"}
{"target":"h2::codec::framed_write","message":"send","frame":"GoAway { error_code: FRAME_SIZE_ERROR, last_stream_id: StreamId(0) }"}
{"target":"h2::proto::connection","message":"Connection::poll; connection error","error":"GoAway(b\"\", FRAME_SIZE_ERROR, Library)"}
{"target":"hyper_util::client::legacy::client","message":"client connection error: http2 error"}
{"target":"tower_http::trace::on_failure","message":"response failed","classification":"Status code: 500 Internal Server Error","latency":"132 ms"}

My suspicion is that the upstream can handle HTTP/2, but it can't handle HTTP/2 cleartext. I would've maybe naively hoped that when I send the request off with Hyper, it magically wraps it back up into a normal HTTP/2 request. Is there a way to get this to work, assuming my diagnosis of the issue is correct?

Another possible issue is the upstream only supports HTTP/2 after first upgrading to it from HTTP/1.1. I was hoping enabling alpn on hyper-tls would help me with that though.

In short if I receive a HTTP/2 request (h2c or not) I want to be able to proxy it to a https upstream that supports HTTP/2 (though perhaps only via an upgrade, I'm not exactly sure).

My relevant dependencies:

axum = { version = "0.7.4", features = ["http2", "ws"] } 
http = "1.3.1"
http-body = "1.0.1"
http-body-util = "0.1.3"
hyper = { version = "1.7.0" }
hyper-tls = { version = "0.6.0", features = ["alpn"] }
hyper-util = "0.1.17"

[banool]

Note that if I don't set http2_only when building the client I instead get this error in the proxy when the client tries to send the HTTP/2 request upstream:

Connection is HTTP/1, but request requires HTTP/2

[banool]

Okay I see this comment for http2_only in the code:

/// The destination must either allow HTTP2 Prior Knowledge, or the
/// `Connect` should be configured to do use ALPN to upgrade to `h2`
/// as part of the connection process. This will not make the `Client`
/// utilize ALPN by itself.

So the question is just how to make this happen. Like I mentioned earlier, I thought https://docs.rs/hyper-tls/latest/hyper_tls with the alpn feature enabled would handle this.

[banool]

With rustls, you might need to configure the negotiation protocols (alpn) when creating the connector.

I can't see how to do this with hyper-rustls. It says I should leave the alpn stuff undefined if providing my own config and leave it to the library to figure it out based on enable_http<version>. hyper-tls with the alpn features should do that automagically right?

The http2_only setting in hyper means it will use http/2 with the endpoint no matter what, often called prior knowledge. h2c is supported, hyper doesn't care if the transport is encrypted.

In my case I'm receiving a h2c request but I want to send it to a HTTP/2 (but not h2c) capable upstream with https. So I need it to be put back in encrypted format / potentially do the HTTP/1.1 -> 2 upgrade dance. I can't figure out how to get this to work. Downgrading the HTTP/2 request to HTTP/1.1 doesn't seem to help either:

{"timestamp":"2025-09-22T12:57:44.416445Z","level":"DEBUG","target":"h2::codec::framed_write","message":"send","frame":"Headers { stream_id: StreamId(3), flags: (0x5: END_HEADERS | END_STREAM) }"}
{"timestamp":"2025-09-22T12:57:44.437783Z","level":"DEBUG","target":"h2::codec::framed_read","message":"received","frame":"Reset { stream_id: StreamId(3), error_code: PROTOCOL_ERROR }"}
{"timestamp":"2025-09-22T12:57:44.439352Z","level":"ERROR","target":"tower_http::trace::on_failure","message":"response failed","classification":"Status code: 500 Internal Server Error","latency":"25 ms"}

I understand hyper-rustls is maintained by a different team, though maybe you have some tips on how to get this to work with hyper-tls? (Appreciate all the help btw!)

[seanmonstar]

No, you need to add it yourself, I think. Here's where I've done so: https://github.com/seanmonstar/reqwest/blob/master/src/async_impl/client.rs#L495.

hyper at its base doesn't care what the transport layer does. The legacy client tries to help detect what protocol is used.

I'll reiterate from higher up:

• If your endpoint must use ALPN to allow h2, you need to configure you TLS implementation to send that.
• If it works with h2 without any negotiation, you can tell hyper to use http2_only and skip any ALPN.

If your endpoint expects an HTTP/1.1 Upgrade to h2, hyper does not support that. It's a dying feature, all the major browsers got rid of the feature. However, most people don't actually ever need this, it just ends up being a conflation of vocabulary that sends people down the wrong path.

[banool]

You need to add it yourself, I think

Interestingly hyper-rustls seems to do the same thing you do in that code snippet: https://github.com/rustls/hyper-rustls/blob/5e9b69b4128abfc147e8b6fa55977eac4c3f575c/src/connector/builder.rs#L275.

If your endpoint expects an HTTP/1.1 Upgrade to h2, hyper does not support that.

It helps a lot to know this. Exactly like you say, I'm getting confused with Upgrade in the HTTP/1.x context and ALPN. My proxy in the normal case receives HTTP/1.1 requests and proxies them upstream, but it must be doing the ALPN procedure (HTTP/1.1 request over https, switch to HTTP/2 if supported as part of ALPN).

Now I'm adding support for receiving HTTP/1.x and HTTP/2 cleartext (h2c) requests (see #3956). I suppose I should do this:

• Receive requests as HTTP/1.x or h2c (courtesy of Cloud Run HTTP/2 mode): https://cloud.google.com/run/docs/configuring/http2).
• If h2c, downgrade the request to HTTP/1.1 using *request.version_mut() = http::Version::HTTP_11
  • If I don't do this I get GoAway { error_code: FRAME_SIZE_ERROR }.
  • If the upstream supports h2 without negotiation I don't need to do this, like you say.
• Send request to upstream as HTTP/1.1. ALPN magic happens if upstream supports HTTP/2, it gets upgraded to HTTP/2.

This is what I was missing, that I need to set the version back to HTTP/1.1 so that ALPN can happen properly and get negotiated back to HTTP/2. I suppose if I send a request as HTTP/2 it is assumed that the upstream can do HTTP/2 without ALPN negotiation (which it seems like is not generally supported out of the box).

I think this is a little odd, I would've expected that it doesn't matter whether the request is HTTP/1.1 or HTTP/2, we'd do ALPN no matter what if it's an https request. Only if I set something like no_alpn would we skip ALPN. But http2_only doesn't behave that way:

/// The destination must either allow HTTP2 Prior Knowledge, or the
/// `Connect` should be configured to do use ALPN to upgrade to `h2`
/// as part of the connection process. This will not make the `Client`
/// utilize ALPN by itself.

All that said, I think I was running into some red herrings because of #3958. After fixing that and then switching to hyper-rustls with this config everything seems to be working:

pub fn get_hyper_client_builder() -> Builder {
    Client::builder(TokioExecutor::new())
}

pub type HyperClient = Client<hyper_rustls::HttpsConnector<HttpConnector>, axum::body::Body>;

pub fn build_http_client() -> HyperClient {
    let connector = hyper_rustls::HttpsConnectorBuilder::new()
        .with_native_roots()
        .expect("Failed to initialize HTTPS connector with native roots")
        .https_or_http()
        .enable_all_versions()
        .build();
    get_hyper_client_builder().build::<_, axum::body::Body>(connector)
}

I still need to downgrade HTTP/2 requests to HTTP/1.1 if I want ALPN to happen, which you pretty much always want to happen.

Also important to note for future readers is the --http2-prior-knowledge curl flag doesn't mean you skip ALPN if you use https: https://curl.se/docs/manpage.html#--http2-prior-knowledge.

[banool]

Okay I was getting a million things mixed up but I think I've got it all sorted finally.

With the below config I can receive HTTP/2 requests over http:// requests, but I'm unable to send HTTP/2 requests to an upstream over http://. With the client described above, I get this error in the logs:

Connection is HTTP/1, but request requires HTTP/2

However if I use this client it works:

pub fn build_h2c_client() -> H2cClient {
    Client::builder(TokioExecutor::new())
        .http2_only(true)
        .build::<_, axum::body::Body>(hyper_tls::HttpsConnector::new())
}

So in summary, here's what you do.

Build two clients:

pub fn get_hyper_client_builder() -> Builder {
    Client::builder(TokioExecutor::new())
}

/// Use this for all requests except HTTP/2 over http://
pub type HyperClient = Client<hyper_rustls::HttpsConnector<HttpConnector>, axum::body::Body>;

pub fn build_http_client() -> HyperClient {
    let connector = hyper_rustls::HttpsConnectorBuilder::new()
        .with_native_roots()
        .expect("Failed to initialize HTTPS connector with native roots")
        .https_or_http()
        .enable_all_versions()
        .build();
    get_hyper_client_builder().build::<_, axum::body::Body>(connector)
}

/// Use this only for HTTP/2 over http:// (h2c) requests.
pub type H2cClient = Client<hyper_tls::HttpsConnector<HttpConnector>, axum::body::Body>;

pub fn build_h2c_client() -> H2cClient {
    Client::builder(TokioExecutor::new())
        .http2_only(true)
        .build::<_, axum::body::Body>(hyper_tls::HttpsConnector::new())
}

Then for each upstream mark whether we know it can handle HTTP/2 requests. This is optional for upstreams that we connect to via https:// because it'll do ALPN. But for http:// upstreams that we want to use HTTP/2 with you must set it to true. Then you need code like this:

if request.version() == http::Version::HTTP_2 && !state.get_can_handle_http_two_requests() {
    *request.version_mut() = http::Version::HTTP_11;
}

Plus this:

let response = if request.version() == http::Version::HTTP_2 && upstream.url().scheme() == "http" {
    state.h2c_client.request(request).await
} else {
    state.http_client.request(request).await
};

Note that it won't be possible to proxy HTTP/1.x requests to a http:// upstream that can only handle HTTP/2 requests (very rare case), we don't upgrade HTTP/1.x to HTTP/2.

tl;dr: It is necessary to use hyper-rustls instead of hyper-tls for the first client because it will do ALPN even if the request is HTTP/2, whereas with hyper-tls it makes the assumption that you only need ALPN if the request is HTTP/1.1. I think this is a bit of a footgun; if you're connecting over https:// you might as well do ALPN unless the user specifically opts out of it with some kind of no_alpn flag on the client / request.