Ability to specify the path used by negotiationType(PLAINTEXT_UPGRADE)

[cowwoc]

Is your feature request related to a problem?

Yes. Docker uses BuildKit under the hood. In order to establish a gRPC connection with BuildKit, we are supposed to send an HTTP POST request to a /grpc REST endpoint with the following headers:

Connection: Upgrade
Upgrade: h2c

Here is the relevant code in the Docker codebase:

• https://github.com/docker/buildx/blob/f8de3c3bdcb956aa4b6671d36644c7af7a7989c5/driver/docker/driver.go#L60
• https://github.com/moby/moby/blob/3fc5bfd56569accf6129e5fcb4157349a1d0b79c/client/hijack.go#L43
• https://github.com/moby/moby/blob/3fc5bfd56569accf6129e5fcb4157349a1d0b79c/client/hijack.go#L55

Currently, NettyChannelBuilder.negotiationType(NegotiationType.PLAINTEXT_UPGRADE) is hard-coded to use a context path of /: https://github.com/grpc/grpc-java/blob/master/netty/src/main/java/io/grpc/netty/ProtocolNegotiators.java#L881

Describe the solution you'd like

Provide a way to specify which context path to apply PLAINTEXT_UPGRADE against.

Describe alternatives you've considered

Copy/paste the existing code into my own codebase, and manually change the hard-coded path.

[cowwoc]

@ejona86 Would you accept a PR that add a NettyChannelBuilder.forAddress(String host, int port, String path) overload or do you envision handling this differently?

[ejona86]

To my knowledge and based on that code, grpc-go doesn't support the h2c upgrade mechanism. It seems this is something different, as h2c requires the HTTP2-Settings header which we don't see here. I'm worried this will cause compatibility problems because gRPC will send SETTINGS and expect the server to respect them. But it seems such SETTINGS would be discarded instead. Also, FWIW, h2c upgrade is also considered obsolete and dead by RFC 9113.

I think because that is not adhering to RFC 7540 it requires you to manually do the same hack in Java and configure grpc-java to use normal InsecureChannelCredentials/PLAINTEXT. Easiest way would be to make a local in-process proxy and point grpc-netty at its TCP port. It is also possible to configure gRPC to use Netty's LocalChannel if the proxy is written to use Netty as well.

If we were to support alternate paths for the h2c upgrade, we'd add it to InsecureFromHttp1ChannelCredentials.

[cowwoc]

@ejona86 jumping on the proxy idea, do you mean that I could create a localhost "relay" HTTP/2 server so that any incoming request triggers the following behavior?

1. Establish an HTTP/1.1 connection with Docker at /grpc
2. Do the non-standard "upgrade" thing to change the protocol to h2c
3. Push any output by Docker's connection to the relay's client, and vice-versa

@thaJeztah @tonistiigi Is there any chance of you introducing something more standard-friendly than the current mechanism?

[ejona86]

Yeah, it'd be a localhost "relay," but actually more HTTP/1 or TCP, because at (3) it would just copy bytes back-and-forth. It wouldn't need to understand HTTP/2 at all. But I think we're on the same page.

[cowwoc]

@ejona86 I ran into a snag trying to implement the LocalChannel approach: I need to instruct ManagedChannel to avoid trying to establish a HTTP/2 connection; otherwise, I get:

Caused by: io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 504f535420
  at io.netty.codec.http2@4.1.119.Final/io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:107)

Currently, I am invoking:

NettyChannelBuilder.
  forAddress(grpcRelay.getAddress()).
  channelType(LocalChannel.class).
  eventLoopGroup(grpcRelay.getEventLoopGroup()).
  usePlaintext().
  build();

where grpcRelay.getAddress() is a LocalAddress.

How do I proceed?

[cowwoc]

Nevermind. I was sending a message to the client instead of the server. Ignore the last post :)

[cowwoc]

@ejona86 After wasting 2 weeks of my life trying to use of Netty, I think we don't need a relay after all...

curl -vv -X POST --http2 -i http://localhost:2375/grpc
Note: Using embedded CA bundle, for proxies (233263 bytes)
06:29:22.061000 [0-0] * Host localhost:2375 was resolved.
06:29:22.063000 [0-0] * IPv6: ::1
06:29:22.065000 [0-0] * IPv4: 127.0.0.1
06:29:22.066000 [0-0] * [SETUP] added
06:29:22.068000 [0-0] *   Trying [::1]:2375...
06:29:22.071000 [0-0] * [SETUP] Curl_conn_connect(block=0) -> 0, done=0
06:29:22.072000 [0-0] * [SETUP] Curl_conn_connect(block=0) -> 0, done=1
06:29:22.074000 [0-0] * Connected to localhost (::1) port 2375
06:29:22.077000 [0-0] * using HTTP/1.x
06:29:22.079000 [0-0] > POST /grpc HTTP/1.1
06:29:22.079000 [0-0] > Host: localhost:2375
06:29:22.079000 [0-0] > User-Agent: curl/8.13.0
06:29:22.079000 [0-0] > Accept: */*
06:29:22.079000 [0-0] > Connection: Upgrade, HTTP2-Settings
06:29:22.079000 [0-0] > Upgrade: h2c
06:29:22.079000 [0-0] > HTTP2-Settings: AAMAAABkAAQAAQAAAAIAAAAA
06:29:22.079000 [0-0] >
06:29:22.093000 [0-0] < HTTP/1.1 101 Switching Protocols
HTTP/1.1 101 Switching Protocols
06:29:22.095000 [0-0] < Connection: Upgrade
Connection: Upgrade
06:29:22.097000 [0-0] < Upgrade: h2c
Upgrade: h2c
06:29:22.099000 [0-0] <

06:29:22.100000 [0-0] * Received 101, Switching to HTTP/2
06:29:22.102000 [0-0] * [HTTP/2] added
06:29:22.104000 [0-0] * [HTTP/2] upgrading connection to HTTP/2
06:29:22.106000 [0-0] * Copied HTTP/2 data in stream buffer to connection buffer after upgrade: len=39
06:29:22.108000 [0-0] * [HTTP/2] created session via Upgrade
06:29:22.110000 [0-0] * [HTTP/2] [0] created h2 session (via h1 upgrade)
06:29:22.112000 [0-0] * [HTTP/2] [0] -> FRAME[SETTINGS, len=18]
06:29:22.114000 [0-0] * [HTTP/2] [0] -> FRAME[WINDOW_UPDATE, incr=1048510465]
06:29:22.116000 [0-0] * [HTTP/2] cf_connect() -> 0, 1,
06:29:22.118000 [0-0] * [HTTP/2] Process 39 bytes in connection buffer
06:29:22.120000 [0-0] * [HTTP/2] [0] <- FRAME[SETTINGS, len=30]
06:29:22.122000 [0-0] * [HTTP/2] [0] MAX_CONCURRENT_STREAMS: 250
06:29:22.124000 [0-0] * [HTTP/2] [0] ENABLE_PUSH: TRUE
06:29:22.126000 [0-0] * [HTTP/2] [0] notify MAX_CONCURRENT_STREAMS: 250
06:29:22.128000 [0-0] * [HTTP/2] [1] DRAIN select_bits=3
06:29:22.130000 [0-0] * [HTTP/2] [0] progress ingress: done
06:29:22.131000 [0-0] * [HTTP/2] [0] -> FRAME[SETTINGS, ack=1]
06:29:22.133000 [0-0] * [HTTP/2] [0] egress: wrote 73 bytes
06:29:22.136000 [0-0] * [HTTP/2] [1] cf_recv(len=102400) -> -1 81, window=0/65535, connection 1048576000/1048576000
06:29:22.140000 [0-0] * Request completely sent off
06:29:22.142000 [0-0] * [HTTP/2] [0] ingress: read 22 bytes
06:29:22.144000 [0-0] * [HTTP/2] [0] <- FRAME[WINDOW_UPDATE, incr=983041]
06:29:22.146000 [0-0] * [HTTP/2] [0] <- FRAME[SETTINGS, ack=1]
06:29:22.147000 [0-0] * [HTTP/2] [0] progress ingress: inbufg=0
06:29:22.149000 [0-0] * [HTTP/2] [0] progress ingress: done

In other words, Docker respects the settings frame and seems to be standards-compliant. The only missing bit is to support alternate paths for the h2c upgrade using InsecureFromHttp1ChannelCredentials as you mentioned.

Is this possible today? Or does the library need to be altered?

[ejona86]

In other words, Docker respects the settings frame and seems to be standards-compliant.

@cowwoc, why do you say that?

I see:

06:29:22.079000 [0-0] > HTTP2-Settings: AAMAAABkAAQAAQAAAAIAAAAA
...
06:29:22.112000 [0-0] * [HTTP/2] [0] -> FRAME[SETTINGS, len=18]

It looks like curl re-sends the HTTP2-Settings after the Upgrade, as the HTTP2-Settings is 18 bytes long (after removing base64 encoding). So I don't know how you can conclude that the server actually saw the settings in HTTP2-Settings instead of the duplicate SETTINGS after the upgrade.

After the upgrade the server must still respond to the POST /grpc request, and I don't see any response in your dump. I expect curl hung in your test. curl would exit normally after receiving the response if h2c upgrade was supported. I don't see any evidence it is actually h2c upgrade as defined in RFC 7540.

[cowwoc]

@ejona86 Is there any way you could test the aforementioned request on your end?

My HTTP/2 experience is rather limited, but going by Wireshark I see a settings frame being sent from the client to docker, and docker responding with its own settings frame. So at the very least, the server seems to do something with this frame.

curl keeps on repeating the following lines past the output I pasted above:

04:10:01.193000 [0-0] * [HTTP/2] [1] cf_recv(len=102400) -> -1 81, window=0/65536, connection 1048576000/1048576000
04:10:01.393000 [0-0] * [HTTP/2] [0] progress ingress: done

which I interpret as docker keeping the connection open for follow-up grpc requests.

What is your interpretation of what is going on?

If docker is not HTTP/2 compliant based on the above information, what do you expect the relay to do in this case? It would still try to initiate an HTTP/1 -> HTTP/2 upgrade, but what would you expect it to do differently than a HTTP/2 compliant client (on the end that connects to docker)?

[ejona86]

but going by Wireshark I see a settings frame being sent from the client to docker

That is a curl/nghttp2-ism. gRPC Java won't do that. (gRPC Java can send additional SETTINGS, but it won't re-send the HTTP2-Settings.)

So at the very least, the server seems to do something with this frame.

Oh, I expected it would within the HTTP/2 connection. But I still expect it is ignoring the initial HTTP2-Settings, which is still a problem, as the client will assume the server processed that.

which I interpret as docker keeping the connection open for follow-up grpc requests.

That log doesn't really mean anything to me, but curl should exit once the request is complete. Yes, the server would keep the connection alive in case curl sent more, but curl should be closing the connection as soon as the request is complete. We do see in the log that there is no response sent (HEADERS/DATA).

If docker is not HTTP/2 compliant based on the above information, what do you expect the relay to do in this case?

The docker approach seems to be very similar to HTTP CONNECT. It is a handshake that has no relation to the communication after the handshake. So the relay would do that handshake and then treat it as a regular TCP connection for "HTTP/2 with Prior Knowledge", which is just a plain HTTP/2 connection.

[cowwoc]

@ejona86 For what it's worth, I finally got the relay working.

1. ManagedChannel connects to a LocalServerChannel and exchanges SETTINGS frames.
2. I connect to the docker server and initiate a standard HTTP/2 upgrade. My client and the docker server exchange SETTINGS frames.
3. For every HTTP/2 stream created by ManagedChannel I create a corresponding stream on the docker end.
4. For every data frame I receive, I clone the frame, update streamId to the value expected by the other end, and relay the frame.

This seems to work. Given that I'm using all the standard Netty classes for doing this (not changing anything about the HTTP/2 upgrade steps) I think this again points to the fact that Docker server is fine and does not actually need a relay. We should be able to connect ManagedChannel directly to it so long as it sends a standard h2c upgrade against a custom context path.

I can invoke Docker's gRPC APIs, though I am getting errors because they are not documented and I'm having to figure this out through trial and error.

How would you like to proceed? Are you willing to add support for a custom context path? Should I try to clone the grpc repository, hard-code a change of the context path from / to /grpc and seeing if it works? Do you want me to send you the source-code for the relay that works for me?