new: Support Bazel Remote APIs.

[milesj]

Will start with caching.

[harlequin]

Hey,

I have seen that this feature was not introduced into the 1.29 release.
What are the missing pieces, despite the merge conflicts.

Maybe I can support here.

[milesj]

@harlequin The requests from moon to bazel-remote fail with cryptic http2 errors that I've been unable to figure out.

hyperium/tonic#1949

[harlequin]

As far as I remember HTTP2 implies that you have SSL enabled. Will try to setup an environment for testing.

[milesj]

Yeah I think the problem has to do with the bazel/tonic libraries being http2, while bazel-remote is http1, and the communication between the 2 just isn't working. This area isn't an expertise of mine.

[harlequin]

I have played around with the remote service ...

What I have found so far

1. TLS has to be activated and the endpoint has to include .assume_http2(true)

https://docs.rs/tonic/latest/tonic/transport/channel/struct.ClientTlsConfig.html#method.assume_http2

endpoint = endpoint
            .tls_config(
                ClientTlsConfig::new()
                    .with_enabled_roots()
                    .assume_http2(true)                    
            )
            .unwrap();

With this setting I can connect to the remote service

2. GZIP, ZSTD compression

So with the enabled connection moon tries to upload the cache into the remote service

bazel-remote-service shows

2024/10/15 07:37:48 POST 400 192.168.44.1 /build.bazel.remote.execution.v2.ContentAddressableStorage/BatchUpdateBlobs

But on moon side I am receiving

[crates\remote\src\cache_api.rs:96:13] &error = Status {
    code: Internal,
    message: "protocol error: received message with invalid compression flag: 114 (valid flags are 0 and 1) while receiving response with status: 400 Bad Request",
    metadata: MetadataMap {
        headers: {
            "content-type": "text/plain; charset=utf-8",
            "x-content-type-options": "nosniff",
            "date": "Tue, 15 Oct 2024 07:37:48 GMT",
            "content-length": "126",
            "set-cookie": "4afd39e673bbfe62f6f7f4a9622d53cd=9c659af26f02f4f78a396f6570abab67; path=/; HttpOnly; Secure; SameSite=None",
        },
    },
    source: None,
}
[crates\remote\src\cache_api.rs:97:13] error.code() = Internal
[crates\remote\src\cache_api.rs:98:13] error.metadata() = MetadataMap {
    headers: {
        "content-type": "text/plain; charset=utf-8",
        "x-content-type-options": "nosniff",
        "date": "Tue, 15 Oct 2024 07:37:48 GMT",
        "content-length": "126",
        "set-cookie": "4afd39e673bbfe62f6f7f4a9622d53cd=9c659af26f02f4f78a396f6570abab67; path=/; HttpOnly; Secure; SameSite=None",
    },
}
[crates\remote\src\cache_api.rs:99:13] error.message() = "protocol error: received message with invalid compression flag: 114 (valid flags are 0 and 1) while receiving response with status: 400 Bad Request"
Error:   × Main thread panicked.
  ├─▶ at C:\Users\rustmec\code\contributions\moon\crates\remote\src\cache_api.rs:101:13
  ╰─▶ Status { code: Internal, message: "protocol error: received message with invalid compression flag: 114 (valid flags are 0 and 1) while receiving response with status: 400 Bad Request", metadata: MetadataMap { headers: {"content-type": "text/plain; charset=utf-8", "x-content-type-    
      options": "nosniff", "date": "Tue, 15 Oct 2024 07:37:48 GMT", "content-length": "126", "set-cookie": "4afd39e673bbfe62f6f7f4a9622d53cd=9c659af26f02f4f78a396f6570abab67; path=/; HttpOnly; Secure; SameSite=None"} }, source: None }
  help: set the `RUST_BACKTRACE=1` environment variable to display a backtrace.

So what I have seen bazel-remote supports uncompressed and zstd for communication.
In moon is only gzip available as config.

My best guess by today, that something has to change additional to the compression ...

Hope this helps

[harlequin]

Hey @milesj

I have looked into the complete topic of bazel-remote-api and the specifications.

So this is what I have found out:

• bazel defines 2 major protocol specifications, http1.1 (legacy) and grpc (new)

  http1.1 is only handling CAS and AC storage via PUT, GET and HEAD commands reference

  Implementing this would decrease the complexity inside moon and would have the benefit that mostly every http server can handle this kind of requests.

  Even some people have implemented this directly with s3 storages. (can't find a good reference for it)

  grpc is a next iteration for the bazel-remote-api, which includes many service like CAS, AC, remote-execution, log stream and so on.

• the server which you are choosen bazel-remote is not the interface implemented for the asset api, here we shall choose another product

• bazel differentiates the protocols with the https or grpcs protocol settings inside the url

• the crate bazel-remote-api, which is used now in moon can only handle grpc protocol

Summary

I would go the way to split this tasks into, at least, 2 seperate Issues / Features

1. Implement bazel-remote-api with HTTP1.1 and reqwest crate (incl. reqwest-netrc) for authentication

   This would have the benefit that the functionallity from moonbase in regards of remote-caching is equal and can be replaced.

2. Implement CAS and AC bazel-remote-api with GRPC

   With this task you can define on the roadmap, which services moon will support, as mentioned CAS and AC would be a great starting point.

All other services from the bazel specification can introduced over time in the moon application.

[milesj]

Awesome work. I wasn't even aware that the bazel APIs had 2 separate protocols. But that definitely makes sense for the issues I've been having.

I also agree with splitting this up into 2 separate features. I'm going to try and focus on this for 1.30. It would be nice to land this and python support in the next release.