[Streamable] Store brokers in meta chunk

[AaronVasquez]

{
    // ...
    brokers: [
        alpha: "alphaBroker",
        beta: "betaBroker"
    ],
}

[AaronVasquez]

@EdmundMai do you have bandwidth for this?

[EdmundMai]

Yep I'll work on this next

[nullpilot]

Is this exclusively for the temporary downloads before full attachment?

Considering the potential effect this has on decentralisation and anonymity, I think this should be optional, at best.

[CKH4]

I agree with Jet and would like to add that if we go this route, we should make it a part of streamable to allow custom data to be stored in the meta chunk. Then have the webinterface store the alpha and beta addresses.

[rfornea]

I agree, let's make it optional. Some box that the user can check (on an intelligent wrapper service) that says something like "Enable early download."

I also think it'd be useful, like Connor said, to let streamable allow custom data in the metadata. The brokernode addresses could be the first custom data we support.

For the broker, we could make a whole new repo for the early download logic. That repo could include an endpoint, something like :someOtherPortBesides3000/early-download-enabled, which returns true. If the request fails or it returns false, the wrapper wouldn't use that broker if the user wanted to be able to do an early download. (This won't matter while we still control all the brokers, but someday we'll have public brokers and not all public brokers will necessarily want to support this). If early download is enabled on this broker, it calls various endpoints that are included in the early download repo to retrieve the chunks. Doing it like this would make it truly optional and wouldn't mix together core protocol stuff with non-core features. If the early-download directory is always located in a certain directory relative to the brokernode directory, then the running early-download code would still have access to the brokernoode's .env file, for accessing credentials, knowing the SQL database URL, etc. The early-download logic would only ever read, not write.

[EdmundMai]

Ok. I think we could just make it open ended for optional metadata. Let the users of oyster-streamable store whatever they want, whether it be alpha, beta, foo, bar, etc. We could store it under a custom: {} key so that it doesn't pollute the top level of the metadata object.

What do you guys think about that?

[nullpilot]

Fully agreed. Since the metadata format from the rev2 reference implementation needs adjusting anyway, this could easily be carried over as well.

[CKH4]

I agree on that Edmund. I think that we should have the special entries only be storable by function and should have special checks associated to ensure that the metadata doesn't overflow the chunk before we have dynamic meta sizes