Support `Transfer-Encoding: chunked` responses in HttpStore

[aldenks]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I'm not able to use the HttpStore to read data returned with Transfer-Encoding: chunked. In my case, I'm hoping to read through a proxy and it would be inefficient if the proxy buffered the full file before returning a response so it's using chunked encoding.

Describe the solution you'd like
To read data returned with chunked transfer encoding in the same way as other responses.

Describe alternatives you've considered

• The proxy could buffer the full file, but that increases latency considerably and requires potentially unbounded memory usage in the proxy.
• I considered trying to work around the current check for a valid Content-Length by seeing if I could modify the proxy to return one even if it was using Transfer-Encoding: chunked but that's very clearly not standards compliant and im not sure if it would solve the issue.

[tustvold]

Response buffering is normally a configuration option of the proxy and not a property of the transport encoding.

For example

https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering

Chunked transfer encoding makes little sense when payload sizes are known, it merely adds overhead. It also is removed in more recent HTTP versions

[mattijn]

Hi @tustvold, we are struggling with the strict behaviour of object_store with failing retrievals when content-length header is missing from the response (see issue at: vega/vegafusion#569). I raised an issue at jsDelivr detailing the situation (jsdelivr/jsdelivr#18661) where my question is answered by stating:

I recommend updating the client code to correctly handle Transfer-Encoding: chunked. When done correctly, it provides the same level of consistency guarantee as checking the Content-Length.

The answer also includes a reference to the http specification stating that clients must be able to parse and decode the chunked encoding. (source here: https://httpwg.org/specs/rfc9112.html#chunked.encoding).

While recent versions of the http protocol don't rely on chunking, I hope we still can find a way to make object_store compliant as a http/1.1 client.

[kylebarron]

I think a core part of the confusion here, is that @mattijn and @aldenks you're using HTTPStore as a generic HTTP client. The HTTPStore is not a generic HTTP client but rather an abstraction to integrate with WebDAV, which then fits into the object-store oriented abstractions of this crate.

Object stores have stronger guarantees than the HTTP spec. Object stores guarantee that the content length is never missing. So if you're hitting this error, you're using the wrong abstraction for your requirements.

[aldenks]

ahhh, that makes sense, thank you @kylebarron! closing

[jonmmease]

Side question @kylebarron — is there a recommended way to query files (e.g. CSV, Parquet) from a generic HTTP server using DataFusion? DataFusion's HTTP support in the CLI goes through the object_store crate's HttpStore, which as you note assumes WebDAV semantics. For servers that don't guarantee Content-Length or range request support (like CDNs using chunked encoding), is there a recommended pattern or an alternative ObjectStore implementation that handles this more permissively?

[kylebarron]

At the Python level we're wrestling with this in developmentseed/obspec-utils#65: what abstraction lets us handle both object stores and http sources that may or may not have a content length defined?

For Parquet specifically, the Parquet crate's AsyncFileReader trait is very general. But, since Parquet's metadata is at the end of the file, you need to either be able to perform suffix range requests or be able to know the content length. For HTTP sources that may or may not guarantee a content length, it's hard to generically load Parquet sources.

In the context of DataFusion specifically, I think DataFusion is pretty tied to the ObjectStore abstraction. I'm not sure how to swap out data fetching in DataFusion for something more general, even if you only need data fetching, not other ObjectStore methods. I guess that's where you define your custom TableProvider then? Not sure if you can override the AsyncFileReader that the DataFusion Parquet reader uses?

[tustvold]

I would just like to clarify that whilst @kylebarron is correct that HttpStore is designed around WebDav it can certainly be used against a regular HTTP server, provided it returns a Content-Length, and if range requests are being used - that it supports them.

or servers that don't guarantee Content-Length or range request support (like CDNs using chunked encoding)

What CDN are we talking about here, all the CDNs I've encountered preserve the origin-behaviour, or at least can be configured to do so. Separately most are using HTTP/2 or even HTTP/3 these days which don't even have a concept of chunked encoding...

[mattijn]

My comment above links to jsDelivr CDN. Not necessarily a small CDN.., see also discussion there: jsdelivr/jsdelivr#18661.

Do you maybe know how to utilize HTTP/2+ on jsDelivr or how to find out if they provide support for this?

[tustvold]

Have you tried enabling HTTP2? https://docs.rs/object_store/latest/object_store/client/struct.ClientOptions.html#method.with_allow_http2

[mattijn]

Not yet, but that is something very worthwhile to try! Thanks a lot for pointing that out!