perf(httpfs): pool HTTP connections and cache remote file sizes - #62
Merged
Conversation
The xet:// remote path was paying enormous per-request churn: - A fresh httplib::Client was created per request/retry, so every range GET tore down and re-established TLS (~240 connects, ~5 per socket). - Each openFile() -> initMetadata() issued a HEAD (899 HEAD round-trips) because the ice-disk CSR scan re-opens the same parquet files per morsel. Fixes: - Process-wide per-host pool of no-follow httplib clients (getSharedNoRedirectClient/evictAndGetSharedNoRedirectClient). All requests to a host share one keep-alive TLS socket; retries evict a stale pool entry instead of reusing a dead socket. - Scope the shared-connection lock to a single host request and release it before following a redirect (avoids self-deadlock on recursion). - Process-wide cache of remote file sizes keyed by URL so openFile() skips the HEAD on repeat opens. Result on the profile query: connects 240 -> 2, HEAD round-trips 899 -> 5, end-to-end wall time ~155s -> ~31s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the pathological network churn in the xet:// remote read path that made queries like:
very slow over HTTP (originally >400 tcp_sendmsg and ~155s+ to complete).
Root cause
httplib::Clientwas created per request/retry, so every range GET and HEAD tore down and re-established TLS — ~240 connects, ~5 per socket.openFile() → initMetadata()issued aHEAD— ~899 HEAD round-trips — because the ice-disk CSR scan re-opens the same parquet files per morsel.Changes
getSharedNoRedirectClient/evictAndGetSharedNoRedirectClienthold one keep-alive TLS socket per host reused across all requests; retries evict a stale pool entry instead of reusing a broken socket.openFile()skips the HEAD.Result (profile query)
No functional change; queries still return correct results.