(FastAPI) forward multi-file streaming to httpx POST request, without writing to disk?

[hsunami10]

Just like the title says! I’ve noticed that the use of FileTarget writes to disk. But, for my use case, I want to forward multiple files to a httpx POST request. I’ve already implemented this answer here, but it still writes to disk. Is there any way to do this without writing to the filesystem, and instead keeping streaming directly to another HTTP request? (thus keeping memory usage low)

[hsunami10]

so it looks like I'll have to make a custom Target class of sorts - and implement on_data_received and pop_lines myself, similar to CSVTarget. but the problem is, I'm not sure how to stream on_data_received to a httpx.post request's form data.

[siddhantgoel]

What's your setup? Are you directly uploading to the server, or are you uploading to a server that in turn uploads to a server? You need streaming-form-data only in the second case.

If you're directly uploading to a server (the first scenario I mentioned), I think just an httpx.post should be enough. According to https://www.python-httpx.org/advanced/clients/#multipart-file-encoding:

It is safe to upload large files this way. File uploads are streaming by default, meaning that only one chunk will be loaded into memory at a time.

[hsunami10]

@siddhantgoel mine is the second case, where I'm uploading to a server that "forwards" the uploading to another server. My FastAPI app uses streaming-form-data and I'm trying to figure out how to forward that to another upload request.

is it possible to use on_data_received to forward those bytes to another streaming upload while they arrive?

[siddhantgoel]

OK, yes in that case a custom target might work, at least from the client side. I haven't looked at the StackOverflow answer in too much detail. But from a first glance it seems to use FileTarget which is probably the reason the file is being written to disk. Have you tried creating a new target by inheriting BaseTarget?

[hsunami10]

yeah I was thinking of that - I'll try it in a couple days and let you know if it works! will post the solution here if it does