How would I write an HTMX extension for "ReadableStream" or "Transfer-Encoding: Chunked"?

[ancientstraits]

For my website, the user has to perform a process that takes a long time, and thus requires some status updates. I just wanted to have the HTTP server to send status data to the client over a period of time, by using a single HTTP request, and then let the request finish once the process was done. I was able to do this using the js fetch API, but not using the XMLHttpRequest API, which is bad, because HTMX requires the XMLHttpRequest API to make HTTP requests.

For instance, this could be what the server does (if it runs on Express):

const sleep = delay => new Promise((resolve) => setTimeout(resolve, delay))

app.post('/thing', async (req, res) => {
    res.write('hi\n')
    await sleep(1000)
    res.write('what\n')
    await sleep(1000)
    res.write('bye\n')
    res.end()
})

So when the client does POST /thing, the server should instantly write hi\n, then write what\n after 1 second, then write bye\n and end the request after 2 seconds. When I do curl http://localhost:8080, it works just as expected.

When I use this JS code that uses fetch on the browser, it also works:

const resp = await fetch('/thing', { method: 'POST' })
const reader = resp.body.getReader()
const decoder = new TextDecoder()
let result = await reader.read()
while (!result.done) {
    const text = decoder.decode(result.value)
    console.log(text)
    result = await reader.read()
}

However, no matter how much I try, the XMLHttpRequest API cannot do the same:

function handleEvent(e) {
    console.log(e.type)
}

function addListeners(xhr) {
  xhr.addEventListener("loadstart", handleEvent);
  xhr.addEventListener("load", handleEvent);
  xhr.addEventListener("loadend", handleEvent);
  xhr.addEventListener("progress", handleEvent);
  xhr.addEventListener("error", handleEvent);
  xhr.addEventListener("abort", handleEvent);
}

const xhr = new XMLHttpRequest();
addListeners(xhr);
xhr.open("GET", "/thing");
xhr.send();

When I do this, none of the event listeners of xhr are triggered 1 second after the request has been send (which is when the server sends what\n. And no matter how hard I try, it stays that way. Some of the handlers are triggered when the request is started, and some are triggered when the request is finished, but none are triggered when the server sends intermediate data, like what\n.

This is not good, because I want to use HTMX in my website to make everything run simpler, and to write an extension, I have to use the XMLHttpRequest API instead of the fetch API, meaning I cannot handle incremental HTTP responses.

I wonder if there is a way to change the code on the server to make this work, or if there is something about the XMLHttpRequest API that I am missing. Is there any way to make this work?

[bgdev-guy]

For the response, you need to set the transfer encoding to chunked. This will cause xhr.progress to be triggered when the data is flushed from the server.

res.setHeader('transfer-encoding', 'chunked');

Note that HTMX doesn't currently support the chunked encoding.

[ethnoxcom]

Look at SSE extension, it could help.

https://v1.htmx.org/extensions/server-sent-events/