feat(streams): new LimitDelimiterStream()

[BlackAsLight]

This pull request offers a new TransformStream into the mix. While we do already have DelimiterStream, it isn't suitable if your delimiter is unlikely to appear for long stretches of bytes, meaning you could end up with huge chunks at a time. This pull request offers LimitDelimiterStream which as the name implies, offers a maximum length that a chunk can be.

Example

import { assertEquals } from "@std/assert";
import {
  LimitDelimiterStream,
} from "@std/streams/unstable-limit-delimiter-stream";

const encoder = new TextEncoder();

const readable = ReadableStream.from(["foo;beeps;;bar;;"])
  .pipeThrough(new TextEncoderStream())
  .pipeThrough(
    new LimitDelimiterStream({
      delimiter: encoder.encode(";"),
      limit: 4,
    }),
  );

assertEquals(
  await Array.fromAsync(readable),
  [
    { match: true, value: encoder.encode("foo") },
    { match: false, value: encoder.encode("beep") },
    { match: true, value: encoder.encode("s") },
    { match: true, value: encoder.encode("") },
    { match: true, value: encoder.encode("bar") },
    { match: true, value: encoder.encode("") },
  ],
);

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.12%. Comparing base (206ed42) to head (a1cc7ae).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6890   +/-   ##
=======================================
  Coverage   94.11%   94.12%           
=======================================
  Files         581      582    +1     
  Lines       42707    42763   +56     
  Branches     6796     6804    +8     
=======================================
+ Hits        40194    40250   +56     
  Misses       2463     2463           
  Partials       50       50           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[timreichen]

Is there a particular use case for this stream?
I think it is kinda random to split with a delimiter as well as a max chunk size.
To me a some kind of SplitStream would make more sense and possibly could be made more flexible.
The name is also very confusing because one might think it would limit the chunk count instead of the individual chunk size as the Limited*Stream in @std/streams do.

[BlackAsLight]

Is there a particular use case for this stream?

Yes. I'd like to offer a streaming FormData encoder and decoder and in it I found something like this useful. The normal DelimiterStream isn't suitable as it would likely produce huge chunks, possibly too big, when processing the files defeating the purpose of streaming.

I think it is kinda random to split with a delimiter as well as a max chunk size.

To me a some kind of SplitStream would make more sense and possibly could be made more flexible.

I'd need you to be more specific on what this SplitStream would do.

The name is also very confusing because one might think it would limit the chunk count instead of the individual chunk size as the Limited*Stream in @std/streams do.

I'm not attached to the name and open to suggestions if you think there is something better to communicate its intended behaviour.

[kt3k]

Yes. I'd like to offer a streaming FormData encoder and decoder and in it I found something like this useful. The normal DelimiterStream isn't suitable as it would likely produce huge chunks, possibly too big, when processing the files defeating the purpose of streaming.

Sounds interesting to me. Can you add some note about such situation in API doc? (An example illustrating that situation would aslo be very helpful.)

[BlackAsLight]

I added a note explaining more, but I can't really think of an example that's simple to demonstrate the point.