[FEAT] Can Chonkie directly split a Markdown file by headers?

[qiankunli]

1. Split the Markdown file according to the smallest level of header detected.
2. After splitting by headers, if the length of a certain block exceeds a threshold, split that block by threshold.

[bhavnicksm]

Hey @qiankunli! 😄

Thanks for opening an issue~

To answer you, in short, yes! It's quite straight-forward to do so with the RecursiveChunker and we have a tutorial on this as well (see: Cookbook on one way to go about it)

Currently, in the src we have the v0.5.0 available, which has the include_delim="next" option, which is helpful for markdown headers to be included in the next chunk, rather than the prev which is default behavior. You can also test that out if you wish.

Hope that helps~

Thanks 😊