Recursive urls extraction

[dkurdomanenko]

Is it possible to extract all page contents in MD recursively using crawl4ai from the provided web site?

[PeterLappo]

I couldn't find anything in the code that did recursive crawling ie follow the links crawling.

[aravindkarnam]

@PeterLappo @dkurdomanenko It's part of the upcoming v0.5 release this week. A feature called deep crawl that allows you to explore an entire sites upto specified depth with different traversal techniques along with filtering and sorting of links. Stay tuned on our changelog for more updates, but it's going to be likely released within this week.

[PeterLappo]

Great. Looking forward to test it out.

[ehsanahmadkhan525]

Looking forward to it.