[Feature Request]: generate_schema supports chunk_token_threshold parameters

[Wadehl]

What needs to be done?

I would like the generate_schema method in either JsonXPathExtractionStrategy or JsonCssExtractionStrategy to support the chunk_token_threshold parameter.

What problem does this solve?

When I aim to build a system for users who are not familiar with code, I want them to be able to describe the core data schema of a page using natural language. Then, they can use XPath or JsonCSS strategies to quickly and stably extract data, replacing the LLM strategy for direct data extraction. This approach will make data parsing faster and the service more efficient.
While passing a brief HTML_CONTENT as a parameter can achieve the desired functionality and is beneficial for developers familiar with HTML, it requires additional learning and understanding for ordinary users, both in terms of obtaining and parsing HTML. Allowing users to directly solve problems through natural language descriptions using LLM is, in my opinion, a highly meaningful improvement.

Here's my code may help understanding:

result = await crawler.arun(
    url,
    config=CrawlerRunConfig(
        # css_selector=BASE_SELECTOR,
        cache_mode=CacheMode.BYPASS,
    ),
)
html_content = result.html # get the target website's html content

xpath_schema = JsonXPathExtractionStrategy.generate_schema(
    html=html_content,
    schema_type="xpath",
    provider="deepseek/deepseek-coder",
    api_token=CRAWL_API_KEY,
    api_base="https://api.deepseek.com",
    chunk_token_threshold=4096,
    prompt="""
      {ANY PROMPT HERE}
    """,
)

print(xpath_schema)

Target users/beneficiaries

No response

Current alternatives/workarounds

No response

Proposed approach

No response

[aravindkarnam]

@Wadehl Given that now plenty of LLMs support like gemini support 1M context window and also relatively cheap, why not use them for generate_schema tasks? Chunking is quite unreliable for JSON based extraction tasks, that's why suggesting this approach. It works great for tasks like summarisation( that don't need context overlapping)

Another idea is to pass cleaned_html instead of raw html to conserve on tokens.