Cannot wait_for css

[minhvq36]

I test the code in https://docs.crawl4ai.com/core/page-interaction/

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
config = CrawlerRunConfig(
# Wait for at least 30 items on Hacker News
wait_for="css:.athing:nth-child(30)"
)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://news.ycombinator.com",
config=config
)
print("We have at least 30 items loaded!")
# Rough check
print("Total items in HTML:", result.cleaned_html.count("athing"))

if name == "main":
asyncio.run(main())

The result always returned time out even though I set page_timeout quite high. How can I fix it?

[aravindkarnam]

@minhvq36

I've tried the example in this page. Here the key issue is that there's interweaved tr elements which makes the application of nth-child in css a bit tricky. As in for every submission there's many spacing elements(with same tr tag) in between.

So what would actually work for wait condition is css:tr.athing.submission:nth-child(88) as there three tr elements for each submission. However I find this to be a bit of a trickshot(requires a css expert 😅). So I'll go with js based waiting in this case.

Another catch is that cleaned_html will not have class names on html tags. So the result.cleaned_html.count("athing") will not tell you if the wait_for did it's job. I would check this on raw html(result.html) instead.

Here's the code sample that would work for you

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    js_condition="()=>{return document.querySelectorAll('tr.athing.submission')[29]}"
    config = CrawlerRunConfig(
    # Wait for at least 30 items on Hacker News
    wait_for=f"js:{js_condition}",
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
        url="https://news.ycombinator.com",
        config=config
        )
        print("We have at least 30 items loaded!")
        # Rough check
        print("Total items in HTML:", result.html.count("athing"))

if __name__ == "__main__":
    asyncio.run(main())

Thanks for bringing this up. I'll update the docs soon with a working example.