Please explain caching behaviour

[BigNerd]

As it seems, crawl4ai uses a cache and there are configuration options to control usage of the cache, e.g. by-pass the cache for a requested url. However, it is not clear from the documentation, as far as I can see, what the entry and eviction criteria and the maximum size of the cache are. Will every crawling result be cached (if cache is not by-passed)? Or will crawl4ai respect cache control headers from the source web sites, in order to only cache what is marked as cacheable and as long as it is permitted to cache according to HTTP cache control headers? Also, is there a maximum size applied to the cache and how can it be controlled or will the cache grow infinitely until the storage resources of the underlying machine are used up, leading to errors finally?

[unclecode]

Hi @BigNerd

Crawl4ai employs a cache to store crawling results, and you can manage its usage through configuration options like bypassing it for specific URLs. Let's clarify the cache's behavior based on your questions:

Entry Criteria:

By default, if caching is enabled (which is the default CacheMode.ENABLED unless you specify otherwise), crawl4ai will attempt to cache every successful crawling result. Specifically, within the arun method of AsyncWebCrawler (crawl4ai/async_webcrawler.py), if a CrawlResult is successfully obtained (meaning html is not empty), and the CacheContext determines it should_write(), the CrawlResult will be written to the cache database.

Eviction Criteria and Maximum Size:

Based on the provided codebase, crawl4ai does not implement explicit eviction criteria or a maximum size for the cache. The cache database, managed by async_db_manager (crawl4ai/async_database.py), appears to grow indefinitely, limited only by the storage capacity of the underlying machine. There is no logic for automatically removing older or less frequently used entries to control the cache size within the provided code.

One very important thing to mention is that the caching mechanism has two layers. We store data in a database and in the file system, especially for large content like HTML markdown. We create a hash for this content and store the hash code in the database. Then, we use the hash code as the name of the file and store it on a disk in the home directory of Crawl4ai. We always retrieve from there. This way, the database does not bloat and grow larger due to such large files.

Cache-Control Headers:

I'm happy you shared this point with us. This has been the backlog, and I will bring it up very soon. It's a good thing to consider, and I really like the idea. We added an option similar to how we handle the robot.txt and other items. Right now, the caching behavior solely depends on the cache mode you set in your crawler1 config and whether a matching entry exists in the local cache database. I will definitely prioritize this. Thanks again.

Controlling Cache Usage:

You can control cache usage through the cache_mode parameter within CrawlerRunConfig (crawl4ai/async_configs.py). The available options, defined in the CacheMode enum (crawl4ai/cache_context.py), are:

• CacheMode.ENABLED: (Default) Reads from and writes to the cache.
• CacheMode.BYPASS: Bypasses the cache entirely, always fetching fresh content and not writing to the cache.
• CacheMode.DISABLED: Disables caching completely (legacy option, equivalent to BYPASS).
• CacheMode.READ_ONLY: Only reads from the cache, does not write new results.
• CacheMode.WRITE_ONLY: Only writes to the cache, does not read existing results.

To manage cache size, you would need to implement manual cache clearing or size monitoring and control mechanisms, as the library does not provide built-in features for cache eviction or size limits. You can use methods like aclear_cache() and aflush_cache() in AsyncWebCrawler (crawl4ai/async_webcrawler.py) for manual cleanup.

Future Plan
The team will fundamentally rewrite the caching mechanism from scratch. The new approach will use a strategy design pattern that allows people to create their own caching strategies through inheritance. This model will consider size and other parameters for efficiency. I expect this to be ready by the second quarter of 2025.

@aravindkarnam

[BigNerd]

Thank you for the quick and thorough response. This answers my questions and helps me to use your fantastic solution appropriately. As I am using your docker image running as a service, I will send all requests with CacheMode.BYPASS to avoid stale responses and infinite cache growth. I am looking forward to a redesigned cache mode.