Docker REST API Crawl with Authentication

[brendandebeer]

Hi,

I am really looking forward to using this impressive tool to crawl secure pages (Confluence) that I can access with a bearer token (Personal Token) - I would have though this to be a trivial task and from the documentation is also supported.
I found the following in the documentation :

and created the following crawl task based on this :

curl --location --request POST 'http://127.0.0.1:11235/crawl' \
--header 'Authorization: Bearer 12345' \
--header 'Content-Type: application/json' \
--data-raw '{
    "urls": "https://my.site/display/Home",
    "crawler_params": {
        "verbose": "True",
        "headers": {
            "Authorization": "Bearer <token>"
        }
    }
}'

but this always returns the login page, and crawl4ai obviously shows success as the login page was crawled correctly.

To verify the page is accessible, I executed the following :

curl --location --request GET 'https://my.site/display/Home' \
--header 'Authorization: Bearer <token>' 

which returns the page with all content as I would expect. It seems as if crawl4ai is not passing the header when crawing the specified url. The logs also do not seem to give much insight.

It would be great if someone with more experience could give some tips as to what I am missing or even suggest a better way to access confluence pages maybe.

Thanks in advance for the help !