rustdoc-json: Unavailible crates return 403 Forbidden instead of 404 Not Found

[aDotInTheVoid]

Some packages don't have rustdoc-json available, because that package was released before docs.rs started building rustdoc-json. When this happens, I'd expect to get a 404 Not Found response when requesting the json.

Instead, the returned response is 403 Forbidden, which (in my mind) indicates that I don't have permission to access it.

alona@Ashtabula:~/tmp$ curl -I https://docs.rs/crate/libc/0.2.172/json
HTTP/2 302
content-length: 0
location: https://static.docs.rs/rustdoc-json/libc/0.2.172/x86_64-unknown-linux-gnu/libc_0.2.172_x86_64-unknown-linux-gnu_latest.json
server: nginx/1.14.0 (Ubuntu)
date: Tue, 27 May 2025 17:55:56 GMT
content-security-policy: default-src 'none'; base-uri 'none'; img-src 'self' https:
x-cache: Hit from cloudfront
via: 1.1 e33b4b6feeff38ab42cf696cd82a5c1e.cloudfront.net (CloudFront)
x-amz-cf-pop: LHR61-P7
alt-svc: h3=":443"; ma=86400
x-amz-cf-id: j7G8-4F-mqcd6FMDIYCTbguNpuw2scWoOwyCWq03m-uYQqQPNY4Cgg==
age: 471

alona@Ashtabula:~/tmp$ curl -I  https://static.docs.rs/rustdoc-json/libc/0.2.172/x86_64-unknown-linux-gnu/libc_0.2.172_x86_64-unknown-linux-gnu_latest.json
HTTP/2 403
content-type: application/xml
server: AmazonS3
date: Tue, 27 May 2025 18:03:55 GMT
x-cache: Error from cloudfront
via: 1.1 e6606d7d3401505cbf3e6ea0e411484c.cloudfront.net (CloudFront)
x-amz-cf-pop: LHR50-P4
alt-svc: h3=":443"; ma=86400
x-amz-cf-id: K4TV4MCr3aJKJ0SKvDCtCZJqjHIVwnOBJcycmhWNJdZ2a5zuHgpYNw==

[syphar]

Thank you for all your feedback!

[syphar]

From what I know, S3 only allows this when we give ListObjects permissions to the world, which means they can generate directory lists, which feels risky.

Easy way around is similar to what we're doing with the archive downloads, which means the redirect handler checks existance of the target.

[syphar]

There is probably also a way to convert the 403 from S3 at cloudfront level

[syphar]

fixed in our redirect view in #2837