Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong content-length header for datasets #261

Open
fdietze opened this issue Sep 7, 2024 · 1 comment
Open

Wrong content-length header for datasets #261

fdietze opened this issue Sep 7, 2024 · 1 comment

Comments

@fdietze
Copy link

fdietze commented Sep 7, 2024

Describe the bug
When downloading the datasets from https://x.com/i/communitynotes/download-data using wget, it hangs, not receiving more data, because the content-length header is too big (566M) for the file being served (185M).

To Reproduce

wget https://ton.twimg.com/birdwatch-public-data/2024/09/07/notes/notes-00000.tsv
wget https://ton.twimg.com/birdwatch-public-data/2024/09/07/notes/notes-00000.tsv
--2024-09-07 13:43:04--  https://ton.twimg.com/birdwatch-public-data/2024/09/07/notes/notes-00000.tsv
Resolving ton.twimg.com (ton.twimg.com)... 2606:2800:233:7ee2:97c:ab4c:6c70:be36, 152.199.21.140
Connecting to ton.twimg.com (ton.twimg.com)|2606:2800:233:7ee2:97c:ab4c:6c70:be36|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 593120688 (566M) [text/tab-separated-values]
Saving to: ‘notes-00000.tsv’

notes-00000.tsv         32%[=======>                   ] 185.19M  --.-KB/s    eta 2m 58s

Expected behavior
The content-length header should be set to the file size.

@jbaxter
Copy link
Collaborator

jbaxter commented Sep 9, 2024

Interesting, I can repro this. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants