denied access while copying shards from aws s3 bucket

[emirkaan5]

Hi,

i was trying to replicate the steps for training a model using DCLM-baseline, however, while trying to copy the shard contents from aws-s3 bucket, I get permission denied errors. What should I do?

aws s3 cp --recursive s3://commoncrawl/contrib/datacomp/DCLM-baseline/global-shard_03_of_10/local-shard_1_of_10/ dclm_local fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

[GeorgiosSmyrnis]

Hi @emirkaan5,

Have you set your AWS credentials using either environment variables or aws configure? The commoncrawl bucket requires you to be logged in in order to access it.

[emirkaan5]

hi, yes i configured using aws configure. I created an user in IAM console for myself for access key and secret. Although I assume this link already has the data that is necessary, so I guess I don't need to use aws-s3?

[GeorgiosSmyrnis]

Hi @emirkaan5 ,

Interesting - it might be the case that the permissions set with the IAM account are not enough (I think it has to be explicitly allowed s3 read access)?

Regardless, if HF is a good solution, then downloading it from there works - the dataset that we released can be found under the mlfoundations HF (in jsonl: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0 and parquet: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet formats). I haven't looked into the link from AI2 that you sent, but I think it is the same data as well.

Let me know if you have further questions!