Questions about dataset

[pierowu]

According to the paper, there a 2 datasets for finetuning which come from Github and books seperately. But it seems that only Github Corpus is released under your huggingface account. Could you share another one?

Another question is that I' ve noticed that there is another dataset in your huggingface account called CodeGen_RE_data. Could you explain the source of this dataset?

Thanks!

[martinwz]

Could you please share the url of Github Corpus found here? I also need it to build my model. Thank you.

[shailja-thakur]

Hi, thanks for reaching out. The statistics in the paper reflects the one on top of the processed samples, and the data in the hugging face is the original raw data, which is reflective from its size , in GBs. Do apply the processing steps as described in the paper, they are very nominal but will result in filtering the samples.

…

On Thu, Mar 20, 2025, 9:14 p.m. Li Juqi ***@***.***> wrote: I have the same problem as pierowu <https://github.com/pierowu>. Additionally, it seems that the size of this dataset <https://huggingface.co/datasets/shailja/Verilog_GitHub> does not correspond with the statistics mentioned in the paper (400 MB after filtering). — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXKPBOQWLHMPSC5PW4R6LL2VLO5BAVCNFSM6AAAAABMV2NWCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBQHEYDENZXHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [image: vhagor]*vhagor* left a comment (shailja-thakur/CodeGen-Fine-Tuning#6) <#6 (comment)> I have the same problem as pierowu <https://github.com/pierowu>. Additionally, it seems that the size of this dataset <https://huggingface.co/datasets/shailja/Verilog_GitHub> does not correspond with the statistics mentioned in the paper (400 MB after filtering). — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXKPBOQWLHMPSC5PW4R6LL2VLO5BAVCNFSM6AAAAABMV2NWCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBQHEYDENZXHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>