Skip to content

Dataset hosting and verification  #97

Open
@mikeheddes

Description

@mikeheddes
Member

Currently the datasets are hosted on the UCI machine learning repository and Google Drive. However, the Google Drive links are not very robust as they modify their download flow sometimes. We are thus looking for a host that provides great spreed, reliability and is easy to download from programmatically. Something like Google Cloud Storage or AWS S3, however these are not free.

At the same time for security and usability reasons we should verify if the download succeeded by checking the file's checksum.

Activity

mikeheddes

mikeheddes commented on Mar 6, 2023

@mikeheddes
MemberAuthor

I recently got this exception:

Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.

Although I was downloading the datasets pretty often so the exception is understandable, if this library is to grow among the community I expect this error to happen more often which is not an acceptable user experience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @mikeheddes

        Issue actions

          Dataset hosting and verification · Issue #97 · hyperdimensional-computing/torchhd