Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GNN] Acc eval script change: incompatibility or corrupted file? #2119

Open
attafosu opened this issue Feb 19, 2025 · 3 comments
Open

[GNN] Acc eval script change: incompatibility or corrupted file? #2119

attafosu opened this issue Feb 19, 2025 · 3 comments

Comments

@attafosu
Copy link
Contributor

After the change to avoid using memmap when loading the labels #2081, we're encountering pickling errors via numpy.
We're unable to load the labels with numpy.load. The previous loading via np.memmap (be6ff52) did work on our environments without issues

Steps to reproduce:

  1. wget -c https://igb-public.s3.us-east-2.amazonaws.com/IGBH/processed/paper/node_label_2K.npy
  2. python -c "import numpy as np; labels = np.load('node_label_2K.npy', mmap_mode=None)"
  3. Error: ValueError: Cannot load file containing pickled data when allow_pickle=False

After setting allow_pickle=True, we face another error: _pickle.UnpicklingError: invalid load key, '\x00'

Packages used: numpy==1.26.4, torch==2.1.0+cpu

Curious if there's a dependency issue on my side.
cc: @arjunsuresh @nv-alicheng

Was the previous mode of loading the labels incorrect leading to incorrect labels? Or the current modification fixes a previously missed bug? If the latest change doesn't concern correctness of loaded labels, can we bring back the old way so as the default, and switch to np.load if --no-memmap is passed?

@arjunsuresh
Copy link
Contributor

Hi @attafosu I can confirm that this issue is there. @ashwin @nvzhihanj can you please confirm from Nvidia side?

@arjunsuresh
Copy link
Contributor

Not sure if this is the best fix, but is an option to get the script working.

https://github.com/mlcommons/inference/pull/2123/files

@attafosu
Copy link
Contributor Author

The fix looks good to me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants