Add multi dataset option for GB24 #221

jychoi-hpc · 2024-04-01T17:53:03Z

Two update:

using multiple datasets for GB24
addiiotn HYDRAGNN_USE_VARIABLE_GRAPH_SIZE env to skip the check if graph size is variable or not at starting.

allaffa

I think it would be better if we create a standalone example to pull multiple datasets together.

examples/multidataset/train.py

hydragnn/utils/config_utils.py

allaffa · 2024-04-01T20:37:31Z

examples/multidataset/train.py

+
+        assert not (args.shmem and args.ddstore), "Cannot use both ddstore and shmem"
+        if args.ddstore:
+            opt = {"ddstore_width": args.ddstore_width, "local": True}


Why do we use local=True?
Is it because we want different ranks to import different datasets?

It is intended to indicate that the dataset will setup for DDStore is a set of "locally" owned and thus no need to split internally. Previously, we provide a global view to DDStore and DDstore split them.

Would this affect global shuffling?

allaffa

Some clarifications needed :)

allaffa

One more question.

* add multi dataset option * create a separate directory and update * black * black * replace OC2020 with GFM

add multi dataset option

b4aa734

jychoi-hpc requested a review from allaffa April 1, 2024 17:53

allaffa requested changes Apr 1, 2024

View reviewed changes

jychoi-hpc added 2 commits April 1, 2024 16:18

create a separate directory and update

b11841c

black

75ab7f1

jychoi-hpc requested a review from allaffa April 1, 2024 20:24

black

40bba7b