Update scripts and README for downloading datasets from Google Drive

Author: Jiong Zhu

README.md

@@ -45,7 +45,7 @@ As a general note, TensorFlow 1.15 can be used for all code requiring TensorFlow
 
 ### Download Datasets
 
-The datasets can be downloaded using the bash scripts provided in `/experiments/h2gcn/scripts`, which also prepare the datasets for use in our experimental framework based on `signac`.
+The datasets can be downloaded using the bash scripts provided in `/experiments/h2gcn/scripts` (requires the latest version of [`gdown`](https://github.com/wkentaro/gdown) to be installed), which also prepare the datasets for use in our experimental framework based on `signac`.
 
 We make use of `signac` to index and manage the datasets: the datasets and experiments are stored in hierarchically organized signac jobs, with the **1st level** storing different graphs, **2nd level** storing different sets of features, and **3rd level** storing different training-validation-test splits. Each level contains its own state points and job documents to differentiate with other jobs.
 

experiments/h2gcn/scripts/get-real-cora_full.sh

@@ -9,10 +9,17 @@ SHA256SUM=b04a3db58aee34ddec4e24970665a3ef094125f39e2051c6e5024f124caa5053
 cd "$(dirname ${BASH_SOURCE[0]})/.."
 mkdir -p archives
 
-# Automatic downloading script adopted from https://stackoverflow.com/a/38937732
-filename="$(curl -sc /tmp/gcokie "${ggURL}&id=${ggID}" | grep -o '="uc-name.*</span>' | sed 's/.*">//;s/<.a> .*//')"  
-getcode="$(awk '/_warning_/ {print $NF}' /tmp/gcokie)"
-curl -Lb /tmp/gcokie "${ggURL}&confirm=${getcode}&id=${ggID}" -o "${TARGET}"
+# # Automatic downloading script adopted from https://stackoverflow.com/a/38937732
+# filename="$(curl -sc /tmp/gcokie "${ggURL}&id=${ggID}" | grep -o '="uc-name.*</span>' | sed 's/.*">//;s/<.a> .*//')"  
+# getcode="$(awk '/_warning_/ {print $NF}' /tmp/gcokie)"
+# curl -Lb /tmp/gcokie "${ggURL}&confirm=${getcode}&id=${ggID}" -o "${TARGET}"
+
+if ! command -v gdown &> /dev/null
+then
+    read -p "Prerequisite package gdown is not installed. Press any key to install (pip install --upgrade gdown)"
+    pip install --upgrade gdown
+fi
+gdown -O "${TARGET}" ${ggID}
 
 echo "$SHA256SUM  $TARGET" | sha256sum -c
 test $? -eq 0 || read -p "Failed to verify SHA256 checksum. Press any key to continue anyway." -n 1 -r

experiments/h2gcn/scripts/get-real-geomgcn.sh

@@ -9,10 +9,17 @@ SHA256SUM=06bf9a52cb272b3b25227530eafc2a40681fa7c548641ec00ca2427812fbe39f
 cd "$(dirname ${BASH_SOURCE[0]})/.."
 mkdir -p archives
 
-# Automatic downloading script adopted from https://stackoverflow.com/a/38937732
-filename="$(curl -sc /tmp/gcokie "${ggURL}&id=${ggID}" | grep -o '="uc-name.*</span>' | sed 's/.*">//;s/<.a> .*//')"  
-getcode="$(awk '/_warning_/ {print $NF}' /tmp/gcokie)"
-curl -Lb /tmp/gcokie "${ggURL}&confirm=${getcode}&id=${ggID}" -o "${TARGET}"
+# # Automatic downloading script adopted from https://stackoverflow.com/a/38937732
+# filename="$(curl -sc /tmp/gcokie "${ggURL}&id=${ggID}" | grep -o '="uc-name.*</span>' | sed 's/.*">//;s/<.a> .*//')"  
+# getcode="$(awk '/_warning_/ {print $NF}' /tmp/gcokie)"
+# curl -Lb /tmp/gcokie "${ggURL}&confirm=${getcode}&id=${ggID}" -o "${TARGET}"
+
+if ! command -v gdown &> /dev/null
+then
+    read -p "Prerequisite package gdown is not installed. Press any key to install (pip install --upgrade gdown)"
+    pip install --upgrade gdown
+fi
+gdown -O "${TARGET}" ${ggID}
 
 echo "$SHA256SUM  $TARGET" | sha256sum -c
 test $? -eq 0 || read -p "Failed to verify SHA256 checksum. Press any key to continue anyway." -n 1 -r

experiments/h2gcn/scripts/get-syn-cora.sh

@@ -9,10 +9,17 @@ SHA256SUM=93a5329054bc36d742f394589a4de9b6239f8a19ccb6b7b894228841887a413b
 cd "$(dirname ${BASH_SOURCE[0]})/.."
 mkdir -p archives
 
-# Automatic downloading script adopted from https://stackoverflow.com/a/38937732
-filename="$(curl -sc /tmp/gcokie "${ggURL}&id=${ggID}" | grep -o '="uc-name.*</span>' | sed 's/.*">//;s/<.a> .*//')"  
-getcode="$(awk '/_warning_/ {print $NF}' /tmp/gcokie)"
-curl -Lb /tmp/gcokie "${ggURL}&confirm=${getcode}&id=${ggID}" -o "${TARGET}"
+# # Automatic downloading script adopted from https://stackoverflow.com/a/38937732
+# filename="$(curl -sc /tmp/gcokie "${ggURL}&id=${ggID}" | grep -o '="uc-name.*</span>' | sed 's/.*">//;s/<.a> .*//')"  
+# getcode="$(awk '/_warning_/ {print $NF}' /tmp/gcokie)"
+# curl -Lb /tmp/gcokie "${ggURL}&confirm=${getcode}&id=${ggID}" -o "${TARGET}"
+
+if ! command -v gdown &> /dev/null
+then
+    read -p "Prerequisite package gdown is not installed. Press any key to install (pip install --upgrade gdown)"
+    pip install --upgrade gdown
+fi
+gdown -O "${TARGET}" ${ggID}
 
 echo "$SHA256SUM  $TARGET" | sha256sum -c
 test $? -eq 0 || read -p "Failed to verify SHA256 checksum. Press any key to continue anyway." -n 1 -r

experiments/h2gcn/scripts/get-syn-products.sh

@@ -9,10 +9,17 @@ SHA256SUM=ee92199881159dbb259c9b1c580984e3a9a7681b0b5da35ac2d2e36c9e240f26
 cd "$(dirname ${BASH_SOURCE[0]})/.."
 mkdir -p archives
 
-# Automatic downloading script adopted from https://stackoverflow.com/a/38937732
-filename="$(curl -sc /tmp/gcokie "${ggURL}&id=${ggID}" | grep -o '="uc-name.*</span>' | sed 's/.*">//;s/<.a> .*//')"  
-getcode="$(awk '/_warning_/ {print $NF}' /tmp/gcokie)"
-curl -Lb /tmp/gcokie "${ggURL}&confirm=${getcode}&id=${ggID}" -o "${TARGET}"
+# # Automatic downloading script adopted from https://stackoverflow.com/a/38937732
+# filename="$(curl -sc /tmp/gcokie "${ggURL}&id=${ggID}" | grep -o '="uc-name.*</span>' | sed 's/.*">//;s/<.a> .*//')"  
+# getcode="$(awk '/_warning_/ {print $NF}' /tmp/gcokie)"
+# curl -Lb /tmp/gcokie "${ggURL}&confirm=${getcode}&id=${ggID}" -o "${TARGET}"
+
+if ! command -v gdown &> /dev/null
+then
+    read -p "Prerequisite package gdown is not installed. Press any key to install (pip install --upgrade gdown)"
+    pip install --upgrade gdown
+fi
+gdown -O "${TARGET}" ${ggID}
 
 echo "$SHA256SUM  $TARGET" | sha256sum -c
 test $? -eq 0 || read -p "Failed to verify SHA256 checksum. Press any key to continue anyway." -n 1 -r

npz-datasets/README.md

@@ -8,7 +8,7 @@ Note that the new `npz` format does NOT keep the same training, validation and t
 
 ### Download Datasets
 
-The datasets can be downloaded using the bash scripts provided in the `scripts` folder under this folder.
+The datasets can be downloaded using the bash scripts provided in the `scripts` folder under this folder. Installation of the latest version of [`gdown`](https://github.com/wkentaro/gdown) is required.
 
 ### Example Usage