-
Notifications
You must be signed in to change notification settings - Fork 137
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #518 from digitalearthafrica/gdrive
Tools for gdrive
- Loading branch information
Showing
6 changed files
with
318 additions
and
2 deletions.
There are no files selected for viewing
232 changes: 232 additions & 0 deletions
232
Frequently_used_code/Integrating_with_Google_Drive.ipynb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,232 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "markdown", | ||
"id": "89bb88ae-35d8-44c1-bbf3-9af915ac5086", | ||
"metadata": {}, | ||
"source": [ | ||
"# Exporting to Google Drive from the Sandbox" | ||
] | ||
}, | ||
{ | ||
"cell_type": "raw", | ||
"id": "36320092-0334-460c-ad65-fafc3d7ebad0", | ||
"metadata": {}, | ||
"source": [ | ||
"**Keywords** :index:`package; google`" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "7247cc3c-e692-477c-b626-82a2b13807aa", | ||
"metadata": {}, | ||
"source": [ | ||
"## Background\n", | ||
"\n", | ||
"Google Drive API enables developers to integrate Google Drive functionality into applications, offering access to file storage, sharing, and synchronization capabilities. It allows users to manage files, folders, and permissions programmatically, facilitating seamless integration of cloud storage features into various software solutions. The API supports multiple programming languages and provides robust documentation, making it versatile for a wide range of applications from document management to collaborative platforms. More information can be found [here](https://developers.google.com/drive/api/guides/about-sdk).\n", | ||
"\n", | ||
"To connect to the Google Drive API from the Sandbox using a service account, a `service account` is created in the Google Cloud Console. Users then need to download its JSON key file, and configure permissions on both Google Drive and the service account. This approach bypasses user authentication and allows server-to-server interactions, making it suitable for automated tasks or applications where user interaction is not required. The service account JSON key is securely stored and used by Python code within Jupyter notebooks to authenticate and authorize API requests, enabling seamless integration of Google Drive functionalities directly into data science workflows without user intervention. \n", | ||
"\n", | ||
"Detailed instructions on integrating Google Drive are [documented in the DE Africa user guide](https://docs.digitalearthafrica.org/en/latest/platform_tools/googledrive_access.html).\n", | ||
"\n", | ||
"**Security considerations:**\n", | ||
"Users should take care to ensure the security of credentials uploaded into the Sandbox, including by ensuring they are not incidentally included in uploads to public repositories like GitHub." | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "b9b67d58-9fb6-4a8a-8fb0-4105d33afafd", | ||
"metadata": {}, | ||
"source": [ | ||
"## Description\n", | ||
"\n", | ||
"This notebook provides a brief introduction to accessing and using Digital Earth Africa's sandbox with Google Drive:\n", | ||
"\n", | ||
"1. The packages to load up\n", | ||
"2. How to use the list files in the google drive\n", | ||
"3. How to upload file files in the google drive\n", | ||
"\n", | ||
"***" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "8a42fd34-7c4e-4d58-b914-41216d9f018a", | ||
"metadata": {}, | ||
"source": [ | ||
"## Getting started\n", | ||
"\n", | ||
">##### To use this notebook, make sure to follow these steps [here](https://docs.digitalearthafrica.org/en/latest/platform_tools/googledrive_access.html) before proceeding to the next cell. \n" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "21bba818-ee4d-4031-876e-edf42492c6b5", | ||
"metadata": {}, | ||
"source": [ | ||
"### Load packages\n", | ||
"Import Python packages that are used for the analysis." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 1, | ||
"id": "aa0cf6dc-d5d4-42b3-8e6e-bb3bc676a908", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"from deafrica_tools.externaldrive import list_gdrive, upload_to_gdrive" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "1e9f3df4-620b-4ab0-b06b-16ae8e8a3d5b", | ||
"metadata": {}, | ||
"source": [ | ||
"#### List Files in Google Drive\n", | ||
"\n", | ||
"`list_gdrive` function list the files and folder within the service account drive." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 2, | ||
"id": "160bfce9-1183-4259-9a61-ecf66a103094", | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"Files:\n", | ||
"Animated_timeseries.ipynb (19oBKPC8BKN77d45STpk1xAmA8_2gxYLO)\n", | ||
"deafricaout (1Cn3e0m8T0OPjZ4dZOnoeXBiGrNiuLo0Y)\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"list_gdrive()" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "b84357e8-912c-4b0c-afdf-89d95df7022a", | ||
"metadata": { | ||
"tags": [] | ||
}, | ||
"source": [ | ||
"#### Upload to Google Drive\n", | ||
"\n", | ||
"`upload_to_gdrive` function uploads the files to google drive. it takes two parameter\n", | ||
"\n", | ||
"`file_path` : path to the file to upload" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 3, | ||
"id": "633c9aba-58da-4dbc-8cd4-2f9594329609", | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"File Uploaded successful\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"upload_to_gdrive(file_path='Animated_timeseries.ipynb')" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "17f3f25c-d11f-4a7c-9061-ce1ba76ff103", | ||
"metadata": {}, | ||
"source": [ | ||
"***\n", | ||
"\n", | ||
"## Additional information\n", | ||
"\n", | ||
"**License:** The code in this notebook is licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). \n", | ||
"Digital Earth Africa data is licensed under the [Creative Commons by Attribution 4.0](https://creativecommons.org/licenses/by/4.0/) license.\n", | ||
"\n", | ||
"**Contact:** If you need assistance, please post a question on the [Open Data Cube Slack channel](http://slack.opendatacube.org/) or on the [GIS Stack Exchange](https://gis.stackexchange.com/questions/ask?tags=open-data-cube) using the `open-data-cube` tag (you can view previously asked questions [here](https://gis.stackexchange.com/questions/tagged/open-data-cube)).\n", | ||
"If you would like to report an issue with this notebook, you can file one on [Github](https://github.com/digitalearthafrica/deafrica-sandbox-notebooks).\n", | ||
"\n", | ||
"**Compatible datacube version:** " | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 4, | ||
"id": "1d478268-bbd0-4d12-a9a9-df514f88af1f", | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"1.8.15\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"import datacube\n", | ||
"print(datacube.__version__)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "7caaffb8-9aa9-4be9-a8ca-f5f5d9574811", | ||
"metadata": {}, | ||
"source": [ | ||
"**Last Tested:**" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 5, | ||
"id": "c74752ab-aa62-4813-ac81-14d5d3419839", | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"data": { | ||
"text/plain": [ | ||
"'2024-07-25'" | ||
] | ||
}, | ||
"execution_count": 5, | ||
"metadata": {}, | ||
"output_type": "execute_result" | ||
} | ||
], | ||
"source": [ | ||
"from datetime import datetime\n", | ||
"datetime.today().strftime('%Y-%m-%d')" | ||
] | ||
} | ||
], | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "Python 3 (ipykernel)", | ||
"language": "python", | ||
"name": "python3" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"name": "ipython", | ||
"version": 3 | ||
}, | ||
"file_extension": ".py", | ||
"mimetype": "text/x-python", | ||
"name": "python", | ||
"nbconvert_exporter": "python", | ||
"pygments_lexer": "ipython3", | ||
"version": "3.10.12" | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 5 | ||
} |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
Upload the json file downloaded from google service account here. | ||
Rename the download json file to `credentials.json`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,82 @@ | ||
import os.path | ||
import google.auth | ||
|
||
from google.oauth2.credentials import Credentials | ||
from googleapiclient.discovery import build | ||
from googleapiclient.errors import HttpError | ||
from googleapiclient.http import MediaFileUpload | ||
from google.oauth2 import service_account | ||
|
||
# If modifying these scopes, delete the file credentials. | ||
SCOPES = ["https://www.googleapis.com/auth/drive"] | ||
credential_path = '../Supplementary_data/DriveCredentials/credentials.json' | ||
|
||
|
||
def create_token(): | ||
'''' | ||
credential: provide the json creditials you would get from google service. | ||
''' | ||
creds = None | ||
creds = service_account.Credentials.from_service_account_file(credential_path, scopes=SCOPES) | ||
return creds | ||
|
||
def list_gdrive(): | ||
''' | ||
List the 10 recent files from the google drive | ||
''' | ||
creds = create_token() | ||
try: | ||
service = build("drive", "v3", credentials=creds) | ||
|
||
results = (service.files().list(pageSize=20, fields="nextPageToken, files(id, name)").execute()) | ||
items = results.get("files", []) | ||
|
||
if not items: | ||
print("No files found.") | ||
return | ||
print("Files:") | ||
for item in items: | ||
print(f"{item['name']} ({item['id']})") | ||
except HttpError as error: | ||
print(f"An error occurred: {error}") | ||
|
||
|
||
def upload_to_gdrive(file_path=None): | ||
''' | ||
Uploading files to google drive | ||
''' | ||
creds = create_token() | ||
try: | ||
# create drive api client | ||
service = build("drive", "v3", credentials=creds) | ||
folder_path = '../Supplementary_data/DriveCredentials/googledrive_id.txt' | ||
#read the first line of the file | ||
folder_id = open(folder_path, "r").readline() | ||
|
||
file_metadata = {"name": file_path, "parents": [folder_id]} | ||
media = MediaFileUpload(file_path, resumable=True) | ||
# pylint: disable=maybe-no-member | ||
file = (service.files().create(body=file_metadata, media_body=media).execute()) | ||
print('File Uploaded successful') | ||
except HttpError as error: | ||
print(f"An error occurred: {error}") | ||
file = None | ||
return | ||
|
||
|
||
def delete_to_gdrive(file_id=None): | ||
''' | ||
deleting file from google drive | ||
''' | ||
creds = create_token() | ||
try: | ||
# create drive api client | ||
service = build("drive", "v3", credentials=creds) | ||
|
||
# pylint: disable=maybe-no-member | ||
response = service.files().delete(fileId=file_id).execute() | ||
print('File deleted successful') | ||
except HttpError as error: | ||
print(f"An error occurred: {error}") | ||
file = None | ||
return |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters