Skip to content

Commit

Permalink
Merge pull request #518 from digitalearthafrica/gdrive
Browse files Browse the repository at this point in the history
Tools for gdrive
  • Loading branch information
nanaboamah89 authored Aug 29, 2024
2 parents a4671ef + ebcdc26 commit 70a630c
Show file tree
Hide file tree
Showing 6 changed files with 318 additions and 2 deletions.
232 changes: 232 additions & 0 deletions Frequently_used_code/Integrating_with_Google_Drive.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,232 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "89bb88ae-35d8-44c1-bbf3-9af915ac5086",
"metadata": {},
"source": [
"# Exporting to Google Drive from the Sandbox"
]
},
{
"cell_type": "raw",
"id": "36320092-0334-460c-ad65-fafc3d7ebad0",
"metadata": {},
"source": [
"**Keywords** :index:`package; google`"
]
},
{
"cell_type": "markdown",
"id": "7247cc3c-e692-477c-b626-82a2b13807aa",
"metadata": {},
"source": [
"## Background\n",
"\n",
"Google Drive API enables developers to integrate Google Drive functionality into applications, offering access to file storage, sharing, and synchronization capabilities. It allows users to manage files, folders, and permissions programmatically, facilitating seamless integration of cloud storage features into various software solutions. The API supports multiple programming languages and provides robust documentation, making it versatile for a wide range of applications from document management to collaborative platforms. More information can be found [here](https://developers.google.com/drive/api/guides/about-sdk).\n",
"\n",
"To connect to the Google Drive API from the Sandbox using a service account, a `service account` is created in the Google Cloud Console. Users then need to download its JSON key file, and configure permissions on both Google Drive and the service account. This approach bypasses user authentication and allows server-to-server interactions, making it suitable for automated tasks or applications where user interaction is not required. The service account JSON key is securely stored and used by Python code within Jupyter notebooks to authenticate and authorize API requests, enabling seamless integration of Google Drive functionalities directly into data science workflows without user intervention. \n",
"\n",
"Detailed instructions on integrating Google Drive are [documented in the DE Africa user guide](https://docs.digitalearthafrica.org/en/latest/platform_tools/googledrive_access.html).\n",
"\n",
"**Security considerations:**\n",
"Users should take care to ensure the security of credentials uploaded into the Sandbox, including by ensuring they are not incidentally included in uploads to public repositories like GitHub."
]
},
{
"cell_type": "markdown",
"id": "b9b67d58-9fb6-4a8a-8fb0-4105d33afafd",
"metadata": {},
"source": [
"## Description\n",
"\n",
"This notebook provides a brief introduction to accessing and using Digital Earth Africa's sandbox with Google Drive:\n",
"\n",
"1. The packages to load up\n",
"2. How to use the list files in the google drive\n",
"3. How to upload file files in the google drive\n",
"\n",
"***"
]
},
{
"cell_type": "markdown",
"id": "8a42fd34-7c4e-4d58-b914-41216d9f018a",
"metadata": {},
"source": [
"## Getting started\n",
"\n",
">##### To use this notebook, make sure to follow these steps [here](https://docs.digitalearthafrica.org/en/latest/platform_tools/googledrive_access.html) before proceeding to the next cell. \n"
]
},
{
"cell_type": "markdown",
"id": "21bba818-ee4d-4031-876e-edf42492c6b5",
"metadata": {},
"source": [
"### Load packages\n",
"Import Python packages that are used for the analysis."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "aa0cf6dc-d5d4-42b3-8e6e-bb3bc676a908",
"metadata": {},
"outputs": [],
"source": [
"from deafrica_tools.externaldrive import list_gdrive, upload_to_gdrive"
]
},
{
"cell_type": "markdown",
"id": "1e9f3df4-620b-4ab0-b06b-16ae8e8a3d5b",
"metadata": {},
"source": [
"#### List Files in Google Drive\n",
"\n",
"`list_gdrive` function list the files and folder within the service account drive."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "160bfce9-1183-4259-9a61-ecf66a103094",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Files:\n",
"Animated_timeseries.ipynb (19oBKPC8BKN77d45STpk1xAmA8_2gxYLO)\n",
"deafricaout (1Cn3e0m8T0OPjZ4dZOnoeXBiGrNiuLo0Y)\n"
]
}
],
"source": [
"list_gdrive()"
]
},
{
"cell_type": "markdown",
"id": "b84357e8-912c-4b0c-afdf-89d95df7022a",
"metadata": {
"tags": []
},
"source": [
"#### Upload to Google Drive\n",
"\n",
"`upload_to_gdrive` function uploads the files to google drive. it takes two parameter\n",
"\n",
"`file_path` : path to the file to upload"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "633c9aba-58da-4dbc-8cd4-2f9594329609",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"File Uploaded successful\n"
]
}
],
"source": [
"upload_to_gdrive(file_path='Animated_timeseries.ipynb')"
]
},
{
"cell_type": "markdown",
"id": "17f3f25c-d11f-4a7c-9061-ce1ba76ff103",
"metadata": {},
"source": [
"***\n",
"\n",
"## Additional information\n",
"\n",
"**License:** The code in this notebook is licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). \n",
"Digital Earth Africa data is licensed under the [Creative Commons by Attribution 4.0](https://creativecommons.org/licenses/by/4.0/) license.\n",
"\n",
"**Contact:** If you need assistance, please post a question on the [Open Data Cube Slack channel](http://slack.opendatacube.org/) or on the [GIS Stack Exchange](https://gis.stackexchange.com/questions/ask?tags=open-data-cube) using the `open-data-cube` tag (you can view previously asked questions [here](https://gis.stackexchange.com/questions/tagged/open-data-cube)).\n",
"If you would like to report an issue with this notebook, you can file one on [Github](https://github.com/digitalearthafrica/deafrica-sandbox-notebooks).\n",
"\n",
"**Compatible datacube version:** "
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "1d478268-bbd0-4d12-a9a9-df514f88af1f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.8.15\n"
]
}
],
"source": [
"import datacube\n",
"print(datacube.__version__)"
]
},
{
"cell_type": "markdown",
"id": "7caaffb8-9aa9-4be9-a8ca-f5f5d9574811",
"metadata": {},
"source": [
"**Last Tested:**"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "c74752ab-aa62-4813-ac81-14d5d3419839",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'2024-07-25'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from datetime import datetime\n",
"datetime.today().strftime('%Y-%m-%d')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Empty file.
2 changes: 2 additions & 0 deletions Supplementary_data/DriveCredentials/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Upload the json file downloaded from google service account here.
Rename the download json file to `credentials.json`.
2 changes: 1 addition & 1 deletion Tools/deafrica_tools/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
__locales__ = __path__[0] + '/locales'

__version__ = '2.4.4'
__version__ = '2.4.5'

def set_lang(lang=None):
if lang is None:
Expand Down
82 changes: 82 additions & 0 deletions Tools/deafrica_tools/externaldrive.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
import os.path
import google.auth

from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from googleapiclient.http import MediaFileUpload
from google.oauth2 import service_account

# If modifying these scopes, delete the file credentials.
SCOPES = ["https://www.googleapis.com/auth/drive"]
credential_path = '../Supplementary_data/DriveCredentials/credentials.json'


def create_token():
''''
credential: provide the json creditials you would get from google service.
'''
creds = None
creds = service_account.Credentials.from_service_account_file(credential_path, scopes=SCOPES)
return creds

def list_gdrive():
'''
List the 10 recent files from the google drive
'''
creds = create_token()
try:
service = build("drive", "v3", credentials=creds)

results = (service.files().list(pageSize=20, fields="nextPageToken, files(id, name)").execute())
items = results.get("files", [])

if not items:
print("No files found.")
return
print("Files:")
for item in items:
print(f"{item['name']} ({item['id']})")
except HttpError as error:
print(f"An error occurred: {error}")


def upload_to_gdrive(file_path=None):
'''
Uploading files to google drive
'''
creds = create_token()
try:
# create drive api client
service = build("drive", "v3", credentials=creds)
folder_path = '../Supplementary_data/DriveCredentials/googledrive_id.txt'
#read the first line of the file
folder_id = open(folder_path, "r").readline()

file_metadata = {"name": file_path, "parents": [folder_id]}
media = MediaFileUpload(file_path, resumable=True)
# pylint: disable=maybe-no-member
file = (service.files().create(body=file_metadata, media_body=media).execute())
print('File Uploaded successful')
except HttpError as error:
print(f"An error occurred: {error}")
file = None
return


def delete_to_gdrive(file_id=None):
'''
deleting file from google drive
'''
creds = create_token()
try:
# create drive api client
service = build("drive", "v3", credentials=creds)

# pylint: disable=maybe-no-member
response = service.files().delete(fileId=file_id).execute()
print('File deleted successful')
except HttpError as error:
print(f"An error occurred: {error}")
file = None
return
2 changes: 1 addition & 1 deletion Tools/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "deafrica-tools"
version = "2.4.4"
version = "2.4.5"
# reflect version changes in deafrica_tools/__init__.py

description = "Functions and algorithms for analysing Digital Earth Africa data."
Expand Down

0 comments on commit 70a630c

Please sign in to comment.