Upgrade to Python 3 (#210)

* try to upgrade python 3

Signed-off-by: Niket <[email protected]>

* remove login admin restriction to see if it works

Signed-off-by: Niket <[email protected]>

* add script auto

Signed-off-by: Niket <[email protected]>

* remove stringIO

Signed-off-by: Niket <[email protected]>

* use string IO in python 3

Signed-off-by: Niket <[email protected]>

* fix syntax error

Signed-off-by: Niket <[email protected]>

* use BytesIO in python3

Signed-off-by: Niket <[email protected]>

* Replace cStringIO import with io

* Import gaerpytz directly, update it and move files to lib

* Remove removal of _sass.so

* Remove reports

* Remove application_readable from app.yaml

* Replace skip-files with gcloudignore file

* Make app.yaml runtime Python 3.7

* Remove dot

* Remove mail/weekly (test)

* Use new VPC network

Signed-off-by: Niket <[email protected]>

* Merge branch 'python_3' of https://github.com/ocadotechnology/codeforlife-deploy-appengine into python_3

* require django

Signed-off-by: Niket <[email protected]>

* fix requirements.txt

Signed-off-by: Niket <[email protected]>

* put cfl apps in requirements file

Signed-off-by: Niket <[email protected]>

* install mysqlclient

Signed-off-by: Niket <[email protected]>

* change main.py to set django application variable to ‘app’

Signed-off-by: Niket <[email protected]>

* use double quotes for secrets

Signed-off-by: Niket <[email protected]>

* no build sh

Signed-off-by: Niket <[email protected]>

* install aimmo

Signed-off-by: Niket <[email protected]>

* put back build.sh for now

Signed-off-by: Niket <[email protected]>

* rearrange things in app yaml tmpl

Signed-off-by: Niket <[email protected]>

* test if removing the secret makes the other env variables work

Signed-off-by: Niket <[email protected]>

* double quotes

Signed-off-by: Niket <[email protected]>

* formatting

Signed-off-by: Niket <[email protected]>

* go back to single quotes

Signed-off-by: Niket <[email protected]>

* use handlers in the demo app to see if they were the ast issue

Signed-off-by: Niket Shah <[email protected]>

* Revert to single quotes again

* remove empty env variable

Signed-off-by: Niket Shah <[email protected]>

* use double quotes with single quotes with env var causing error

* add django secret back

Signed-off-by: Niket Shah <[email protected]>

* fix the “if on google appengine” check

Signed-off-by: Niket Shah <[email protected]>

* disable email backend for now

Signed-off-by: Niket Shah <[email protected]>

* use redis for caching

Signed-off-by: Niket Shah <[email protected]>

* fix syntax error in settings.py

Signed-off-by: Niket Shah <[email protected]>

* use normal string for redis location

Signed-off-by: Niket Shah <[email protected]>

* use normal string for redix location

Signed-off-by: Niket Shah <[email protected]>

* remove explicit mentions of our app in the requirements folder

use build.sh instead

Signed-off-by: Niket Shah <[email protected]>

* add apps in requirements back in

Signed-off-by: Niket Shah <[email protected]>

* remove quotes in the string

Signed-off-by: Niket <[email protected]>

* print module name env var

Signed-off-by: Niket Shah <[email protected]>

* use quotes again for ast.literal_eval

Signed-off-by: Niket Shah <[email protected]>

* Test removing lib-specific installs

* use GAE_SERVICE instead of DJANGO_MODULE_NAME

Signed-off-by: Niket <[email protected]>

* Merge branch 'python_3' of https://github.com/ocadotechnology/codeforlife-deploy-appengine into python_3

* Revert back to lib installs

* Merge branch 'python_3' of https://github.com/ocadotechnology/codeforlife-deploy-appengine into python_3

* Try single quotes on GAE_SERVICE

* Remove requirements.txt to see what the error is

* Try from lib import

* Add __init__.py to lib folder

* Bring back requirements

* Test to see if requirements generation works

* Trying lib again

* Remove last new line

* Add requirements.txt to .gitignore

* Fix versions for other packages

* Specify patch versions to test if fixes invalid argument

* Fix syntax error

* Remove echoes

* Comment out gaerpytz step to see if deployment still works

* Remove gaerpytz

* Remove requirements.txt that was added to git before gitignore

* Use Redis IP and PORT env vars

* Remove Nones

* Export env vars

* Declare variables in app.yaml.tmpl

* Setup Amazon SES

* Add comma in INSTALLED_APPS

* Add anymail to requirements

* Put anymail back in lib

* Add region to AWS config

* Fix format of region setting

* Move Anymail settings before email backend setting

* Move credentials to anymail dict

* Enable Debugger agents

* Try adding credentials outside of dict too

* Setup boto client with credentials

* Add env vars to app.yaml.tmpl

* Cleanup unnecessary code

* Ignore files and folders starting with .

* Remove EMAIL_QUEUE_NAME setting

Author: faucomte97

.gcloudignore

@@ -0,0 +1,10 @@
+.*
+*.scss
+*#
+*~
+*.pyc
+*.pyo
+lib/*info/**
+lib/PIL/**
+lib/setuptools/script (dev).tmpl
+lib/**/static/**

.gitignore

@@ -1,5 +1,5 @@
 app.yaml
 lib
 static
-
+requirements.txt
 \.idea/

app.yaml.tmpl

@@ -1,103 +1,38 @@
-runtime: python27
+runtime: python37
 instance_class: F4
-api_version: 1
-threadsafe: true
 service: ${MODULE_NAME}
 
-handlers:
-- url: /favicon\.ico
-  static_files: static/portal/img/favicon.ico
-  upload: static/portal/img/favicon\.ico
-
-- url: /robots\.txt
-  static_files: static/robots.txt
-  upload: static/robots\.txt
-
-- url: /apple-touch-icon\.png
-  static_files: static/apple-touch-icon.png
-  upload: static/apple-touch-icon\.png
-
-- url: /apple-touch-icon-76x76\.png
-  static_files: static/apple-touch-icon-76x76.png
-  upload: static/apple-touch-icon-76x76\.png
-
-- url: /apple-touch-icon-120x120\.png
-  static_files: static/apple-touch-icon-120x120.png
-  upload: static/apple-touch-icon-120x120\.png
-
-- url: /apple-touch-icon-152x152\.png
-  static_files: static/apple-touch-icon-152x152.png
-  upload: static/apple-touch-icon-152x152\.png
-
-- url: /apple-touch-icon-180x180\.png
-  static_files: static/apple-touch-icon-180x180.png
-  upload: static/apple-touch-icon-180x180\.png
-
-- url: /static
-  static_dir: static
-  application_readable: true
-  secure: always
-
-- url: /_ah/stats.*
-  script: google.appengine.ext.appstats.ui.app
-  secure: always
+vpc_access_connector:
+  name: "projects/decent-digit-629/locations/europe-west1/connectors/cfl-connector"
 
-- url: /mail/weekly
-  script: main.application
-  login: admin
-  secure: always
-
-- url: /aimmo/api/.*
-  script: main.application
-  secure: always
+handlers:
+  # This configures Google App Engine to serve the files in the app's static
+  # directory.
+  - url: /static
+    static_dir: static/
 
-- url: /api/.*
-  script: main.application
-  login: admin
-  secure: always
+  # This handler routes all requests not caught above to your main app. It is
+  # required when static routes are defined, but can be omitted (along with
+  # the entire handlers section) when there are no static files defined.
+  - url: /.*
+    script: auto
 
-- url: /.*
-  script: main.application
-  secure: always
+inbound_services:
+  - warmup
 
 env_variables:
   DJANGO_SETTINGS_MODULE: 'django_site.settings'
   HTTPS: 'on'
   CACHE_PREFIX: '${CACHE_PREFIX}'
   DATABASE_NAME: '${DATABASE_NAME}'
-  DJANGO_SECRET: '${DJANGO_SECRET}'
   RECAPTCHA_PRIVATE_KEY: '${RECAPTCHA_PRIVATE_KEY}'
   RECAPTCHA_PUBLIC_KEY: '${RECAPTCHA_PUBLIC_KEY}'
   DJANGO_PORTAL_CONTACT_FORM_EMAIL: '${DJANGO_PORTAL_CONTACT_FORM_EMAIL}'
-  DJANGO_CREATOR_AUTH_TOKEN: '${DJANGO_CREATOR_AUTH_TOKEN}'
   DJANGO_SALESFORCE_URL: '${DJANGO_SALESFORCE_URL}'
   DJANGO_SALESFORCE_OID: '${DJANGO_SALESFORCE_OID}'
   DJANGO_SALESFORCE_RT: '${DJANGO_SALESFORCE_RT}'
-  DJANGO_MODULE_NAME: '${MODULE_NAME}'
-
-libraries:
-- name: MySQLdb
-  version: latest
-- name: ssl
-  version: latest
-- name: PIL
-  version: latest
-
-builtins:
-- appstats: on
-
-inbound_services:
-- warmup
-
-skip_files:
-- ^(.*/)?.*\.scss$
-- ^(.*/)?#.*#$
-- ^(.*/)?.*~$
-- ^(.*/)?.*\.py[co]$
-- ^(.*/)?.*unity\/Development\/.*$
-- ^(.*/)?.*/RCS/.*$
-- ^(.*/)?\..*$
-- ^lib\/.*info$
-- ^lib\/PIL
-- ^lib\/setuptools\/script.*dev.*.tmpl
-- ^lib\/.*/static/.*$
+  DJANGO_SECRET: '"${DJANGO_SECRET}"'
+  REDIS_IP: '${REDIS_IP}'
+  REDIS_PORT: '${REDIS_PORT}'
+  AWS_ACCESS_KEY_ID: '${AWS_ACCESS_KEY_ID}'
+  AWS_SECRET_ACCESS_KEY: '${AWS_SECRET_ACCESS_KEY}'

build.sh

@@ -3,20 +3,20 @@
 export ENVIRONMENT="$1"
 
 rbenv rehash
-pip install beautifulsoup4
 pip install requests
 pip install -t lib requests-toolbelt
 
 pip install -t lib codeforlife-portal
+
+pip install -t lib django-anymail[amazon_ses]
+
 if [ "$ENVIRONMENT" = "default" ]
 then
     pip install -t lib --upgrade --no-deps aimmo
 else
     pip install -t lib --pre --upgrade --no-deps aimmo
 fi
 
-python install_gaerpytz.py
+python generate_requirements.py
 
 ./manage.py collectstatic --noinput
-
-rm ./lib/_sass.so

deploy.sh

@@ -23,7 +23,7 @@ export GOOGLE_APPLICATION_CREDENTIALS=/home/runner/codeforlife-deploy-appengine/
 
 # Install the dependencies for the following deploy script.
 # Kubernetes is a TEMPORARY solution. See issue 68.
-pip install kubernetes==5.0.0
+pip install kubernetes
 pip install pyyaml
 
 # Authenticate the cluster by updating kubeconfig.
@@ -51,7 +51,6 @@ envsubst <app.yaml.tmpl >app.yaml
 ${GCLOUD} app --quiet deploy app.yaml --project ${APP_ID} --version ${VERSION} --no-promote
 ${GCLOUD} app --quiet deploy cron.yaml --project ${APP_ID} --version ${VERSION} --no-promote
 
-
 # Test the site
 ./test.sh ${MODULE_NAME} ${VERSION}
 

django_site/settings.py

@@ -8,8 +8,6 @@
 https://docs.djangoproject.com/en/1.6/ref/settings/
 """
 
-from django.core.urlresolvers import reverse_lazy
-
 # Build paths inside the project like this: rel(rel_path)
 import os
 
@@ -31,10 +29,10 @@
 # Application definition
 
 INSTALLED_APPS = (
+    "anymail",
     "deploy",
     "portal",
     "captcha",
-    "reports",
     "game",
     #'djangocms_admin_style',  # for the admin skin. You **must** add 'djangocms_admin_style' in the list **before** 'django.contrib.admin'.
     "django.contrib.admin",
@@ -116,20 +114,30 @@
 
 ALLOWED_HOSTS = [".appspot.com", ".codeforlife.education"]
 
-if os.getenv("SERVER_SOFTWARE", "").startswith("Google App Engine"):
+ANYMAIL = {
+    "AMAZON_SES_CLIENT_PARAMS": {
+        "aws_access_key_id": os.getenv("AWS_ACCESS_KEY_ID"),
+        "aws_secret_access_key": os.getenv("AWS_SECRET_ACCESS_KEY"),
+        "region_name": "eu-west-1",
+    },
+}
+
+if os.getenv("GAE_APPLICATION", None):
     # Running on production App Engine, so use a Google Cloud SQL database.
     DATABASES = {
         "default": {
             "ENGINE": "django.db.backends.mysql",
-            "HOST": "/cloudsql/decent-digit-629:db",
+            "HOST": "/cloudsql/decent-digit-629:europe-west1:db",
             "NAME": os.getenv("DATABASE_NAME"),
             "USER": "root",
         }
     }
     CACHES = {
         "default": {
-            "BACKEND": "django.core.cache.backends.memcached.MemcachedCache",
+            "BACKEND": "django_redis.cache.RedisCache",
+            "LOCATION": f"redis://{os.getenv('REDIS_IP')}:{os.getenv('REDIS_PORT')}/0",
             "KEY_PREFIX": os.getenv("CACHE_PREFIX"),
+            "OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient"},
         }
     }
     PIPELINE_ENABLED = True
@@ -140,9 +148,7 @@
     if lib_path not in sys.path:
         sys.path.append(lib_path)
     # setup email on app engine
-    EMAIL_BACKEND = "deploy.mail.EmailBackend"
-    # Specify a queue name for the async. email backend.
-    EMAIL_QUEUE_NAME = "default"
+    EMAIL_BACKEND = "anymail.backends.amazon_ses.EmailBackend"
 
     SOCIAL_AUTH_PANDASSO_KEY = "code-for-life"
     SOCIAL_AUTH_PANDASSO_SECRET = os.getenv("PANDASSO_SECRET")
@@ -208,8 +214,9 @@
 
 CMS_TEMPLATES = (("portal/base.html", "Template One"),)
 
+
 AIMMO_GAME_SERVER_URL_FUNCTION = lambda game: (
-    os.getenv("DJANGO_MODULE_NAME") + "-aimmo.codeforlife.education",
+    f"{os.getenv('GAE_SERVICE')}-aimmo.codeforlife.education",
     "/game-%s" % game,
 )
 

django_site/urls.py

@@ -5,7 +5,6 @@
 from portal import urls as portal_urls
 from game import urls as game_urls
 from aimmo import urls as aimmo_urls
-from reports import urls as reports_urls
 
 js_info_dict = {"packages": ("conf.locale",)}
 
@@ -15,7 +14,6 @@
     url(r"^", include(portal_urls)),
     url(r"^administration/", include(admin.site.urls)),
     url(r"^rapidrouter/", include(game_urls)),
-    url(r"^reports/", include(reports_urls)),
     url(r"^kurono/", include(aimmo_urls)),
     url(r"^versions/$", versions, name="versions"),
 ]

django_site/wsgi.py

@@ -10,5 +10,5 @@
 import os
 os.environ.setdefault("DJANGO_SETTINGS_MODULE", "django_site.settings")
 
-from django.core.wsgi import get_wsgi_application
+from lib.django.core.wsgi import get_wsgi_application
 application = get_wsgi_application()

generate_requirements.py

@@ -0,0 +1,17 @@
+from lib.portal import __version__ as portal_version
+from lib.aimmo import __version__ as aimmo_version
+
+requirements = (
+    f"codeforlife-portal=={portal_version}\n"
+    f"aimmo=={aimmo_version}\n"
+    f"requests-toolbelt==0.9.*\n"
+    f"mysqlclient==1.4.*\n"
+    f"redis==3.3.*\n"
+    f"django-redis==4.11.*\n"
+    f"django-anymail[amazon_ses]==7.0.*"
+)
+
+requirements_path = "requirements.txt"
+requirements_file = open(requirements_path, "w")
+requirements_file.write(requirements)
+requirements_file.close()

install_gaerpytz.py

@@ -1,21 +1,6 @@
-import StringIO
-import zipfile
+import os
+import shutil
 
-import requests
-from bs4 import BeautifulSoup
-
-# Get HTML of the gaerpytz home page
-gaerpytz_url = "https://gaerpytz.appspot.com"
-gaerpytz_webpage = requests.get(gaerpytz_url)
-gaerpytz_html = gaerpytz_webpage.text
-
-# Parse HTML to get the zipfile's download link
-soup = BeautifulSoup(gaerpytz_html, "html.parser")
-zip_href = soup.find(title="Download the latest build").get("href")
-zip_url = gaerpytz_url + zip_href
-
-# Download the zipfile and extract it to the lib folder. The zipfile contains
-# the module folder for pytz and will overwrite the one currently in lib
-request = requests.get(zip_url, stream=True)
-zip_file = zipfile.ZipFile(StringIO.StringIO(request.content))
-zip_file.extractall("lib")
+# Copy over the files from gaerpytz directory to the one in the lib folder
+for pytz_file in os.listdir("gaerpytz"):
+    shutil.copy("gaerpytz/" + pytz_file, "lib/pytz")

lib/__init__.py

@@ -1,3 +1,3 @@
-import django_site.wsgi
+from django_site.wsgi import application
 
-application = django_site.wsgi.application
+app = application