CKANEXT Canada

Government of Canada CKAN Extension - Extension à CKAN du Gouvernement du Canada

Table of Contents
Requirements
Installation
Plugins
Configurations
SOLR
Localization
Migrations
Data Flows

Requirements

Compatibility with core CKAN versions:

CKAN version	Compatible?
2.6 and earlier	no
2.7	no
2.8	no
2.9	no
2.10	yes
2.11	no

Compatibility with Python versions:

Python version	Compatible?
2.7 and earlier	no
3.7 and later	yes

Required extensions, forks, and branches:

• CKAN Fork (canada-v2.10 branch)
• CKANAPI
• CKANEXT Recombinant
• CKANEXT Fluent
• CKANEXT Scheming
• CKANEXT Security Fork (canada-v2.10 branch)
• CKANEXT Validation Fork (canada-v2.10 branch)
• Frictionless-py Fork (canada-v2.10 branch)
• CKANEXT XLoader Fork (canada-v2.10 branch)
• CKANEXT CloudStorage Fork (canada-v2.10 branch)
• CKANEXT DCAT Fork (canada-v2.10 branch)
• CKANEXT Power BI View
• CKANEXT Open API View
• CKANEXT GC Notify

Installation

To install ckanext-canada:

1. Activate your CKAN virtual environment, for example:

   . /usr/lib/ckan/default/bin/activate

2. Clone the source and install it on the virtualenv:

git clone https://github.com/open-data/ckanext-canada.git
cd ckanext-canada
pip install -e .
  pip install -r requirements.txt
python setup.py develop

3. Add the plugin entry points to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/ckan.ini).

Plugins

Theme

canada_theme adds templates, template helpers, and webassets . This plugin should load first.

Forms

canada_forms adds dataset and resource forms and extra blueprint functionality for them. This should load after canada_theme but before the other canada plugins

Public

canada_public adds actions, logic, and functionality specific to the Canada data portal.

Internal

canada_internal adds actions, logic, and functionality specific to the Canada data registry.

Datasets

canada_datasets extends the ckanext-scheming plugin, modifying specific functionality for the Canada data portal.

Security

canada_security extends the ckanext-security plugin, modifying specific functionality for the Canada data portal.

Configurations

Portal

The CKAN ini file needs the following settings for the portal:

ckan.plugins = canada_theme
               activity
               dcat
               dcat_json_interface
               canada_forms
               canada_public
               canada_datasets
               scheming_organizations
               fluent
               recombinant
               cloudstorage
               canada_security
               datastore
               text_view
               image_view
               datatables_view
               webpage_view
               openapi_view
               power_bi_view

Registry

The CKAN ini file needs the following settings for the registry:

ckan.plugins =  canada_theme
                activity
                validation
                canada_forms
                canada_internal
                canada_public
                recombinant
                datastore
                dsaudit
                canada_datasets
                scheming_organizations
                fluent
                cloudstorage
                canada_security
                xloader
                datatables_view
                image_view
                text_view
                webpage_view
                openapi_view
                power_bi_view
                gcnotify

General

Both applications need at least the following:

ckanext.power_bi.internal_i18n = true

recombinant.definitions = ckanext.canada:tables/ati.yaml
                          ckanext.canada:tables/briefingt.yaml
                          ckanext.canada:tables/qpnotes.yaml
                          ckanext.canada:tables/contracts.yaml
                          ckanext.canada:tables/contractsa.yaml
                          ckanext.canada:tables/grants.yaml
                          ckanext.canada:tables/hospitalityq.yaml
                          ckanext.canada:tables/reclassification.yaml
                          ckanext.canada:tables/travela.yaml
                          ckanext.canada:tables/travelq.yaml
                          ckanext.canada:tables/wrongdoing.yaml
                          ckanext.canada:tables/inventory.yaml
                          ckanext.canada:tables/consultations.yaml
                          ckanext.canada:tables/service.yaml
                          ckanext.canada:tables/dac.yaml
                          ckanext.canada:tables/nap5.yaml
                          ckanext.canada:tables/experiment.yaml
                          ckanext.canada:tables/adminaircraft.yaml

recombinant.tables = ckanext.canada:recombinant_tables.yaml

scheming.dataset_schemas =
    ckanext.canada:schemas/dataset.yaml
    ckanext.canada:schemas/info.yaml
    ckanext.canada:schemas/prop.yaml

scheming.presets =
    ckanext.scheming:presets.json
    ckanext.fluent:presets.json
    ckanext.canada:schemas/presets.yaml
    ckanext.validation:presets.json

scheming.organization_schemas = ckanext.canada:schemas/organization.yaml

ckanext.csrf_filter.same_site = Strict
ckanext.csrf_filter.exempt_rules = ^/datatable.*
ckanext.xloader.clean_datastore_tables = True
ckanext.validation.clean_validation_reports = True
ckan.feeds.pretty = True
ckan.feeds.include_private = True
ckan.auth.create_dataset_if_not_in_organization = false
ckan.activity_streams_email_notifications = false
ckan.record_private_activity = True
ckan.activity_streams_enabled = True
ckan.csrf_protection.ignore_extensions = False

licenses_group_url = file://<path to this extension>/ckanext/canada/public/static/licenses.json

ckanext.canada.datastore_source_domain_allow_list = canada.ca www.canada.ca

ckan.locales_offered = en fr

SOLR

This extension uses a custom Solr schema based on the CKAN 2.10 schema. Overwrite the default CKAN Solr schema with this one in order to enable search faceting over custom metadata fields.

You will need to rebuild your search index using:

ckan -c <INI> search-index rebuild

Localization

To update strings in the translation files:

python setup.py extract_messages

Extract messages will gather gettext calls in Python, JS, and Jinja2 files. It will also use th custom PD extractor to get specific strings for the Recombinant YAML files.

To update the English and French catalog files:

python setup.py update_catalog

This will update both English and French PO files. You will need to confirm that there are NO fuzzy translations in either of the PO files.

After updating the PO files and ensuring that there are no fuzzies, you may commit the two PO files along with the POT file.

Compiling localization strings

Each time you install or update this extension you need to install the updated translations by running:

python setup.py compile_catalog

Migrations

Creating a plugin migration

If adding or modifying custom database models in the CKAN framework, you can create a migration following the docs.

After making the migration, you can apply it with:

ckan -c <INI> db upgrade --plugin=<PLUGIN NAME>

or (execute twice):

ckan -c <INI> db pending-migrations --apply
ckan -c <INI> db pending-migrations --apply

Manually reloading Proactive Disclosure Data

This needs to be done if the data types change in the schema. The data will need to be exported, deleted from the database, and then reloaded so the new datatypes are created properly in the database.

1. Extract the current version of the data from the registry for each table, e.g for contracts migrations we need contracts.csv and contracts-nil.csv:

mkdir migrate-contracts  # a new working directory
ckan -c <INI> recombinant combine contracts contracts-nil -d migrate-contracts

2. Remove the old tables from the database. Deleting contracts will delete both the contracts and contracts-nil tables:

ckan -c <INI> recombinant delete contracts

3. Reload the data

cd migrate-contracts
ckan -c <INI> recombinant load-csv contracts.csv contracts-nil.csv

Advanced data migrations

Any advanced data migrations should be created in the DataOps repo and should adopt the concept that the Python script runs independatly from this plugin code and CKAN application context.

Data Flows

Open Data Flow

The custom command ckan -c <INI> canada portal-update is a worker function that copies datasets, resources, views, and datastore tables from the Registry to the Portal. This relies on the activity plugin and the capability to record Private activity.

1. Datasets that are "ready to publish" and have a "published date" are proccessed into the canada copy-datasets command;
2. Gathers resources, views, and datastore tables from the Registry;
3. If there are differences between the Registry and Portal dataset/resources, then copies the Registry data to the Portal (including dumping the Registry datastore table into the Portal datastore table).

The Django Search App uses an action API endpoint from the activity plugin to add/update/remove datasets from the Django Search Index.

Proactive Disclosure Data Flow

1. ckanext-canada (this repository)
   • PD yaml files are read by ckanext-recombinant and used to generate most of the pages, tables, triggers and metadata shown.
   • add+edit forms use form snippets from ckanext-scheming and validation enforced by datastore triggers. They are currently part of the ckanext-canada extension but should be moved into ckanext-recombinant or another reusable extension once the trigger-validation pattern becomes standardized
   • datatable preview is part of ckanext-canada because this code predates the datatables view feature that is now part of ckan. It should be removed from here so we can use the ckan datatable view instead
   • filter scripts cover all the business logic required to "clean" PD data before it is released to the public. a Makefile is used to extract raw CSV data, make backups, run these filters and publish CSV data
2. ckanext-recombinant
   • XLSX data dictionary
   • reference lists
   • API docs
   • schema json
   • delete form
   • XLSX template UL/DL
   • combine command
3. ckan
   • datastore API
   • datastore triggers
   • datastore tables
   • dataset metadata
4. CSV files
   • raw CSV data
   • nightly backups
   • published CSV data
5. deplane
   • data element profile
6. oc_search
   • advanced search