Hello World Module

A Helm Chart for an example Fybrik module

Introduction

This helm chart defines a common structure to deploy a Kubernetes job for an Fybrik module.

The configuration for the chart is in the values file.

Prerequisites

• Kubernetes cluster 1.10+
• Helm 3.7.x and above

Installation

Modify values in Makefile

In Makefile:

• Change DOCKER_USERNAME, DOCKER_PASSWORD, DOCKER_HOSTNAME, DOCKER_NAMESPACE, DOCKER_TAGNAME, DOCKER_NAME, HELM_TAG to your own preferences.

Build Docker image for Python application

make docker-build

Push Docker image to your preferred container registry

make docker-push

Configure the chart

• When testing the chart, configure settings by editing the values.yaml directly.
• Modify repository in values.yaml to your preferred Docker image.
• Modify copy/read action as needed with appropriate values.
• At runtime, the fybrik-manager will pass in the copy/read values to the module so you can leave them blank in your final chart.

Register as a Fybrik module

To register HWM (Hello World Module) as a Fybrik module apply hello-world-module.yaml to the fybrik-system namespace of your cluster.

To install the latest release run:

kubectl apply -f https://github.com/fybrik/hello-world-module/releases/latest/download/hello-world-module.yaml -n fybrik-system

Version compatbility matrix

Fybrik	HWM	Command
0.5.x	0.5.x	https://github.com/fybrik/hello-world-module/releases/download/v0.5.0/hello-world-module.yaml
0.6.x	0.6.x	https://github.com/fybrik/hello-world-module/releases/download/v0.6.0/hello-world-module.yaml
master	main	https://raw.githubusercontent.com/fybrik/hello-world-module/main/hello-world-module.yaml

Login to Helm registry

make helm-login

Lint and install Helm chart

make helm-verify

Push the Helm chart

make helm-chart-push

Uninstallation

make helm-uninstall

Deploy and test Fybrik module

Follow this section to deploy and test the module on a single cluster.

Before you begin

Install Fybrik using the Quick Start guide. This sample assumes the use of the built-in catalog, Open Policy Agent (OPA) and flight module.

Notice: Please follow version compatbility matrix section above for deploying the correct version of Fybrik and this module. For deploying older versions of hello-world-module please refer to the README.md file in the relevant release. For older version of Fybrik please follow the instructions in Fybrik site which match the release.

Deploy DataShim

Deploy datashim on the cluster. For deployment options based on your environment please refer to the datashim site.

Deploy Fybrik module

Deploy FybrikModule in fybrik-system namespace:

kubectl create -f hello-world-module.yaml -n fybrik-system

Test using Fybrik Notebook sample

1. Execute all the sections in Fybrik Notebook sample until Define data access policies section (excluded).

2. Deploy the follwing secret resource and FybrikStorageAccount resource from hack/test-module directory. Note that the module version should match the latter version. These resources are used by the Fybrik to allocate a new bucket for the copied resource.

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: bucket-creds
  namespace: fybrik-system
type: Opaque
stringData:
  access_key: "${ACCESS_KEY}"
  accessKeyID: "${ACCESS_KEY}"
  secret_key: "${SECRET_KEY}"
  secretAccessKey: "${SECRET_KEY}"
EOF

Define data access policies

Define the following OpenPolicyAgent policy to allow the write operation:

package dataapi.authz

description := "allow the write operation"
rule[{}] {
  input.action.actionType == "write"
  true
}

In this sample only the policy above is applied. Copy the policy to a file named sample-policy.rego and then run:

kubectl -n fybrik-system create configmap sample-policy --from-file=sample-policy.rego
kubectl -n fybrik-system label configmap sample-policy openpolicyagent.org/policy=rego
while [[ $(kubectl get cm sample-policy -n fybrik-system -o 'jsonpath={.metadata.annotations.openpolicyagent\.org/policy-status}') != '{"status":"ok"}' ]]; do echo "waiting for policy to be applied" && sleep 5; done

Deploy Fybrik application which triggers module

Deploy FybrikApplication in default namespace. the resource should be taken from hack/test-module directory. Note that the module version should match the fybrikapplication version.

3. Run the following command to wait until the status of the FybrikApplication is ready:

while [[ $(kubectl get fybrikapplication my-notebook -n default -o 'jsonpath={.status.ready}') != "true" ]]; do echo "waiting for FybrikApplication" && sleep 5; done

4. Check if module was triggered:

kubectl get blueprint -n fybrik-system
kubectl describe blueprint my-notebook-default -n fybrik-system
kubectl get job -n fybrik-blueprints
kubectl get pods -n fybrik-blueprints

If you are using the hello-world-module image, you should see this in the kubectl logs of your completed Pod:

$ kubectl logs my-notebook-default-copy-hello-world-module-chart-xxxx -n fybrik-blueprints

Hello World Module!

Connection name is paysim-csv

Connection format is csv

Vault credential address is http://vault.fybrik-system:8200

Vault credential role is module

Vault credential secret path is /v1/kubernetes-secrets/paysim-csv?namespace=fybrik-notebook-sample

S3 bucket is demo

S3 endpoint is http://localstack.fybrik-notebook-sample.svc.cluster.local:4566

COPY SUCCEEDED

Clean

Run the following command to delete the fybrik application:

kubectl delete FybrikApplication my-notebook -n default

Run the following command to delete the fybrik module:

kubectl delete fybrikmodule hello-world-module -n fybrik-system

Please execute the Cleanup section from Fybrik notebook sample