azureml-module-get-started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build AML Pipeline with azureml modules\n",
    "\n",
    "In this tutorial you will learn how to work with Azure ML module:\n",
    "\n",
    "1. Setup enrivonment - install module CLI and module/pipeline SDK\n",
    "2. Register a few sample modules into your aml workspace using CLI\n",
    "3. Use module/pipeline SDK to create a pipeline with modules registered in step 2\n",
    "\n",
    "## Prerequisite\n",
    "* Install Azure CLI, please follow [the Azure CLI installation instructions](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) to install."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup environment\n",
    "* Install Azure CLI AML extension which includes the _module_ command group\n",
    "* Install Azure ML SDK including the APIs to work with _module_ and _pipeline_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uninstall azure-cli-ml (the `az ml` commands)\n",
    "!az extension remove -n azure-cli-ml \n",
    "\n",
    "# Install local version of azure-cli-ml (which includes `az ml module` commands)\n",
    "!az extension add --source https://azuremlsdktestpypi.azureedge.net/CLI-SDK-Runners-Validation/15510101/azure_cli_ml-0.1.0.15510101-py3-none-any.whl --pip-extra-index-urls https://azuremlsdktestpypi.azureedge.net/CLI-SDK-Runners-Validation/15510101 --yes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify the availability of `az ml module` commands\n",
    "#!az ml pipeline -h\n",
    "!az ml module -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [],
   "source": [
    "# Install azureml-sdk with Pipeline, Module\n",
    "# Important! After install succeed, need to restart kernel\n",
    "\n",
    "%config IPCompleter.greedy=True \n",
    "!pip install azureml-pipeline-wrapper[notebooks]==0.1.0.15510101 --extra-index-url https://azuremlsdktestpypi.azureedge.net/CLI-SDK-Runners-Validation/15510101 --user --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register azureml module\n",
    "\n",
    "You can manage AML module through [azure-cli-ml](https://aka.ms/moduledoc) or [ml.azure.com](https://ml.azure.com/). <br>\n",
    "\n",
    "Module could be registered from:\n",
    "- local path\n",
    "- public Github url\n",
    "- Azure DevOps build artifacts\n",
    "\n",
    "Azureml module support multiple module type:\n",
    "- Basic python module\n",
    "- Mpi module\n",
    "- Parallel run module\n",
    "- Hdi module\n",
    "- Windows module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you need to configure your ws information here\n",
    "\n",
    "subscription_id = '74eccef0-4b8d-4f83-b5f9-fa100d155b22'\n",
    "workspace_name = 'lisal-amlservice'\n",
    "resource_group = 'lisal-dev'\n",
    "\n",
    "# Specify available aml compute in workspace\n",
    "pipeline_compute = \"always-on-ds2v2\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure your aml workspace \n",
    "\n",
    "!az login \n",
    "!az account set -s $subscription_id \n",
    "!az ml folder attach -w $workspace_name -g $resource_group \n",
    "\n",
    "# Configure global .amlignore, it's designed for register module from local development environment\n",
    "# !az configure --defaults module_amlignore_file=./.amlignore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register azureml modules from github url\n",
    "\n",
    "!az ml module register --spec-file=https://github.com/lisagreenview/hello-aml-modules/blob/master/train-score-eval/mpi_train.yaml --set-as-default-version\n",
    "!az ml module register --spec-file=https://github.com/lisagreenview/hello-aml-modules/blob/master/train-score-eval/score.yaml --set-as-default-version\n",
    "!az ml module register --spec-file=https://github.com/lisagreenview/hello-aml-modules/blob/master/train-score-eval/eval.yaml --set-as-default-version\n",
    "!az ml module register --spec-file=https://github.com/lisagreenview/hello-aml-modules/blob/master/train-score-eval/compare2.yaml --set-as-default-version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list available custom module in aml workspace\n",
    "!az ml module list -o table "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create pipeline\n",
    "You can build pipeline through SDK experience, or drag-n-drop way through [Designer](https://ml.azure.com/visualinterface?wsid=/subscriptions/74eccef0-4b8d-4f83-b5f9-fa100d155b22/resourcegroups/kubeflow-demo/workspaces/kubeflow_ws_1&flight=cm,nml,newGraphDetail,newGraphAuthoring,all&tid=72f988bf-86f1-41af-91ab-2d7cd011db47) in workspace portal\n",
    "\n",
    "The new SDK:\n",
    "* Symplified the syntax to provide consistent experience with drag-n-drop\n",
    "* Support intellisense and docstring, free you to work with dict all the time\n",
    "* Support creating a pipeline with unpublished module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Workspace, Run, Dataset\n",
    "from azureml.pipeline.wrapper import Pipeline, Module, dsl\n",
    "\n",
    "ws = Workspace.get(name=workspace_name, subscription_id=subscription_id, resource_group=resource_group)\n",
    "\n",
    "# get modules\n",
    "train_module_func = Module.load(ws, namespace='microsoft.com/aml/samples', name='MPI Train')\n",
    "score_module_func = Module.load(ws, namespace='microsoft.com/aml/samples', name='Score')\n",
    "eval_module_func = Module.load(ws, namespace='microsoft.com/aml/samples', name='Evaluate')\n",
    "compare_module_func = Module.load(ws, namespace='microsoft.com/aml/samples', name='Compare 2 Models')\n",
    "\n",
    "# if you have unpublished module in local or github, below function allow user to test as anounymous module\n",
    "# compare_module_func = Module.from_yaml(ws, yaml_file='./CompareModdels/compare2.yaml')\n",
    "# compare_module_func = Module.from_yaml(ws, yaml_file='https://github.com/lisagreenview/hello-aml-modules/blob/master/train-score-eval/compare2.yaml')\n",
    "\n",
    "# get dataset\n",
    "training_data_name = 'aml_module_training_data'\n",
    "test_data_name = 'aml_module_test_data'\n",
    "\n",
    "if training_data_name not in ws.datasets:\n",
    "    print('Registering a training dataset for sample pipeline ...')\n",
    "    train_data = Dataset.File.from_files(path=['https://dprepdata.blob.core.windows.net/demo/Titanic.csv'])\n",
    "    train_data.register(workspace = ws, \n",
    "                              name = training_data_name, \n",
    "                              description = 'Training data (just for illustrative purpose)')\n",
    "    print('Registerd')\n",
    "else:\n",
    "    train_data = ws.datasets[training_data_name]\n",
    "    print('Training dataset found in workspace')\n",
    "\n",
    "if test_data_name not in ws.datasets:\n",
    "    print('Registering a test dataset for sample pipeline ...')\n",
    "    test_data = Dataset.File.from_files(path=['https://dprepdata.blob.core.windows.net/demo/Titanic.csv'])\n",
    "    test_data.register(workspace = ws, \n",
    "                          name = test_data_name, \n",
    "                          description = 'Test data (just for illustrative purpose)')\n",
    "    print('Registered')\n",
    "else:\n",
    "    test_data = ws.datasets[test_data_name]    \n",
    "    print('Test dataset found in workspace')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = train_module_func()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.runsettings.configure(target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### dsl pipeline \n",
    "* 'Pipeline parameter' is exposed as pipeline function input parameter\n",
    "* Pipeline output is the return of pipeline function\n",
    "\n",
    "### module function\n",
    "* module input can be set through set_inputs() or module initialization function\n",
    "* module parameter can be set through set_parameter() or module initialization function\n",
    "* module runsetting including compute, datastore, data mode and other runtime parameter are set through runsettings.configure()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a sub pipeline\n",
    "@dsl.pipeline(name = 'A sub pipeline including train/score/eval', \n",
    "              description = 'train model and evaluate model perf')\n",
    "def training_pipeline(input_data, test_data, learning_rate):\n",
    "    train = train_module_func(\n",
    "        training_data=input_data, \n",
    "        max_epochs=5, \n",
    "        learning_rate=learning_rate)\n",
    "    \n",
    "# or\n",
    "#    train = train_module_func()\n",
    "#    train.set_inputs(training_data=input_data).set_parameters(learning_rate=learning_rate, max_epochs=5)\n",
    "#\n",
    "\n",
    "    train.runsettings.configure(process_count_per_node = 2, node_count = 2)\n",
    "\n",
    "    score = score_module_func(\n",
    "        model_input=train.outputs.model_output, \n",
    "        test_data=test_data)\n",
    "\n",
    "    eval = eval_module_func(scoring_result=score.outputs.score_output)\n",
    "    \n",
    "    return {'eval_output': eval.outputs.eval_output, 'model_output': train.outputs.model_output}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define pipeline with sub pipeline\n",
    "@dsl.pipeline(name = 'A dummy pipeline that trains multiple models and output the best one', \n",
    "              description = 'select best model trained with different learning rate',\n",
    "              default_compute_target = pipeline_compute)\n",
    "def dummy_automl_pipeline(input_data, test_data):\n",
    "    train_and_evalute_model1 = training_pipeline(input_data, test_data, 0.01)\n",
    "    train_and_evalute_model2 = training_pipeline(input_data, test_data, 0.02)\n",
    "    \n",
    "    compare = compare_module_func(\n",
    "        model1=train_and_evalute_model1.outputs.model_output, \n",
    "        eval_result1=train_and_evalute_model1.outputs.eval_output,\n",
    "        model2=train_and_evalute_model2.outputs.model_output,\n",
    "        eval_result2=train_and_evalute_model2.outputs.eval_output\n",
    "    )\n",
    "\n",
    "    return {**compare.outputs}\n",
    "\n",
    "# create a pipeline\n",
    "pipeline = dummy_automl_pipeline(train_data, test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# validate pipeline and visualize the graph\n",
    "pipeline.validate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save as a draft\n",
    "pipeline.save(experiment_name = 'pipeline-with-azureml-module')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pipeline parameter can be override when submit pipeline\n",
    "run = pipeline.submit(experiment_name='pipeline-with-azureml-module', tags={'mode':'module-SDK'}, pipeline_parameters={'input_data':test_data,'test_data':train_data})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run.wait_for_completion()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.6 64-bit ('amlenv': conda)",
   "language": "python",
   "name": "python37664bitamlenvconda1709d228b5634784bc90b1e7f8c3afa4"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}