04-train-test-pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Training and Validation Pipeline\n",
    "Now that you have created features, you can use them to train one or more models. In this section, you will generate feature vectors with multiple features from one or more feature sets and feed them into an automated ML training and testing pipeline to create high-quality models.\n",
    "\n",
    "The ML pipeline can be triggered and tracked manually during the interactive devel‐ opment, or it can be saved (into Git) and be executed automatically on a given schedule or as a reaction to different events (such as code modification, CI/CD, data changes, model drift, and so on). See [MLRun project and CI/CD documentation](https://docs.mlrun.org/en/stable/projects/project.html) for details.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving and loading projects from GIT\n",
    "\n",
    "After you saved your project and its elements (functions, workflows, artifacts, etc.) you can commit all your changes to a \n",
    "GIT repository. This can be done using standard GIT tools or using MLRun `project` methods such as `pull`, `push`, \n",
    "`remote`, which calls the Git API for you.\n",
    "\n",
    "Projects can then be loaded from Git using MLRun `load_project` method, for example: \n",
    "\n",
    "    project = mlrun.load_project(\"./myproj\", \"git://github.com/mlrun/project-demo.git\", name=project_name)\n",
    "    \n",
    "or using MLRun CLI:\n",
    "\n",
    "    mlrun project -n myproj -u \"git://github.com/mlrun/project-demo.git\" ./myproj\n",
    "    \n",
    "Projects can be loaded or created by using MLRun `get_or_create_project` method.\n",
    "    \n",
    "Read [CI/CD integration](../../projects/ci-integration.html) for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLRun installation and configuration\n",
    "Before running this notebook make sure the `mlrun` package is installed together with kfp (`pip install mlrun[kfp18]`) and that you have configured the access to MLRun service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install MLRun if not installed, run this only once (restart the notebook after the install !!!)\n",
    "%pip install mlrun[kfp18]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The history saving thread hit an unexpected error (DatabaseError('database disk image is malformed')).History will not be written to the database.\n",
      "> 2024-10-08 13:19:58,328 [info] Project loaded successfully: {\"project_name\":\"fraud-demo-felipe\"}\n"
     ]
    }
   ],
   "source": [
    "import mlrun\n",
    "project = mlrun.get_or_create_project(\n",
    "    name=\"fraud-demo\",\n",
    "    context=\"./\",\n",
    "    user_project=True,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating and Evaluating a Feature Vector\n",
    "\n",
    "Models are trained with multiple features, which can arrive from different feature sets and be collected into training (feature) vectors. Feature stores know how to correctly combine the features into a vector by implementing smart JOINs and assessing the time dimension (time traveling).\n",
    "To define a feature vector, you need to specify a name, the list of features it contains, the target features (labels), and other optional parameters. Features are specified as `<FeatureSet>.<Feature> or <FeatureSet>.*`  (all the features in a feature set). The following part demonstrates how to create and use a feature vector.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a feature vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import MLRun's Feature Store\n",
    "import mlrun.feature_store as fstore\n",
    "\n",
    "# Define the list of features to use\n",
    "features = ['events.*',\n",
    "            'transactions.amount_max_2h', \n",
    "            'transactions.amount_sum_2h', \n",
    "            'transactions.amount_count_2h',\n",
    "            'transactions.amount_avg_2h', \n",
    "            'transactions.amount_max_12h', \n",
    "            'transactions.amount_sum_12h',\n",
    "            'transactions.amount_count_12h', \n",
    "            'transactions.amount_avg_12h', \n",
    "            'transactions.amount_max_24h',\n",
    "            'transactions.amount_sum_24h', \n",
    "            'transactions.amount_count_24h', \n",
    "            'transactions.amount_avg_24h',\n",
    "            'transactions.es_transportation_sum_14d', \n",
    "            'transactions.es_health_sum_14d',\n",
    "            'transactions.es_otherservices_sum_14d', \n",
    "            'transactions.es_food_sum_14d',\n",
    "            'transactions.es_hotelservices_sum_14d', \n",
    "            'transactions.es_barsandrestaurants_sum_14d',\n",
    "            'transactions.es_tech_sum_14d', \n",
    "            'transactions.es_sportsandtoys_sum_14d',\n",
    "            'transactions.es_wellnessandbeauty_sum_14d', \n",
    "            'transactions.es_hyper_sum_14d',\n",
    "            'transactions.es_fashion_sum_14d', \n",
    "            'transactions.es_home_sum_14d', \n",
    "            'transactions.es_travel_sum_14d', \n",
    "            'transactions.es_leisure_sum_14d',\n",
    "            'transactions.gender_F',\n",
    "            'transactions.gender_M',\n",
    "            'transactions.step', \n",
    "            'transactions.amount', \n",
    "            'transactions.timestamp_hour',\n",
    "            'transactions.timestamp_day_of_week']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a feature vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the feature vector name for future reference\n",
    "fv_name = 'transactions-fraud'\n",
    "\n",
    "# Define the feature vector using the feature store (fstore)\n",
    "transactions_fv = fstore.FeatureVector(fv_name, \n",
    "                          features, \n",
    "                          label_feature=\"labels.label\",\n",
    "                          description='Predicting a fraudulent transaction')\n",
    "\n",
    "# Save the feature vector in the feature store\n",
    "transactions_fv.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2024-10-08 13:20:06,351 [info] Merger detected timestamp resolution incompatibility between feature set labels and others: datetime64[us] and datetime64[ms]. Converting feature set timestamp column 'timestamp' to type datetime64[us].\n",
      "> 2024-10-08 13:20:06,399 [info] wrote target: {'size': 151272, 'name': 'parquet', 'updated': '2024-10-08T13:20:06.399382+00:00', 'partitioned': True, 'path': 'v3io:///projects/fraud-demo-felipe/FeatureStore/transactions-fraud/parquet/vectors/transactions-fraud-latest.parquet', 'kind': 'parquet', 'status': 'ready'}\n"
     ]
    }
   ],
   "source": [
    "# test get_offline_features function\n",
    "\n",
    "from mlrun.datastore.targets import ParquetTarget\n",
    "data = transactions_fv.get_offline_features(target=ParquetTarget())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.feature_store.feature_vector.OfflineVectorResponse at 0x7f7deccdcfa0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>event_password_change</th>\n",
       "      <th>event_details_change</th>\n",
       "      <th>event_login</th>\n",
       "      <th>amount_max_2h</th>\n",
       "      <th>amount_sum_2h</th>\n",
       "      <th>amount_count_2h</th>\n",
       "      <th>amount_avg_2h</th>\n",
       "      <th>amount_max_12h</th>\n",
       "      <th>amount_sum_12h</th>\n",
       "      <th>amount_count_12h</th>\n",
       "      <th>...</th>\n",
       "      <th>es_home_sum_14d</th>\n",
       "      <th>es_travel_sum_14d</th>\n",
       "      <th>es_leisure_sum_14d</th>\n",
       "      <th>gender_F</th>\n",
       "      <th>gender_M</th>\n",
       "      <th>step</th>\n",
       "      <th>amount</th>\n",
       "      <th>timestamp_hour</th>\n",
       "      <th>timestamp_day_of_week</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.83</td>\n",
       "      <td>1.83</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.830000</td>\n",
       "      <td>1.83</td>\n",
       "      <td>1.83</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>1.83</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>18.72</td>\n",
       "      <td>40.22</td>\n",
       "      <td>3.0</td>\n",
       "      <td>13.406667</td>\n",
       "      <td>18.72</td>\n",
       "      <td>40.22</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>18.72</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25.92</td>\n",
       "      <td>64.86</td>\n",
       "      <td>3.0</td>\n",
       "      <td>21.620000</td>\n",
       "      <td>25.92</td>\n",
       "      <td>64.86</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>25.92</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24.75</td>\n",
       "      <td>30.17</td>\n",
       "      <td>2.0</td>\n",
       "      <td>15.085000</td>\n",
       "      <td>24.75</td>\n",
       "      <td>30.17</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>141.0</td>\n",
       "      <td>24.75</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>64.18</td>\n",
       "      <td>65.17</td>\n",
       "      <td>2.0</td>\n",
       "      <td>32.585000</td>\n",
       "      <td>64.18</td>\n",
       "      <td>65.17</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>124.0</td>\n",
       "      <td>64.18</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   event_password_change  event_details_change  event_login  amount_max_2h  \\\n",
       "0                      0                     0            1           1.83   \n",
       "1                      0                     0            1          18.72   \n",
       "2                      1                     0            0          25.92   \n",
       "3                      1                     0            0          24.75   \n",
       "4                      1                     0            0          64.18   \n",
       "\n",
       "   amount_sum_2h  amount_count_2h  amount_avg_2h  amount_max_12h  \\\n",
       "0           1.83              1.0       1.830000            1.83   \n",
       "1          40.22              3.0      13.406667           18.72   \n",
       "2          64.86              3.0      21.620000           25.92   \n",
       "3          30.17              2.0      15.085000           24.75   \n",
       "4          65.17              2.0      32.585000           64.18   \n",
       "\n",
       "   amount_sum_12h  amount_count_12h  ...  es_home_sum_14d  es_travel_sum_14d  \\\n",
       "0            1.83               1.0  ...              0.0                0.0   \n",
       "1           40.22               3.0  ...              0.0                0.0   \n",
       "2           64.86               3.0  ...              0.0                0.0   \n",
       "3           30.17               2.0  ...              0.0                0.0   \n",
       "4           65.17               2.0  ...              0.0                0.0   \n",
       "\n",
       "   es_leisure_sum_14d  gender_F  gender_M   step  amount  timestamp_hour  \\\n",
       "0                 0.0       0.0       1.0   72.0    1.83             7.0   \n",
       "1                 0.0       0.0       1.0   66.0   18.72             7.0   \n",
       "2                 0.0       0.0       1.0   27.0   25.92             7.0   \n",
       "3                 0.0       0.0       1.0  141.0   24.75             7.0   \n",
       "4                 0.0       1.0       0.0  124.0   64.18             7.0   \n",
       "\n",
       "   timestamp_day_of_week  label  \n",
       "0                    6.0    0.0  \n",
       "1                    6.0    0.0  \n",
       "2                    6.0    0.0  \n",
       "3                    6.0    0.0  \n",
       "4                    6.0    0.0  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.to_dataframe().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have defined the feature vector, you can use `get_offline_features()` to generate the vector dataset and return it as a dataframe or materialize it into a file (CSV or Parquet). The next part demonstrates how to retrieve a vector, materialize it, and view its results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building and Running an Automated Training and Validation Pipeline\n",
    "\n",
    "MLRun allows the building of distributed ML pipelines that can handle data processing, automated feature selection, training, optimization, testing, deployments, and so on. Pipelines are composed of steps that run or deploy custom or library (from the MLRun hub) serverless functions. Pipelines can be run locally (for debugging or small-scale tasks), on a scalable Kubernetes cluster (using Kubeflow), or in a CI/CD system.\n",
    "\n",
    "The example consists of the following pipeline steps (all using pre-defined MLRun hub functions):\n",
    "\n",
    "1. Materialize a feature vector (using `src/get_vector`). \n",
    "2. Select the most optimal features (using `hub://feature_selection`).\n",
    "3. Train the model with multiple algorithms (using `hub://auto_trainer`).\n",
    "4. Evaluate the model (using `hub://auto_trainer`).\n",
    "5. Deploy the model and its application to the test cluster (using `hub://v2_model_server`). The next section will explain the model and application pipeline in detail.\n",
    "\n",
    "Each step can accept the previous steps’ results or data, and generate results, multiple visual artifacts/charts, versioned data objects, and registered models.\n",
    "\n",
    "We have defined the workflow in [`src/new_train_workflow.py`](./src/new_train_workflow.py). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Running the ML pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The workflow/pipeline can be executed using the MLRun SDK (`project.run()` method) or using CLI commands (mlrun project), and can run directly from the source repo (GIT). See details in [MLRun Projects and Automation documentation](https://docs.mlrun.org/en/stable/projects/project.html).\n",
    "\n",
    "You can set arguments and destinations for the different artifacts when you run the workflow. The pipeline progress and results are shown in the notebook. Alternatively, you can check the progress, logs, artifacts, and more, in the MLRun UI or the CI/CD system. The next part demonstrates how to run the pipeline with custom arguments using the SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2024-10-08 13:20:13,879 [warning] WARNING!, You seem to have uncommitted git changes, use .push()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Missing type name was inferred as \"JsonArray\" based on the value \"[]\".\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2024-10-08 13:20:14,738 [info] Pipeline submitted successfully: {\"id\":\"e27f0d6d-8f14-4642-abee-43b9867631ef\",\"pipeline_name\":\"fraud-demo-felipe-main 2024-10-08 13-20-14\"}\n",
      "> 2024-10-08 13:20:14,738 [info] Pipeline run id=e27f0d6d-8f14-4642-abee-43b9867631ef, check UI for progress\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Workflow started in project fraud-demo-felipe id=e27f0d6d-8f14-4642-abee-43b9867631ef<div><a href=\"https://dashboard.default-tenant.app.cust-cs-illl--3-6-0.iguazio-cd2.com/mlprojects/fraud-demo-felipe/jobs/monitor-workflows/workflow/e27f0d6d-8f14-4642-abee-43b9867631ef\" target=\"_blank\">click here to view progress</a></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>Pipeline running (id=e27f0d6d-8f14-4642-abee-43b9867631ef), <a href=\"https://dashboard.default-tenant.app.cust-cs-illl--3-6-0.iguazio-cd2.com/mlprojects/fraud-demo-felipe/jobs/monitor-workflows/workflow/e27f0d6d-8f14-4642-abee-43b9867631ef\" target=\"_blank\"><b>click here</b></a> to view the details in MLRun UI</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: kfp Pages: 1 -->\n",
       "<svg width=\"8pt\" height=\"8pt\"\n",
       " viewBox=\"0.00 0.00 8.00 8.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 4)\">\n",
       "<title>kfp</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-4 4,-4 4,4 -4,4\"/>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f7e34b02370>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2024-10-08 13:20:14,810 [info] Started run workflow fraud-demo-felipe-main with run id = 'e27f0d6d-8f14-4642-abee-43b9867631ef' by kfp engine\n",
      "> 2024-10-08 13:20:14,811 [info] Waiting for pipeline run completion: {\"project\":\"<mlrun.projects.project.MlrunProject object at 0x7f7decca0a30>\",\"run_id\":\"e27f0d6d-8f14-4642-abee-43b9867631ef\"}\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: kfp Pages: 1 -->\n",
       "<svg width=\"248pt\" height=\"260pt\"\n",
       " viewBox=\"0.00 0.00 248.05 260.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 256)\">\n",
       "<title>kfp</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-256 244.05,-256 244.05,4 -4,4\"/>\n",
       "<!-- fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;2645070770 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;2645070770</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"50.05\" cy=\"-18\" rx=\"50.09\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.05\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">evaluate</text>\n",
       "</g>\n",
       "<!-- fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;3318924573 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;3318924573</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"114.05\" cy=\"-234\" rx=\"57.69\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.05\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">get&#45;vector</text>\n",
       "</g>\n",
       "<!-- fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;3653472374 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;3653472374</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"114.05\" cy=\"-162\" rx=\"89.08\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.05\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">feature&#45;selection</text>\n",
       "</g>\n",
       "<!-- fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;3318924573&#45;&gt;fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;3653472374 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;3318924573&#45;&gt;fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;3653472374</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M114.05,-215.7C114.05,-207.98 114.05,-198.71 114.05,-190.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"117.55,-190.1 114.05,-180.1 110.55,-190.1 117.55,-190.1\"/>\n",
       "</g>\n",
       "<!-- fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;67864529 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;67864529</title>\n",
       "<ellipse fill=\"green\" stroke=\"black\" cx=\"114.05\" cy=\"-90\" rx=\"33.29\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"114.05\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">train</text>\n",
       "</g>\n",
       "<!-- fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;3653472374&#45;&gt;fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;67864529 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;3653472374&#45;&gt;fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;67864529</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M114.05,-143.7C114.05,-135.98 114.05,-126.71 114.05,-118.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"117.55,-118.1 114.05,-108.1 110.55,-118.1 117.55,-118.1\"/>\n",
       "</g>\n",
       "<!-- fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;67864529&#45;&gt;fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;2645070770 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;67864529&#45;&gt;fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;2645070770</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M99.84,-73.46C91.63,-64.48 81.11,-52.98 71.87,-42.87\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"74.43,-40.49 65.1,-35.47 69.27,-45.21 74.43,-40.49\"/>\n",
       "</g>\n",
       "<!-- fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;922820315 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;922820315</title>\n",
       "<polygon fill=\"green\" stroke=\"black\" points=\"240.05,-36 122.05,-36 118.05,-32 118.05,0 236.05,0 240.05,-4 240.05,-36\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"236.05,-32 118.05,-32 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"236.05,-32 236.05,0 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"236.05,-32 240.05,-36 \"/>\n",
       "<text text-anchor=\"middle\" x=\"179.05\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">deploy&#45;serving</text>\n",
       "</g>\n",
       "<!-- fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;67864529&#45;&gt;fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;922820315 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;67864529&#45;&gt;fraud&#45;detection&#45;pipeline&#45;6l7vc&#45;922820315</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M128.47,-73.46C136.5,-64.82 146.68,-53.85 155.81,-44.03\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"158.59,-46.17 162.83,-36.46 153.46,-41.41 158.59,-46.17\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f7deca13fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h2>Run Results</h2><h3>[info] Workflow e27f0d6d-8f14-4642-abee-43b9867631ef finished, state=Succeeded</h3><br>click the hyper links below to see detailed results<br><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>uid</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>kind</th>\n",
       "      <th>name</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td><div title=\"82435092a42244458f09090df8fcc251\"><a href=\"https://dashboard.default-tenant.app.cust-cs-illl--3-6-0.iguazio-cd2.com/mlprojects/fraud-demo-felipe/jobs/monitor/82435092a42244458f09090df8fcc251/overview\" target=\"_blank\" >...f8fcc251</a></div></td>\n",
       "      <td>Oct 08 13:21:44</td>\n",
       "      <td>completed</td>\n",
       "      <td>run</td>\n",
       "      <td>evaluate</td>\n",
       "      <td><div class=\"dictlist\">label_columns=label</div><div class=\"dictlist\">model=store://artifacts/fraud-demo-felipe/transaction_fraud_rf:latest@e27f0d6d-8f14-4642-abee-43b9867631ef</div><div class=\"dictlist\">drop_columns=label</div></td>\n",
       "      <td><div class=\"dictlist\">evaluation_accuracy=0.9905</div><div class=\"dictlist\">evaluation_f1_score=0.17391304347826086</div><div class=\"dictlist\">evaluation_precision_score=0.4</div><div class=\"dictlist\">evaluation_recall_score=0.1111111111111111</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"4eef9c89992842cf90ca121a674058e1\"><a href=\"https://dashboard.default-tenant.app.cust-cs-illl--3-6-0.iguazio-cd2.com/mlprojects/fraud-demo-felipe/jobs/monitor/4eef9c89992842cf90ca121a674058e1/overview\" target=\"_blank\" >...674058e1</a></div></td>\n",
       "      <td>Oct 08 13:21:10</td>\n",
       "      <td>completed</td>\n",
       "      <td>run</td>\n",
       "      <td>train</td>\n",
       "      <td><div class=\"dictlist\">sample=-1</div><div class=\"dictlist\">label_column=label</div><div class=\"dictlist\">test_size=0.1</div></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=1</div><div class=\"dictlist\">accuracy=0.9905</div><div class=\"dictlist\">f1_score=0.17391304347826086</div><div class=\"dictlist\">precision_score=0.4</div><div class=\"dictlist\">recall_score=0.1111111111111111</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"f1054ceef99c4b85beb54850af1f9176\"><a href=\"https://dashboard.default-tenant.app.cust-cs-illl--3-6-0.iguazio-cd2.com/mlprojects/fraud-demo-felipe/jobs/monitor/f1054ceef99c4b85beb54850af1f9176/overview\" target=\"_blank\" >...af1f9176</a></div></td>\n",
       "      <td>Oct 08 13:20:42</td>\n",
       "      <td>completed</td>\n",
       "      <td>run</td>\n",
       "      <td>feature-selection</td>\n",
       "      <td><div class=\"dictlist\">output_vector_name=short</div><div class=\"dictlist\">label_column=label</div><div class=\"dictlist\">k=18</div><div class=\"dictlist\">min_votes=2</div><div class=\"dictlist\">ignore_type_errors=True</div></td>\n",
       "      <td><div class=\"dictlist\">top_features_vector=store://feature-vectors/fraud-demo-felipe/short</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"9d5ec830a46640eaae20e2cac1e2831f\"><a href=\"https://dashboard.default-tenant.app.cust-cs-illl--3-6-0.iguazio-cd2.com/mlprojects/fraud-demo-felipe/jobs/monitor/9d5ec830a46640eaae20e2cac1e2831f/overview\" target=\"_blank\" >...c1e2831f</a></div></td>\n",
       "      <td>Oct 08 13:20:21</td>\n",
       "      <td>completed</td>\n",
       "      <td>run</td>\n",
       "      <td>get-vector</td>\n",
       "      <td><div class=\"dictlist\">feature_vector=transactions-fraud</div><div class=\"dictlist\">features=['events.*', 'transactions.amount_max_2h', 'transactions.amount_sum_2h', 'transactions.amount_count_2h', 'transactions.amount_avg_2h', 'transactions.amount_max_12h', 'transactions.amount_sum_12h', 'transactions.amount_count_12h', 'transactions.amount_avg_12h', 'transactions.amount_max_24h', 'transactions.amount_sum_24h', 'transactions.amount_count_24h', 'transactions.amount_avg_24h', 'transactions.es_transportation_sum_14d', 'transactions.es_health_sum_14d', 'transactions.es_otherservices_sum_14d', 'transactions.es_food_sum_14d', 'transactions.es_hotelservices_sum_14d', 'transactions.es_barsandrestaurants_sum_14d', 'transactions.es_tech_sum_14d', 'transactions.es_sportsandtoys_sum_14d', 'transactions.es_wellnessandbeauty_sum_14d', 'transactions.es_hyper_sum_14d', 'transactions.es_fashion_sum_14d', 'transactions.es_home_sum_14d', 'transactions.es_travel_sum_14d', 'transactions.es_leisure_sum_14d', 'transactions.gender_F', 'transactions.gender_M', 'transactions.step', 'transactions.amount', 'transactions.timestamp_hour', 'transactions.timestamp_day_of_week']</div><div class=\"dictlist\">label_feature=labels.label</div><div class=\"dictlist\">target={'name': 'parquet', 'kind': 'parquet'}</div><div class=\"dictlist\">update_stats=True</div></td>\n",
       "      <td><div class=\"dictlist\">return=<mlrun.feature_store.feature_vector.OfflineVectorResponse object at 0x7f1aba100fa0></div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "run_id = project.run(\n",
    "    'main',\n",
    "    arguments={'vector_name':\"transactions-fraud\",\n",
    "               'features': features,\n",
    "                'label_column':\"labels.label\",\n",
    "              }, \n",
    "    dirty=True, watch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the model endpoint\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that your model is deployed using the pipeline, you can invoke it as usual:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2024-10-08 13:41:17,914 [info] Invoking function: {\"method\":\"POST\",\"path\":\"http://nuclio-fraud-demo-felipe-serving.default-tenant.svc.cluster.local:8080/v2/models/fraud/infer\"}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'id': 'c9b5036c-8957-48dc-adba-d7f60ccd7812',\n",
       " 'model_name': 'fraud',\n",
       " 'outputs': [0],\n",
       " 'timestamp': '2024-10-08 13:41:17.934699+00:00',\n",
       " 'model_version': 'latest'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define your serving function\n",
    "serving_fn = project.get_function('serving')\n",
    "\n",
    "# Choose an id for your test\n",
    "sample_id = 'C1000148617'\n",
    "model_inference_path = '/v2/models/fraud/infer'\n",
    "\n",
    "# Send our sample ID for predcition\n",
    "serving_fn.invoke(path=model_inference_path,\n",
    "                  body={'inputs': [[sample_id]]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Done!\n",
    "\n",
    "You've completed part 4 - the model training with the feature store.\n",
    "Proceed to [Part 5](06-real-time-serving-pipeline.ipynb) to learn how to deploy real-time application pipelines."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}