c07-data-analysis-with-python/res/2-Review-Data-Wrangling.ipynb

{
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "name": "python",
      "display_name": "Pyolite",
      "language": "python"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "python",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8"
    }
  },
  "nbformat_minor": 4,
  "nbformat": 4,
  "cells": [
    {
      "cell_type": "markdown",
      "source": "<center>\n    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DA0101EN-SkillsNetwork/labs/Module%202/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n</center>\n\n# Data Wrangling\n\nEstimated time needed: **30** minutes\n\n## Objectives\n\nAfter completing this lab you will be able to:\n\n*   Handle missing values\n*   Correct data format\n*   Standardize and normalize data\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h2>Table of Contents</h2>\n\n<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n<ul>\n    <li><a href=\"https://#identify_handle_missing_values\">Identify and handle missing values</a>\n        <ul>\n            <li><a href=\"https://#identify_missing_values\">Identify missing values</a></li>\n            <li><a href=\"https://#deal_missing_values\">Deal with missing values</a></li>\n            <li><a href=\"https://#correct_data_format\">Correct data format</a></li>\n        </ul>\n    </li>\n    <li><a href=\"https://#data_standardization\">Data standardization</a></li>\n    <li><a href=\"https://#data_normalization\">Data normalization (centering/scaling)</a></li>\n    <li><a href=\"https://#binning\">Binning</a></li>\n    <li><a href=\"https://#indicator\">Indicator variable</a></li>\n</ul>\n\n</div>\n\n<hr>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h2>What is the purpose of data wrangling?</h2>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "Data wrangling is the process of converting data from the initial format to a format that may be better for analysis.\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h3>What is the fuel consumption (L/100k) rate for the diesel car?</h3>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h3>Import data</h3>\n<p>\nYou can find the \"Automobile Dataset\" from the following link: <a href=\"https://archive.ics.uci.edu/ml/machine-learning-databases/autos/imports-85.data?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDA0101ENSkillsNetwork20235326-2021-01-01\">https://archive.ics.uci.edu/ml/machine-learning-databases/autos/imports-85.data</a>. \nWe will be using this dataset throughout this course.\n</p>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Import pandas</h4> \n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "you are running the lab in your  browser, so we will install the libraries using `piplite`\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "import piplite\nawait piplite.install(['pandas'])\nawait piplite.install(['matplotlib'])",
      "metadata": {
        "trusted": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "If you run the lab locally using Anaconda, you can load the correct library and versions by uncommenting the following:\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "#If you run the lab locally using Anaconda, you can load the correct library and versions by uncommenting the following:\n#install specific version of libraries used in lab\n#! mamba install pandas==1.3.3\n#! mamba install numpy=1.21.2\n",
      "metadata": {
        "trusted": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": "import pandas as pd\nimport matplotlib.pylab as plt",
      "metadata": {
        "trusted": true
      },
      "execution_count": 1,
      "outputs": [
        {
          "name": "stderr",
          "text": "/lib/python3.9/site-packages/pandas/compat/__init__.py:124: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.\n  warnings.warn(msg)\n",
          "output_type": "stream"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "This function will download the dataset into your browser\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "#This function will download the dataset into your browser \nfrom pyodide.http import pyfetch\n\nasync def download(url, filename):\n    response = await pyfetch(url)\n    if response.status == 200:\n        with open(filename, \"wb\") as f:\n            f.write(await response.bytes())\n\n",
      "metadata": {
        "trusted": true
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "<h2>Reading the dataset from the URL and adding the related headers</h2>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "First, we assign the URL of the dataset to \"filename\".\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "This dataset was hosted on IBM Cloud object. Click <a href=\"https://cocl.us/corsera_da0101en_notebook_bottom?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDA0101ENSkillsNetwork20235326-2021-01-01\">HERE</a> for free storage.\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "filename = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DA0101EN-SkillsNetwork/labs/Data%20files/auto.csv\"",
      "metadata": {
        "trusted": true
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "Then, we create a Python list <b>headers</b> containing name of headers.\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "headers = [\"symboling\",\"normalized-losses\",\"make\",\"fuel-type\",\"aspiration\", \"num-of-doors\",\"body-style\",\n         \"drive-wheels\",\"engine-location\",\"wheel-base\", \"length\",\"width\",\"height\",\"curb-weight\",\"engine-type\",\n         \"num-of-cylinders\", \"engine-size\",\"fuel-system\",\"bore\",\"stroke\",\"compression-ratio\",\"horsepower\",\n         \"peak-rpm\",\"city-mpg\",\"highway-mpg\",\"price\"]",
      "metadata": {
        "trusted": true
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "you will need to download the dataset; if you are running locally, please comment out the following\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "await download(filename, \"auto.csv\")\nfilename=\"auto.csv\"",
      "metadata": {
        "trusted": true
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "Use the Pandas method <b>read_csv()</b> to load the data from the web address. Set the parameter  \"names\" equal to the Python list \"headers\".\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "\ndf = pd.read_csv(filename, names = headers)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "Use the method <b>head()</b> to display the first five rows of the dataframe.\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "# To see what the data set looks like, we'll use the head() method.\ndf.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 7,
      "outputs": [
        {
          "execution_count": 7,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   symboling normalized-losses         make fuel-type aspiration num-of-doors  \\\n0          3                 ?  alfa-romero       gas        std          two   \n1          3                 ?  alfa-romero       gas        std          two   \n2          1                 ?  alfa-romero       gas        std          two   \n3          2               164         audi       gas        std         four   \n4          2               164         audi       gas        std         four   \n\n    body-style drive-wheels engine-location  wheel-base  ...  engine-size  \\\n0  convertible          rwd           front        88.6  ...          130   \n1  convertible          rwd           front        88.6  ...          130   \n2    hatchback          rwd           front        94.5  ...          152   \n3        sedan          fwd           front        99.8  ...          109   \n4        sedan          4wd           front        99.4  ...          136   \n\n   fuel-system  bore  stroke compression-ratio horsepower  peak-rpm city-mpg  \\\n0         mpfi  3.47    2.68               9.0        111      5000       21   \n1         mpfi  3.47    2.68               9.0        111      5000       21   \n2         mpfi  2.68    3.47               9.0        154      5000       19   \n3         mpfi  3.19    3.40              10.0        102      5500       24   \n4         mpfi  3.19    3.40               8.0        115      5500       18   \n\n  highway-mpg  price  \n0          27  13495  \n1          27  16500  \n2          26  16500  \n3          30  13950  \n4          22  17450  \n\n[5 rows x 26 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>symboling</th>\n      <th>normalized-losses</th>\n      <th>make</th>\n      <th>fuel-type</th>\n      <th>aspiration</th>\n      <th>num-of-doors</th>\n      <th>body-style</th>\n      <th>drive-wheels</th>\n      <th>engine-location</th>\n      <th>wheel-base</th>\n      <th>...</th>\n      <th>engine-size</th>\n      <th>fuel-system</th>\n      <th>bore</th>\n      <th>stroke</th>\n      <th>compression-ratio</th>\n      <th>horsepower</th>\n      <th>peak-rpm</th>\n      <th>city-mpg</th>\n      <th>highway-mpg</th>\n      <th>price</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>3</td>\n      <td>?</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>130</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000</td>\n      <td>21</td>\n      <td>27</td>\n      <td>13495</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3</td>\n      <td>?</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>130</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000</td>\n      <td>21</td>\n      <td>27</td>\n      <td>16500</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>?</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>hatchback</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>94.5</td>\n      <td>...</td>\n      <td>152</td>\n      <td>mpfi</td>\n      <td>2.68</td>\n      <td>3.47</td>\n      <td>9.0</td>\n      <td>154</td>\n      <td>5000</td>\n      <td>19</td>\n      <td>26</td>\n      <td>16500</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>fwd</td>\n      <td>front</td>\n      <td>99.8</td>\n      <td>...</td>\n      <td>109</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>10.0</td>\n      <td>102</td>\n      <td>5500</td>\n      <td>24</td>\n      <td>30</td>\n      <td>13950</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>4wd</td>\n      <td>front</td>\n      <td>99.4</td>\n      <td>...</td>\n      <td>136</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>8.0</td>\n      <td>115</td>\n      <td>5500</td>\n      <td>18</td>\n      <td>22</td>\n      <td>17450</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 26 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "As we can see, several question marks appeared in the dataframe; those are missing values which may hinder our further analysis.\n\n<div>So, how do we identify all those missing values and deal with them?</div> \n\n<b>How to work with missing data?</b>\n\nSteps for working with missing data:\n\n<ol>\n    <li>Identify missing data</li>\n    <li>Deal with missing data</li>\n    <li>Correct data format</li>\n</ol>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h2 id=\"identify_handle_missing_values\">Identify and handle missing values</h2>\n\n<h3 id=\"identify_missing_values\">Identify missing values</h3>\n<h4>Convert \"?\" to NaN</h4>\nIn the car dataset, missing data comes with the question mark \"?\".\nWe replace \"?\" with NaN (Not a Number), Python's default missing value marker for reasons of computational speed and convenience. Here we use the function: \n <pre>.replace(A, B, inplace = True) </pre>\nto replace A by B.\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "import numpy as np\n\n# replace \"?\" to NaN\ndf.replace(\"?\", np.nan, inplace = True)\ndf.head(5)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 8,
      "outputs": [
        {
          "execution_count": 8,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   symboling normalized-losses         make fuel-type aspiration num-of-doors  \\\n0          3               NaN  alfa-romero       gas        std          two   \n1          3               NaN  alfa-romero       gas        std          two   \n2          1               NaN  alfa-romero       gas        std          two   \n3          2               164         audi       gas        std         four   \n4          2               164         audi       gas        std         four   \n\n    body-style drive-wheels engine-location  wheel-base  ...  engine-size  \\\n0  convertible          rwd           front        88.6  ...          130   \n1  convertible          rwd           front        88.6  ...          130   \n2    hatchback          rwd           front        94.5  ...          152   \n3        sedan          fwd           front        99.8  ...          109   \n4        sedan          4wd           front        99.4  ...          136   \n\n   fuel-system  bore  stroke compression-ratio horsepower  peak-rpm city-mpg  \\\n0         mpfi  3.47    2.68               9.0        111      5000       21   \n1         mpfi  3.47    2.68               9.0        111      5000       21   \n2         mpfi  2.68    3.47               9.0        154      5000       19   \n3         mpfi  3.19    3.40              10.0        102      5500       24   \n4         mpfi  3.19    3.40               8.0        115      5500       18   \n\n  highway-mpg  price  \n0          27  13495  \n1          27  16500  \n2          26  16500  \n3          30  13950  \n4          22  17450  \n\n[5 rows x 26 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>symboling</th>\n      <th>normalized-losses</th>\n      <th>make</th>\n      <th>fuel-type</th>\n      <th>aspiration</th>\n      <th>num-of-doors</th>\n      <th>body-style</th>\n      <th>drive-wheels</th>\n      <th>engine-location</th>\n      <th>wheel-base</th>\n      <th>...</th>\n      <th>engine-size</th>\n      <th>fuel-system</th>\n      <th>bore</th>\n      <th>stroke</th>\n      <th>compression-ratio</th>\n      <th>horsepower</th>\n      <th>peak-rpm</th>\n      <th>city-mpg</th>\n      <th>highway-mpg</th>\n      <th>price</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>3</td>\n      <td>NaN</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>130</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000</td>\n      <td>21</td>\n      <td>27</td>\n      <td>13495</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3</td>\n      <td>NaN</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>130</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000</td>\n      <td>21</td>\n      <td>27</td>\n      <td>16500</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>NaN</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>hatchback</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>94.5</td>\n      <td>...</td>\n      <td>152</td>\n      <td>mpfi</td>\n      <td>2.68</td>\n      <td>3.47</td>\n      <td>9.0</td>\n      <td>154</td>\n      <td>5000</td>\n      <td>19</td>\n      <td>26</td>\n      <td>16500</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>fwd</td>\n      <td>front</td>\n      <td>99.8</td>\n      <td>...</td>\n      <td>109</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>10.0</td>\n      <td>102</td>\n      <td>5500</td>\n      <td>24</td>\n      <td>30</td>\n      <td>13950</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>4wd</td>\n      <td>front</td>\n      <td>99.4</td>\n      <td>...</td>\n      <td>136</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>8.0</td>\n      <td>115</td>\n      <td>5500</td>\n      <td>18</td>\n      <td>22</td>\n      <td>17450</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 26 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Evaluating for Missing Data</h4>\n\nThe missing values are converted by default. We use the following functions to identify these missing values. There are two methods to detect missing data:\n\n<ol>\n    <li><b>.isnull()</b></li>\n    <li><b>.notnull()</b></li>\n</ol>\nThe output is a boolean value indicating whether the value that is passed into the argument is in fact missing data.\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "missing_data = df.isnull()\nmissing_data.head(5)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 9,
      "outputs": [
        {
          "execution_count": 9,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   symboling  normalized-losses   make  fuel-type  aspiration  num-of-doors  \\\n0      False               True  False      False       False         False   \n1      False               True  False      False       False         False   \n2      False               True  False      False       False         False   \n3      False              False  False      False       False         False   \n4      False              False  False      False       False         False   \n\n   body-style  drive-wheels  engine-location  wheel-base  ...  engine-size  \\\n0       False         False            False       False  ...        False   \n1       False         False            False       False  ...        False   \n2       False         False            False       False  ...        False   \n3       False         False            False       False  ...        False   \n4       False         False            False       False  ...        False   \n\n   fuel-system   bore  stroke  compression-ratio  horsepower  peak-rpm  \\\n0        False  False   False              False       False     False   \n1        False  False   False              False       False     False   \n2        False  False   False              False       False     False   \n3        False  False   False              False       False     False   \n4        False  False   False              False       False     False   \n\n   city-mpg  highway-mpg  price  \n0     False        False  False  \n1     False        False  False  \n2     False        False  False  \n3     False        False  False  \n4     False        False  False  \n\n[5 rows x 26 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>symboling</th>\n      <th>normalized-losses</th>\n      <th>make</th>\n      <th>fuel-type</th>\n      <th>aspiration</th>\n      <th>num-of-doors</th>\n      <th>body-style</th>\n      <th>drive-wheels</th>\n      <th>engine-location</th>\n      <th>wheel-base</th>\n      <th>...</th>\n      <th>engine-size</th>\n      <th>fuel-system</th>\n      <th>bore</th>\n      <th>stroke</th>\n      <th>compression-ratio</th>\n      <th>horsepower</th>\n      <th>peak-rpm</th>\n      <th>city-mpg</th>\n      <th>highway-mpg</th>\n      <th>price</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>False</td>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>False</td>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>False</td>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 26 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "\"True\" means the value is a missing value while \"False\" means the value is not a missing value.\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Count missing values in each column</h4>\n<p>\nUsing a for loop in Python, we can quickly figure out the number of missing values in each column. As mentioned above, \"True\" represents a missing value and \"False\" means the value is present in the dataset.  In the body of the for loop the method \".value_counts()\" counts the number of \"True\" values. \n</p>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "for column in missing_data.columns.values.tolist():\n    print(column)\n    print (missing_data[column].value_counts())\n    print(\"\")    ",
      "metadata": {
        "trusted": true
      },
      "execution_count": 10,
      "outputs": [
        {
          "name": "stdout",
          "text": "symboling\nFalse    205\nName: symboling, dtype: int64\n\nnormalized-losses\nFalse    164\nTrue      41\nName: normalized-losses, dtype: int64\n\nmake\nFalse    205\nName: make, dtype: int64\n\nfuel-type\nFalse    205\nName: fuel-type, dtype: int64\n\naspiration\nFalse    205\nName: aspiration, dtype: int64\n\nnum-of-doors\nFalse    203\nTrue       2\nName: num-of-doors, dtype: int64\n\nbody-style\nFalse    205\nName: body-style, dtype: int64\n\ndrive-wheels\nFalse    205\nName: drive-wheels, dtype: int64\n\nengine-location\nFalse    205\nName: engine-location, dtype: int64\n\nwheel-base\nFalse    205\nName: wheel-base, dtype: int64\n\nlength\nFalse    205\nName: length, dtype: int64\n\nwidth\nFalse    205\nName: width, dtype: int64\n\nheight\nFalse    205\nName: height, dtype: int64\n\ncurb-weight\nFalse    205\nName: curb-weight, dtype: int64\n\nengine-type\nFalse    205\nName: engine-type, dtype: int64\n\nnum-of-cylinders\nFalse    205\nName: num-of-cylinders, dtype: int64\n\nengine-size\nFalse    205\nName: engine-size, dtype: int64\n\nfuel-system\nFalse    205\nName: fuel-system, dtype: int64\n\nbore\nFalse    201\nTrue       4\nName: bore, dtype: int64\n\nstroke\nFalse    201\nTrue       4\nName: stroke, dtype: int64\n\ncompression-ratio\nFalse    205\nName: compression-ratio, dtype: int64\n\nhorsepower\nFalse    203\nTrue       2\nName: horsepower, dtype: int64\n\npeak-rpm\nFalse    203\nTrue       2\nName: peak-rpm, dtype: int64\n\ncity-mpg\nFalse    205\nName: city-mpg, dtype: int64\n\nhighway-mpg\nFalse    205\nName: highway-mpg, dtype: int64\n\nprice\nFalse    201\nTrue       4\nName: price, dtype: int64\n\n",
          "output_type": "stream"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "Based on the summary above, each column has 205 rows of data and seven of the columns containing missing data:\n\n<ol>\n    <li>\"normalized-losses\": 41 missing data</li>\n    <li>\"num-of-doors\": 2 missing data</li>\n    <li>\"bore\": 4 missing data</li>\n    <li>\"stroke\" : 4 missing data</li>\n    <li>\"horsepower\": 2 missing data</li>\n    <li>\"peak-rpm\": 2 missing data</li>\n    <li>\"price\": 4 missing data</li>\n</ol>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h3 id=\"deal_missing_values\">Deal with missing data</h3>\n<b>How to deal with missing data?</b>\n\n<ol>\n    <li>Drop data<br>\n        a. Drop the whole row<br>\n        b. Drop the whole column\n    </li>\n    <li>Replace data<br>\n        a. Replace it by mean<br>\n        b. Replace it by frequency<br>\n        c. Replace it based on other functions\n    </li>\n</ol>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "Whole columns should be dropped only if most entries in the column are empty. In our dataset, none of the columns are empty enough to drop entirely.\nWe have some freedom in choosing which method to replace data; however, some methods may seem more reasonable than others. We will apply each method to many different columns:\n\n<b>Replace by mean:</b>\n\n<ul>\n    <li>\"normalized-losses\": 41 missing data, replace them with mean</li>\n    <li>\"stroke\": 4 missing data, replace them with mean</li>\n    <li>\"bore\": 4 missing data, replace them with mean</li>\n    <li>\"horsepower\": 2 missing data, replace them with mean</li>\n    <li>\"peak-rpm\": 2 missing data, replace them with mean</li>\n</ul>\n\n<b>Replace by frequency:</b>\n\n<ul>\n    <li>\"num-of-doors\": 2 missing data, replace them with \"four\". \n        <ul>\n            <li>Reason: 84% sedans is four doors. Since four doors is most frequent, it is most likely to occur</li>\n        </ul>\n    </li>\n</ul>\n\n<b>Drop the whole row:</b>\n\n<ul>\n    <li>\"price\": 4 missing data, simply delete the whole row\n        <ul>\n            <li>Reason: price is what we want to predict. Any data entry without price data cannot be used for prediction; therefore any row now without price data is not useful to us</li>\n        </ul>\n    </li>\n</ul>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Calculate the mean value for the \"normalized-losses\" column </h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "avg_norm_loss = df[\"normalized-losses\"].astype(\"float\").mean(axis=0)\nprint(\"Average of normalized-losses:\", avg_norm_loss)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 11,
      "outputs": [
        {
          "name": "stdout",
          "text": "Average of normalized-losses: 122.0\n",
          "output_type": "stream"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Replace \"NaN\" with mean value in \"normalized-losses\" column</h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df[\"normalized-losses\"].replace(np.nan, avg_norm_loss, inplace=True)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Calculate the mean value for the \"bore\" column</h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "avg_bore=df['bore'].astype('float').mean(axis=0)\nprint(\"Average of bore:\", avg_bore)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 13,
      "outputs": [
        {
          "name": "stdout",
          "text": "Average of bore: 3.3297512437810943\n",
          "output_type": "stream"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Replace \"NaN\" with the mean value in the \"bore\" column</h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df[\"bore\"].replace(np.nan, avg_bore, inplace=True)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "<div class=\"alert alert-danger alertdanger\" style=\"margin-top: 20px\">\n<h1> Question  #1: </h1>\n\n<b>Based on the example above, replace NaN in \"stroke\" column with the mean value.</b>\n\n</div>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "# Write your code below and press Shift+Enter to execute \navg_stroke = df[\"stroke\"].astype(\"float\").mean(axis=0)\ndf[\"stroke\"].replace(np.nan, avg_stroke, inplace=True)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "<details><summary>Click here for the solution</summary>\n\n```python\n#Calculate the mean vaule for \"stroke\" column\navg_stroke = df[\"stroke\"].astype(\"float\").mean(axis = 0)\nprint(\"Average of stroke:\", avg_stroke)\n\n# replace NaN by mean value in \"stroke\" column\ndf[\"stroke\"].replace(np.nan, avg_stroke, inplace = True)\n```\n\n</details>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Calculate the mean value for the \"horsepower\" column</h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "avg_horsepower = df['horsepower'].astype('float').mean(axis=0)\nprint(\"Average horsepower:\", avg_horsepower)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 23,
      "outputs": [
        {
          "name": "stdout",
          "text": "Average horsepower: 104.25615763546799\n",
          "output_type": "stream"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Replace \"NaN\" with the mean value in the \"horsepower\" column</h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df['horsepower'].replace(np.nan, avg_horsepower, inplace=True)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Calculate the mean value for \"peak-rpm\" column</h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "avg_peakrpm=df['peak-rpm'].astype('float').mean(axis=0)\nprint(\"Average peak rpm:\", avg_peakrpm)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 25,
      "outputs": [
        {
          "name": "stdout",
          "text": "Average peak rpm: 5125.369458128079\n",
          "output_type": "stream"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Replace \"NaN\" with the mean value in the \"peak-rpm\" column</h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df['peak-rpm'].replace(np.nan, avg_peakrpm, inplace=True)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "To see which values are present in a particular column, we can use the \".value_counts()\" method:\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df['num-of-doors'].value_counts()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 27,
      "outputs": [
        {
          "execution_count": 27,
          "output_type": "execute_result",
          "data": {
            "text/plain": "four    114\ntwo      89\nName: num-of-doors, dtype: int64"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "We can see that four doors are the **most common** type. We can also use the \".idxmax()\" method to calculate the most common type automatically:\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df['num-of-doors'].value_counts().idxmax()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 28,
      "outputs": [
        {
          "execution_count": 28,
          "output_type": "execute_result",
          "data": {
            "text/plain": "'four'"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "The replacement procedure is very similar to what we have seen previously:\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "#replace the missing 'num-of-doors' values by the most frequent \ndf[\"num-of-doors\"].replace(np.nan, \"four\", inplace=True)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "Finally, let's drop all rows that do not have price data:\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "# simply drop whole row with NaN in \"price\" column\ndf.dropna(subset=[\"price\"], axis=0, inplace=True)\n\n# reset index, because we droped two rows\ndf.reset_index(drop=True, inplace=True)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": "df.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 31,
      "outputs": [
        {
          "execution_count": 31,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   symboling normalized-losses         make fuel-type aspiration num-of-doors  \\\n0          3             122.0  alfa-romero       gas        std          two   \n1          3             122.0  alfa-romero       gas        std          two   \n2          1             122.0  alfa-romero       gas        std          two   \n3          2               164         audi       gas        std         four   \n4          2               164         audi       gas        std         four   \n\n    body-style drive-wheels engine-location  wheel-base  ...  engine-size  \\\n0  convertible          rwd           front        88.6  ...          130   \n1  convertible          rwd           front        88.6  ...          130   \n2    hatchback          rwd           front        94.5  ...          152   \n3        sedan          fwd           front        99.8  ...          109   \n4        sedan          4wd           front        99.4  ...          136   \n\n   fuel-system  bore  stroke compression-ratio horsepower  peak-rpm city-mpg  \\\n0         mpfi  3.47    2.68               9.0        111      5000       21   \n1         mpfi  3.47    2.68               9.0        111      5000       21   \n2         mpfi  2.68    3.47               9.0        154      5000       19   \n3         mpfi  3.19    3.40              10.0        102      5500       24   \n4         mpfi  3.19    3.40               8.0        115      5500       18   \n\n  highway-mpg  price  \n0          27  13495  \n1          27  16500  \n2          26  16500  \n3          30  13950  \n4          22  17450  \n\n[5 rows x 26 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>symboling</th>\n      <th>normalized-losses</th>\n      <th>make</th>\n      <th>fuel-type</th>\n      <th>aspiration</th>\n      <th>num-of-doors</th>\n      <th>body-style</th>\n      <th>drive-wheels</th>\n      <th>engine-location</th>\n      <th>wheel-base</th>\n      <th>...</th>\n      <th>engine-size</th>\n      <th>fuel-system</th>\n      <th>bore</th>\n      <th>stroke</th>\n      <th>compression-ratio</th>\n      <th>horsepower</th>\n      <th>peak-rpm</th>\n      <th>city-mpg</th>\n      <th>highway-mpg</th>\n      <th>price</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>3</td>\n      <td>122.0</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>130</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000</td>\n      <td>21</td>\n      <td>27</td>\n      <td>13495</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3</td>\n      <td>122.0</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>130</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000</td>\n      <td>21</td>\n      <td>27</td>\n      <td>16500</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>122.0</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>hatchback</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>94.5</td>\n      <td>...</td>\n      <td>152</td>\n      <td>mpfi</td>\n      <td>2.68</td>\n      <td>3.47</td>\n      <td>9.0</td>\n      <td>154</td>\n      <td>5000</td>\n      <td>19</td>\n      <td>26</td>\n      <td>16500</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>fwd</td>\n      <td>front</td>\n      <td>99.8</td>\n      <td>...</td>\n      <td>109</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>10.0</td>\n      <td>102</td>\n      <td>5500</td>\n      <td>24</td>\n      <td>30</td>\n      <td>13950</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>4wd</td>\n      <td>front</td>\n      <td>99.4</td>\n      <td>...</td>\n      <td>136</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>8.0</td>\n      <td>115</td>\n      <td>5500</td>\n      <td>18</td>\n      <td>22</td>\n      <td>17450</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 26 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<b>Good!</b> Now, we have a dataset with no missing values.\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h3 id=\"correct_data_format\">Correct data format</h3>\n<b>We are almost there!</b>\n<p>The last step in data cleaning is checking and making sure that all data is in the correct format (int, float, text or other).</p>\n\nIn Pandas, we use:\n\n<p><b>.dtype()</b> to check the data type</p>\n<p><b>.astype()</b> to change the data type</p>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Let's list the data types for each column</h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df.dtypes",
      "metadata": {
        "trusted": true
      },
      "execution_count": 32,
      "outputs": [
        {
          "execution_count": 32,
          "output_type": "execute_result",
          "data": {
            "text/plain": "symboling              int64\nnormalized-losses     object\nmake                  object\nfuel-type             object\naspiration            object\nnum-of-doors          object\nbody-style            object\ndrive-wheels          object\nengine-location       object\nwheel-base           float64\nlength               float64\nwidth                float64\nheight               float64\ncurb-weight            int64\nengine-type           object\nnum-of-cylinders      object\nengine-size            int64\nfuel-system           object\nbore                  object\nstroke                object\ncompression-ratio    float64\nhorsepower            object\npeak-rpm              object\ncity-mpg               int64\nhighway-mpg            int64\nprice                 object\ndtype: object"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<p>As we can see above, some columns are not of the correct data type. Numerical variables should have type 'float' or 'int', and variables with strings such as categories should have type 'object'. For example, 'bore' and 'stroke' variables are numerical values that describe the engines, so we should expect them to be of the type 'float' or 'int'; however, they are shown as type 'object'. We have to convert data types into a proper format for each column using the \"astype()\" method.</p> \n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Convert data types to proper format</h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df[[\"bore\", \"stroke\"]] = df[[\"bore\", \"stroke\"]].astype(\"float\")\ndf[[\"normalized-losses\"]] = df[[\"normalized-losses\"]].astype(\"int\")\ndf[[\"price\"]] = df[[\"price\"]].astype(\"float\")\ndf[[\"peak-rpm\"]] = df[[\"peak-rpm\"]].astype(\"float\")",
      "metadata": {
        "trusted": true
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "<h4>Let us list the columns after the conversion</h4>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df.dtypes",
      "metadata": {
        "trusted": true
      },
      "execution_count": 34,
      "outputs": [
        {
          "execution_count": 34,
          "output_type": "execute_result",
          "data": {
            "text/plain": "symboling              int64\nnormalized-losses      int32\nmake                  object\nfuel-type             object\naspiration            object\nnum-of-doors          object\nbody-style            object\ndrive-wheels          object\nengine-location       object\nwheel-base           float64\nlength               float64\nwidth                float64\nheight               float64\ncurb-weight            int64\nengine-type           object\nnum-of-cylinders      object\nengine-size            int64\nfuel-system           object\nbore                 float64\nstroke               float64\ncompression-ratio    float64\nhorsepower            object\npeak-rpm             float64\ncity-mpg               int64\nhighway-mpg            int64\nprice                float64\ndtype: object"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<b>Wonderful!</b>\n\nNow we have finally obtained the cleaned dataset with no missing values with all data in its proper format.\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h2 id=\"data_standardization\">Data Standardization</h2>\n<p>\nData is usually collected from different agencies in different formats.\n(Data standardization is also a term for a particular type of data normalization where we subtract the mean and divide by the standard deviation.)\n</p>\n\n<b>What is standardization?</b>\n\n<p>Standardization is the process of transforming data into a common format, allowing the researcher to make the meaningful comparison.\n</p>\n\n<b>Example</b>\n\n<p>Transform mpg to L/100km:</p>\n<p>In our dataset, the fuel consumption columns \"city-mpg\" and \"highway-mpg\" are represented by mpg (miles per gallon) unit. Assume we are developing an application in a country that accepts the fuel consumption with L/100km standard.</p>\n<p>We will need to apply <b>data transformation</b> to transform mpg into L/100km.</p>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<p>The formula for unit conversion is:<p>\nL/100km = 235 / mpg\n<p>We can do many mathematical operations directly in Pandas.</p>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 35,
      "outputs": [
        {
          "execution_count": 35,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   symboling  normalized-losses         make fuel-type aspiration  \\\n0          3                122  alfa-romero       gas        std   \n1          3                122  alfa-romero       gas        std   \n2          1                122  alfa-romero       gas        std   \n3          2                164         audi       gas        std   \n4          2                164         audi       gas        std   \n\n  num-of-doors   body-style drive-wheels engine-location  wheel-base  ...  \\\n0          two  convertible          rwd           front        88.6  ...   \n1          two  convertible          rwd           front        88.6  ...   \n2          two    hatchback          rwd           front        94.5  ...   \n3         four        sedan          fwd           front        99.8  ...   \n4         four        sedan          4wd           front        99.4  ...   \n\n   engine-size  fuel-system  bore  stroke compression-ratio horsepower  \\\n0          130         mpfi  3.47    2.68               9.0        111   \n1          130         mpfi  3.47    2.68               9.0        111   \n2          152         mpfi  2.68    3.47               9.0        154   \n3          109         mpfi  3.19    3.40              10.0        102   \n4          136         mpfi  3.19    3.40               8.0        115   \n\n   peak-rpm city-mpg  highway-mpg    price  \n0    5000.0       21           27  13495.0  \n1    5000.0       21           27  16500.0  \n2    5000.0       19           26  16500.0  \n3    5500.0       24           30  13950.0  \n4    5500.0       18           22  17450.0  \n\n[5 rows x 26 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>symboling</th>\n      <th>normalized-losses</th>\n      <th>make</th>\n      <th>fuel-type</th>\n      <th>aspiration</th>\n      <th>num-of-doors</th>\n      <th>body-style</th>\n      <th>drive-wheels</th>\n      <th>engine-location</th>\n      <th>wheel-base</th>\n      <th>...</th>\n      <th>engine-size</th>\n      <th>fuel-system</th>\n      <th>bore</th>\n      <th>stroke</th>\n      <th>compression-ratio</th>\n      <th>horsepower</th>\n      <th>peak-rpm</th>\n      <th>city-mpg</th>\n      <th>highway-mpg</th>\n      <th>price</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>3</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>130</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000.0</td>\n      <td>21</td>\n      <td>27</td>\n      <td>13495.0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>130</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000.0</td>\n      <td>21</td>\n      <td>27</td>\n      <td>16500.0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>hatchback</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>94.5</td>\n      <td>...</td>\n      <td>152</td>\n      <td>mpfi</td>\n      <td>2.68</td>\n      <td>3.47</td>\n      <td>9.0</td>\n      <td>154</td>\n      <td>5000.0</td>\n      <td>19</td>\n      <td>26</td>\n      <td>16500.0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>fwd</td>\n      <td>front</td>\n      <td>99.8</td>\n      <td>...</td>\n      <td>109</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>10.0</td>\n      <td>102</td>\n      <td>5500.0</td>\n      <td>24</td>\n      <td>30</td>\n      <td>13950.0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>4wd</td>\n      <td>front</td>\n      <td>99.4</td>\n      <td>...</td>\n      <td>136</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>8.0</td>\n      <td>115</td>\n      <td>5500.0</td>\n      <td>18</td>\n      <td>22</td>\n      <td>17450.0</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 26 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": "# Convert mpg to L/100km by mathematical operation (235 divided by mpg)\ndf['city-L/100km'] = 235/df[\"city-mpg\"]\n\n# check your transformed data \ndf.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 36,
      "outputs": [
        {
          "execution_count": 36,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   symboling  normalized-losses         make fuel-type aspiration  \\\n0          3                122  alfa-romero       gas        std   \n1          3                122  alfa-romero       gas        std   \n2          1                122  alfa-romero       gas        std   \n3          2                164         audi       gas        std   \n4          2                164         audi       gas        std   \n\n  num-of-doors   body-style drive-wheels engine-location  wheel-base  ...  \\\n0          two  convertible          rwd           front        88.6  ...   \n1          two  convertible          rwd           front        88.6  ...   \n2          two    hatchback          rwd           front        94.5  ...   \n3         four        sedan          fwd           front        99.8  ...   \n4         four        sedan          4wd           front        99.4  ...   \n\n   fuel-system  bore  stroke  compression-ratio horsepower peak-rpm  city-mpg  \\\n0         mpfi  3.47    2.68                9.0        111   5000.0        21   \n1         mpfi  3.47    2.68                9.0        111   5000.0        21   \n2         mpfi  2.68    3.47                9.0        154   5000.0        19   \n3         mpfi  3.19    3.40               10.0        102   5500.0        24   \n4         mpfi  3.19    3.40                8.0        115   5500.0        18   \n\n  highway-mpg    price  city-L/100km  \n0          27  13495.0     11.190476  \n1          27  16500.0     11.190476  \n2          26  16500.0     12.368421  \n3          30  13950.0      9.791667  \n4          22  17450.0     13.055556  \n\n[5 rows x 27 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>symboling</th>\n      <th>normalized-losses</th>\n      <th>make</th>\n      <th>fuel-type</th>\n      <th>aspiration</th>\n      <th>num-of-doors</th>\n      <th>body-style</th>\n      <th>drive-wheels</th>\n      <th>engine-location</th>\n      <th>wheel-base</th>\n      <th>...</th>\n      <th>fuel-system</th>\n      <th>bore</th>\n      <th>stroke</th>\n      <th>compression-ratio</th>\n      <th>horsepower</th>\n      <th>peak-rpm</th>\n      <th>city-mpg</th>\n      <th>highway-mpg</th>\n      <th>price</th>\n      <th>city-L/100km</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>3</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000.0</td>\n      <td>21</td>\n      <td>27</td>\n      <td>13495.0</td>\n      <td>11.190476</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000.0</td>\n      <td>21</td>\n      <td>27</td>\n      <td>16500.0</td>\n      <td>11.190476</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>hatchback</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>94.5</td>\n      <td>...</td>\n      <td>mpfi</td>\n      <td>2.68</td>\n      <td>3.47</td>\n      <td>9.0</td>\n      <td>154</td>\n      <td>5000.0</td>\n      <td>19</td>\n      <td>26</td>\n      <td>16500.0</td>\n      <td>12.368421</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>fwd</td>\n      <td>front</td>\n      <td>99.8</td>\n      <td>...</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>10.0</td>\n      <td>102</td>\n      <td>5500.0</td>\n      <td>24</td>\n      <td>30</td>\n      <td>13950.0</td>\n      <td>9.791667</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>4wd</td>\n      <td>front</td>\n      <td>99.4</td>\n      <td>...</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>8.0</td>\n      <td>115</td>\n      <td>5500.0</td>\n      <td>18</td>\n      <td>22</td>\n      <td>17450.0</td>\n      <td>13.055556</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 27 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<div class=\"alert alert-danger alertdanger\" style=\"margin-top: 20px\">\n<h1> Question  #2: </h1>\n\n<b>According to the example above, transform mpg to L/100km in the column of \"highway-mpg\" and change the name of column to \"highway-L/100km\".</b>\n\n</div>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "# Write your code below and press Shift+Enter to execute \ndf[\"highway-mpg\"] = 235/df[\"highway-mpg\"]\ndf.rename(columns={'highway-mpg':'highway-L/100km'}, inplace=True)\ndf.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 37,
      "outputs": [
        {
          "execution_count": 37,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   symboling  normalized-losses         make fuel-type aspiration  \\\n0          3                122  alfa-romero       gas        std   \n1          3                122  alfa-romero       gas        std   \n2          1                122  alfa-romero       gas        std   \n3          2                164         audi       gas        std   \n4          2                164         audi       gas        std   \n\n  num-of-doors   body-style drive-wheels engine-location  wheel-base  ...  \\\n0          two  convertible          rwd           front        88.6  ...   \n1          two  convertible          rwd           front        88.6  ...   \n2          two    hatchback          rwd           front        94.5  ...   \n3         four        sedan          fwd           front        99.8  ...   \n4         four        sedan          4wd           front        99.4  ...   \n\n   fuel-system  bore  stroke  compression-ratio horsepower peak-rpm  city-mpg  \\\n0         mpfi  3.47    2.68                9.0        111   5000.0        21   \n1         mpfi  3.47    2.68                9.0        111   5000.0        21   \n2         mpfi  2.68    3.47                9.0        154   5000.0        19   \n3         mpfi  3.19    3.40               10.0        102   5500.0        24   \n4         mpfi  3.19    3.40                8.0        115   5500.0        18   \n\n  highway-L/100km    price  city-L/100km  \n0        8.703704  13495.0     11.190476  \n1        8.703704  16500.0     11.190476  \n2        9.038462  16500.0     12.368421  \n3        7.833333  13950.0      9.791667  \n4       10.681818  17450.0     13.055556  \n\n[5 rows x 27 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>symboling</th>\n      <th>normalized-losses</th>\n      <th>make</th>\n      <th>fuel-type</th>\n      <th>aspiration</th>\n      <th>num-of-doors</th>\n      <th>body-style</th>\n      <th>drive-wheels</th>\n      <th>engine-location</th>\n      <th>wheel-base</th>\n      <th>...</th>\n      <th>fuel-system</th>\n      <th>bore</th>\n      <th>stroke</th>\n      <th>compression-ratio</th>\n      <th>horsepower</th>\n      <th>peak-rpm</th>\n      <th>city-mpg</th>\n      <th>highway-L/100km</th>\n      <th>price</th>\n      <th>city-L/100km</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>3</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000.0</td>\n      <td>21</td>\n      <td>8.703704</td>\n      <td>13495.0</td>\n      <td>11.190476</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>...</td>\n      <td>mpfi</td>\n      <td>3.47</td>\n      <td>2.68</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000.0</td>\n      <td>21</td>\n      <td>8.703704</td>\n      <td>16500.0</td>\n      <td>11.190476</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>two</td>\n      <td>hatchback</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>94.5</td>\n      <td>...</td>\n      <td>mpfi</td>\n      <td>2.68</td>\n      <td>3.47</td>\n      <td>9.0</td>\n      <td>154</td>\n      <td>5000.0</td>\n      <td>19</td>\n      <td>9.038462</td>\n      <td>16500.0</td>\n      <td>12.368421</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>fwd</td>\n      <td>front</td>\n      <td>99.8</td>\n      <td>...</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>10.0</td>\n      <td>102</td>\n      <td>5500.0</td>\n      <td>24</td>\n      <td>7.833333</td>\n      <td>13950.0</td>\n      <td>9.791667</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>gas</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>4wd</td>\n      <td>front</td>\n      <td>99.4</td>\n      <td>...</td>\n      <td>mpfi</td>\n      <td>3.19</td>\n      <td>3.40</td>\n      <td>8.0</td>\n      <td>115</td>\n      <td>5500.0</td>\n      <td>18</td>\n      <td>10.681818</td>\n      <td>17450.0</td>\n      <td>13.055556</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 27 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<details><summary>Click here for the solution</summary>\n\n```python\n# transform mpg to L/100km by mathematical operation (235 divided by mpg)\ndf[\"highway-mpg\"] = 235/df[\"highway-mpg\"]\n\n# rename column name from \"highway-mpg\" to \"highway-L/100km\"\ndf.rename(columns={'highway-mpg':'highway-L/100km'}, inplace=True)\n\n# check your transformed data \ndf.head()\n\n```\n\n</details>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h2 id=\"data_normalization\">Data Normalization</h2>\n\n<b>Why normalization?</b>\n\n<p>Normalization is the process of transforming values of several variables into a similar range. Typical normalizations include scaling the variable so the variable average is 0, scaling the variable so the variance is 1, or scaling the variable so the variable values range from 0 to 1.\n</p>\n\n<b>Example</b>\n\n<p>To demonstrate normalization, let's say we want to scale the columns \"length\", \"width\" and \"height\".</p>\n<p><b>Target:</b> would like to normalize those variables so their value ranges from 0 to 1</p>\n<p><b>Approach:</b> replace original value by (original value)/(maximum value)</p>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "# replace (original value) by (original value)/(maximum value)\ndf['length'] = df['length']/df['length'].max()\ndf['width'] = df['width']/df['width'].max()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "<div class=\"alert alert-danger alertdanger\" style=\"margin-top: 20px\">\n<h1> Question #3: </h1>\n\n<b>According to the example above, normalize the column \"height\".</b>\n\n</div>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "# Write your code below and press Shift+Enter to execute \ndf['height'] = df['height']/df['height'].max()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": "df[[\"length\",\"width\",\"height\"]].head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 40,
      "outputs": [
        {
          "execution_count": 40,
          "output_type": "execute_result",
          "data": {
            "text/plain": "     length     width    height\n0  0.811148  0.890278  0.816054\n1  0.811148  0.890278  0.816054\n2  0.822681  0.909722  0.876254\n3  0.848630  0.919444  0.908027\n4  0.848630  0.922222  0.908027",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>length</th>\n      <th>width</th>\n      <th>height</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0.811148</td>\n      <td>0.890278</td>\n      <td>0.816054</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>0.811148</td>\n      <td>0.890278</td>\n      <td>0.816054</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>0.822681</td>\n      <td>0.909722</td>\n      <td>0.876254</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0.848630</td>\n      <td>0.919444</td>\n      <td>0.908027</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0.848630</td>\n      <td>0.922222</td>\n      <td>0.908027</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<details><summary>Click here for the solution</summary>\n\n```python\ndf['height'] = df['height']/df['height'].max() \n\n# show the scaled columns\ndf[[\"length\",\"width\",\"height\"]].head()\n\n\n```\n\n</details>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "Here we can see we've normalized \"length\", \"width\" and \"height\" in the range of \\[0,1].\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h2 id=\"binning\">Binning</h2>\n<b>Why binning?</b>\n<p>\n    Binning is a process of transforming continuous numerical variables into discrete categorical 'bins' for grouped analysis.\n</p>\n\n<b>Example: </b>\n\n<p>In our dataset, \"horsepower\" is a real valued variable ranging from 48 to 288 and it has 59 unique values. What if we only care about the price difference between cars with high horsepower, medium horsepower, and little horsepower (3 types)? Can we rearrange them into three ‘bins' to simplify analysis? </p>\n\n<p>We will use the pandas method 'cut' to segment the 'horsepower' column into 3 bins.</p>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h3>Example of Binning Data In Pandas</h3>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "Convert data to correct format:\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df[\"horsepower\"]=df[\"horsepower\"].astype(int, copy=True)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "Let's plot the histogram of horsepower to see what the distribution of horsepower looks like.\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "%matplotlib inline\nimport matplotlib as plt\nfrom matplotlib import pyplot\nplt.pyplot.hist(df[\"horsepower\"])\n\n# set x/y labels and plot title\nplt.pyplot.xlabel(\"horsepower\")\nplt.pyplot.ylabel(\"count\")\nplt.pyplot.title(\"horsepower bins\")",
      "metadata": {
        "trusted": true
      },
      "execution_count": 42,
      "outputs": [
        {
          "execution_count": 42,
          "output_type": "execute_result",
          "data": {
            "text/plain": "Text(0.5, 1.0, 'horsepower bins')"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 432x288 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVFUlEQVR4nO3de7CkdX3n8feHEe9cw2QygmGUoLVoGdSJl/XGilcwGdZEIqXuxLBBt9TSWt1kkmyyJhV3UROjUWspjIYxXrEMwoYqFUcZ1HgbDCKIBoIzKhmYAVHQ3ajAd/94fic0M+fMHHSe7jPn935VdfVz7efbv+n59O88/fSvU1VIkvpxwKwLkCRNl8EvSZ0x+CWpMwa/JHXG4Jekzhj8ktQZg18zlWRrkqfNuo6lLskJSb6zh/VnJfmjadak/dc9Zl2ApJ9dVb101jVo/2GPX8tCkmXTiVlOz0VLk8GvpeD4JJcn+X6SDya599yKJL+T5Jok301yQZIHTKyrJC9LcjVwdQZ/mWRHkluSfDXJw9u290ry50m+leSGdmrkPm3dCUm+k+QPktzYTj+9YOI4hyR5d5KdSbYl+e9JDmjrtiV5dJt+QavpYW3+9CQfadMHJNmQ5J+T3JTk3CSHt3Vr2n6nJ/kW8MmFGmoPNZ6T5M92eT6vbm2xPcmLJ7Y9KcnXktya5Lokr/kZ/u20HzL4tRScCjwLeBDwCOC3AJI8Ffhfbf1qYBvwgV32PQV4LHAc8AzgycBDgEPafje17c5sy48Hfgk4Evjjicf5BeCItnw9cHaSh7Z1b22P92DgKcB/AuaCdDNwQpt+CnBtq2FufnObfkWr9SnAA4Cbgbfv8lyeAvw74Jm7NtAiapxv20PatqcDb09yWFv3TuAlVXUQ8HD28EajZaqqvHmb2Q3YCrxwYv4NwFlt+p3AGybW3R/4CbCmzRfw1In1TwX+CXgccMDE8gA/BI6ZWPZ44Jtt+gTgNuB+E+vPBf4IWAH8GDhuYt1LgIvb9OnABW36KuA/Ax9o89uAR02sO3HiMVa353IPYE17Lg/eQzstWGObPgf4s4lt/x9wj4ltdwCPa9Pfas/h4Fn/+3ubzc0ev5aC6yem/y9DwMPQM942t6KqfsDQgz9yYvtvT6z/JPA2hp70jiRnJzkYWAncF7g0yfeSfA/4aFs+5+aq+uHE/LZ2/COAAyfraNNzNWwGnpRkNcObxLnAE5KsYehxX9a2Oxo4b+L4VwG3A6vmey4LWKjG+dxUVbdNzE+2668DJwHbkmxO8vi9HFfLjMGvpexfGAITgCT3A34OuG5im7sML1tVf1VVj2Y49fMQ4L8BNzL0gB9WVYe22yFVdf+JXQ9rjz/nF9vxb2TomR+9y7rr2vGuYQjVVwCXVNUtDG9kZwCfqao72j7fBp49cfxDq+reVbXgc5nHQjXeLVX1papaB/w88BGGNyt1xODXUvZ+4MVJjk9yL+B/Al+oqq3zbZzkV5I8NsmBDKd2/hW4o4XvO4C/TPLzbdsjk+x6Lv1PktwzyZOA5wAfqqrbGYLxdUkOSnI08F+B90zstxl4OXeez794l3mAs9pjHN2OvzLJup+iTXar8e7s3PZ9QZJDquonwC3AHXvbT8uLwa8lq6o+wXCe/cPAduAY4Pl72OVghoC/meE0yE3AG9u63wOuAT6f5BbgE8DkB6PXt/3+BXgv8NKq+npb9wqGN5Jrgc8A7wPeNbHvZuAg4JIF5gHeAlwAfDzJrcDnGT6Uvjv2VOPd8SJga2uHlwIv2Mv2WmZS5Q+xqG9JTgDeU1VHzbgUaSrs8UtSZwx+SeqMp3okqTOjjgmSZCtwK8P1yrdV1dr2NfUPMnxpZStwalXdPGYdkqQ7jdrjb8G/tqpunFj2BuC7VXVmkg3AYVX1e3t6nCOOOKLWrFkzWp2StBxdeumlN1bVyl2Xz2IUwHXcObbJRoZrnvcY/GvWrGHLli3jViVJy0ySbfMtH/vD3WK4bvnSJGe0Zauqanubvp67fmX93yQ5I8mWJFt27tw5cpmS1I+xe/xPrKrr2rclL0pyly+bVFUlmfdcU1WdDZwNsHbtWj+BlqR9ZNQe/9w4JFW1AzgPeAxwQxvQina/Y8waJEl3NVrwJ7lfkoPmphnGSr+C4Wvr69tm64Hzx6pBkrS7MU/1rGIYhnbuOO+rqo8m+RJwbpLTGcZTOXXEGiRJuxgt+KvqWuCX51l+E3DiWMeVJO2ZQzZIUmcMfknqjMEvSZ2ZxTd3NbI1Gy6cyXG3nnnyTI4r6e6xxy9JnTH4JakzBr8kdcbgl6TOGPyS1BmDX5I6Y/BLUmcMfknqjMEvSZ0x+CWpMwa/JHXG4JekzjhI24hmNViaJO2JPX5J6ozBL0mdMfglqTPL/hy/59kl6a7s8UtSZwx+SeqMwS9JnTH4JakzBr8kdcbgl6TOGPyS1BmDX5I6Y/BLUmcMfknqjMEvSZ0x+CWpMwa/JHVm9OBPsiLJPyb5+zZ/eJKLklzd7g8buwZJ0p2m0eN/JXDVxPwGYFNVHQtsavOSpCkZNfiTHAWcDPz1xOJ1wMY2vRE4ZcwaJEl3NXaP/83A7wJ3TCxbVVXb2/T1wKr5dkxyRpItSbbs3Llz3ColqSOjBX+S5wA7qurShbapqgJqgXVnV9Xaqlq7cuXKscqUpO6M+dOLTwB+LclJwL2Bg5O8B7ghyeqq2p5kNbBjxBokSbsYrcdfVb9fVUdV1Rrg+cAnq+qFwAXA+rbZeuD8sWqQJO1uFtfxnwk8PcnVwNPavCRpSsY81fNvqupi4OI2fRNw4jSOK0nand/claTOGPyS1BmDX5I6Y/BLUmcMfknqjMEvSZ0x+CWpMwa/JHXG4Jekzhj8ktQZg1+SOmPwS1JnDH5J6ozBL0mdMfglqTMGvyR1xuCXpM4Y/JLUGYNfkjpj8EtSZwx+SeqMwS9JnTH4JakzBr8kdcbgl6TOGPyS1BmDX5I6Y/BLUmcMfknqjMEvSZ0x+CWpMwa/JHXG4Jekzhj8ktSZ0YI/yb2TfDHJV5JcmeRP2vLDk1yU5Op2f9hYNUiSdjdmj/9HwFOr6peB44FnJXkcsAHYVFXHApvavCRpSkYL/hr8oM0e2G4FrAM2tuUbgVPGqkGStLtRz/EnWZHkMmAHcFFVfQFYVVXb2ybXA6sW2PeMJFuSbNm5c+eYZUpSV0YN/qq6vaqOB44CHpPk4busL4a/Aubb9+yqWltVa1euXDlmmZLUlalc1VNV3wM+BTwLuCHJaoB2v2MaNUiSBmNe1bMyyaFt+j7A04GvAxcA69tm64Hzx6pBkrS7e4z42KuBjUlWMLzBnFtVf5/kc8C5SU4HtgGnjliDJGkXowV/VV0OPHKe5TcBJ451XEnSnvnNXUnqjMEvSZ0x+CWpMwa/JHVmUcGfZNNilkmSlr49XtWT5N7AfYEj2iiaaasOBo4cuTZJ0gj2djnnS4BXAQ8ALuXO4L8FeNt4ZUmSxrLH4K+qtwBvSfKKqnrrlGqSJI1oUV/gqqq3Jvn3wJrJfarq3SPVJUkayaKCP8nfAscAlwG3t8UFGPyStJ9Z7JANa4Hj2jDKkqT92GKv478C+IUxC5EkTcdie/xHAF9L8kWG39IFoKp+bZSqJEmjWWzwv3bMIiRJ07PYq3o2j12IJGk6FntVz63c+du49wQOBH5YVQePVZgkaRyL7fEfNDedJMA64HFjFSVJGs/dHp2zBh8Bnrnvy5EkjW2xp3qeOzF7AMN1/f86SkWSpFEt9qqeX52Yvg3YynC6R5K0n1nsOf4Xj12I9n9rNlw4s2NvPfPkmR1b2t8s9odYjkpyXpId7fbhJEeNXZwkad9b7Ie7fwNcwDAu/wOA/9OWSZL2M4sN/pVV9TdVdVu7nQOsHLEuSdJIFhv8NyV5YZIV7fZC4KYxC5MkjWOxwf/bwKnA9cB24DeA3xqpJknSiBZ7OeefAuur6maAJIcDf87whiBJ2o8stsf/iLnQB6iq7wKPHKckSdKYFhv8ByQ5bG6m9fgX+9eCJGkJWWx4/wXwuSQfavPPA143TkmSpDEt9pu7706yBXhqW/TcqvraeGVJksay6NM1LegNe0naz93tYZklSfs3g1+SOjNa8Cd5YJJPJflakiuTvLItPzzJRUmubveH7e2xJEn7zpg9/tuAV1fVcQw/0/iyJMcBG4BNVXUssKnNS5KmZLTgr6rtVfXlNn0rcBVwJMMPuGxsm20EThmrBknS7qZyjj/JGoZv+n4BWFVV29uq64FV06hBkjQYPfiT3B/4MPCqqrplcl1VFVAL7HdGki1JtuzcuXPsMiWpG6MGf5IDGUL/vVX1d23xDUlWt/WrgR3z7VtVZ1fV2qpau3KlQ/9L0r4y5lU9Ad4JXFVVb5pYdQGwvk2vB84fqwZJ0u7GHGjtCcCLgK8muawt+wPgTODcJKcD2xjG+ZckTclowV9VnwGywOoTxzquJGnP/OauJHXG4Jekzhj8ktQZg1+SOmPwS1JnDH5J6ozBL0mdMfglqTMGvyR1xuCXpM4Y/JLUGYNfkjoz5uic0rK3ZsOFMzv21jNPntmxtX+zxy9JnTH4JakzBr8kdcbgl6TOGPyS1BmDX5I6Y/BLUme8jl/Lwiyvp5f2N/b4JakzBr8kdcbgl6TOGPyS1BmDX5I6Y/BLUmcMfknqjMEvSZ0x+CWpMwa/JHXG4Jekzhj8ktQZg1+SOjNa8Cd5V5IdSa6YWHZ4kouSXN3uDxvr+JKk+Y3Z4z8HeNYuyzYAm6rqWGBTm5ckTdFowV9VlwDf3WXxOmBjm94InDLW8SVJ85v2Of5VVbW9TV8PrJry8SWpezP7cLeqCqiF1ic5I8mWJFt27tw5xcokaXmbdvDfkGQ1QLvfsdCGVXV2Va2tqrUrV66cWoGStNxNO/gvANa36fXA+VM+viR1b8zLOd8PfA54aJLvJDkdOBN4epKrgae1eUnSFN1jrAeuqtMWWHXiWMeUJO2d39yVpM4Y/JLUGYNfkjpj8EtSZwx+SeqMwS9JnTH4JakzBr8kdcbgl6TOGPyS1BmDX5I6Y/BLUmcMfknqjMEvSZ0x+CWpMwa/JHVmtB9ikTSuNRsunMlxt5558kyOq33HHr8kdcbgl6TOGPyS1BmDX5I644e7ku6WWX2oDH6wvK/Y45ekzhj8ktQZg1+SOmPwS1JnDH5J6ozBL0mdMfglqTMGvyR1xuCXpM4Y/JLUGYNfkjrjWD2StBfLbXwie/yS1JmZBH+SZyX5RpJrkmyYRQ2S1KupB3+SFcDbgWcDxwGnJTlu2nVIUq9m0eN/DHBNVV1bVT8GPgCsm0EdktSlWXy4eyTw7Yn57wCP3XWjJGcAZ7TZHyT5xj449hHAjfvgcZYj22Zhts3Cpto2ef20jrRP7JO2+Rmf89HzLVyyV/VU1dnA2fvyMZNsqaq1+/IxlwvbZmG2zcJsm4Ut5baZxame64AHTswf1ZZJkqZgFsH/JeDYJA9Kck/g+cAFM6hDkro09VM9VXVbkpcDHwNWAO+qqiundPh9eupombFtFmbbLMy2WdiSbZtU1axrkCRNkd/claTOGPyS1JllG/xJtib5apLLkmxpyw5PclGSq9v9YbOuc1qSvCvJjiRXTCxbsD2S/H4bUuMbSZ45m6qnY4G2eW2S69rr57IkJ02s66ltHpjkU0m+luTKJK9sy7t/7eyhbZb+a6eqluUN2AocscuyNwAb2vQG4PWzrnOK7fFk4FHAFXtrD4ahNL4C3At4EPDPwIpZP4cpt81rgdfMs21vbbMaeFSbPgj4p9YG3b929tA2S/61s2x7/AtYB2xs0xuBU2ZXynRV1SXAd3dZvFB7rAM+UFU/qqpvAtcwDLWxLC3QNgvprW22V9WX2/StwFUM377v/rWzh7ZZyJJpm+Uc/AV8PMmlbfgHgFVVtb1NXw+smk1pS8ZC7THfsBp7ekEvVy9Pcnk7FTR3KqPbtkmyBngk8AV87dzFLm0DS/y1s5yD/4lV9SiGUUBfluTJkytr+NvLa1kb22M3/xs4Bjge2A78xUyrmbEk9wc+DLyqqm6ZXNf7a2eetlnyr51lG/xVdV273wGcx/An1Q1JVgO0+x2zq3BJWKg9uh9Wo6puqKrbq+oO4B3c+Sd5d22T5ECGYHtvVf1dW+xrh/nbZn947SzL4E9yvyQHzU0DzwCuYBgaYn3bbD1w/mwqXDIWao8LgOcnuVeSBwHHAl+cQX0zMxdqzX9keP1AZ22TJMA7gauq6k0Tq7p/7SzUNvvFa2fWn4yP9Gn7gxk+Pf8KcCXwh235zwGbgKuBTwCHz7rWKbbJ+xn+7PwJw7nF0/fUHsAfMlx18A3g2bOufwZt87fAV4HLGf7Dru60bZ7IcBrncuCydjvJ184e22bJv3YcskGSOrMsT/VIkhZm8EtSZwx+SeqMwS9JnTH4JakzBr+WlSRrJkfZlLQ7g19qkkz9p0h/GvtLnVq6DH4tRyuSvKONkf7xJPdJcnySz7eBs86bGzgrycVJ3tx+s+GVSZ6X5IokX0lySdtmRZI3JvlS2/8lbfkJSS5JcmEbX/2sJAe0dadl+D2IK5K8vi17XpI3telXJrm2TT84yWfb9KOTbG6DC35sYliEu9Q53ebUcmPPQcvRscBpVfU7Sc4Ffh34XeAVVbU5yZ8C/wN4Vdv+nlW1FiDJV4FnVtV1SQ5t608Hvl9Vv5LkXsBnk3y8rXsMwzjr24CPAs9N8g/A64FHAzczjBJ7CvDpVgfAk4CbkhzZpi9p4768FVhXVTuT/CbwOuC3d61T+lkY/FqOvllVl7XpSxlGSjy0qja3ZRuBD01s/8GJ6c8C57Q3jLkByZ4BPCLJb7T5QxjeXH4MfLGq5nru72f4Gv9PgIuramdb/l7gyVX1kST3b+NIPRB4H8OPwDypHeuhwMOBi4ZhYFjBMJTEfHVKPzWDX8vRjyambwcO3cv2P5ybqKqXJnkscDJwaZJHA2H4a+FjkzslOYHdhyPe2xgo/wC8mGGslk8z9OYfD7wa+EXgyqp6/N7qlH4WnuNXD74P3JzkSW3+RcDm+TZMckxVfaGq/hjYydAz/xjwX9qpGJI8pI36CvCYJA9q5/Z/E/gMw4iLT0lyRJIVwGkTx/s08BrgEuAfgf8A/Kiqvs/wZrAyyePbcQ5M8rB91wzSwB6/erEeOCvJfYFrGXrd83ljkmMZevmbGEZ4vRxYA3y5DcW7kzt/avBLwNuAXwI+BZxXVXck2dDmA1xYVXPDFn+a4c3kkqq6Pcm3ga8DVNWP2+mkv0pyCMP/zzczjDAr7TOOzin9lNqpntdU1XNmXIp0t3iqR5I6Y49fkjpjj1+SOmPwS1JnDH5J6ozBL0mdMfglqTP/H9AHDyw7qJUsAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<p>We would like 3 bins of equal size bandwidth so we use numpy's <code>linspace(start_value, end_value, numbers_generated</code> function.</p>\n<p>Since we want to include the minimum value of horsepower, we want to set start_value = min(df[\"horsepower\"]).</p>\n<p>Since we want to include the maximum value of horsepower, we want to set end_value = max(df[\"horsepower\"]).</p>\n<p>Since we are building 3 bins of equal length, there should be 4 dividers, so numbers_generated = 4.</p>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "We build a bin array with a minimum value to a maximum value by using the bandwidth calculated above. The values will determine when one bin ends and another begins.\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "bins = np.linspace(min(df[\"horsepower\"]), max(df[\"horsepower\"]), 4)\nbins",
      "metadata": {
        "trusted": true
      },
      "execution_count": 43,
      "outputs": [
        {
          "execution_count": 43,
          "output_type": "execute_result",
          "data": {
            "text/plain": "array([ 48.        , 119.33333333, 190.66666667, 262.        ])"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "We set group  names:\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "group_names = ['Low', 'Medium', 'High']",
      "metadata": {
        "trusted": true
      },
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "We apply the function \"cut\" to determine what each value of `df['horsepower']` belongs to.\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df['horsepower-binned'] = pd.cut(df['horsepower'], bins, labels=group_names, include_lowest=True )\ndf[['horsepower','horsepower-binned']].head(20)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 45,
      "outputs": [
        {
          "execution_count": 45,
          "output_type": "execute_result",
          "data": {
            "text/plain": "    horsepower horsepower-binned\n0          111               Low\n1          111               Low\n2          154            Medium\n3          102               Low\n4          115               Low\n5          110               Low\n6          110               Low\n7          110               Low\n8          140            Medium\n9          101               Low\n10         101               Low\n11         121            Medium\n12         121            Medium\n13         121            Medium\n14         182            Medium\n15         182            Medium\n16         182            Medium\n17          48               Low\n18          70               Low\n19          70               Low",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>horsepower</th>\n      <th>horsepower-binned</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>111</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>111</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>154</td>\n      <td>Medium</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>102</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>115</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>110</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>110</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>110</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>140</td>\n      <td>Medium</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>101</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>101</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>121</td>\n      <td>Medium</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>121</td>\n      <td>Medium</td>\n    </tr>\n    <tr>\n      <th>13</th>\n      <td>121</td>\n      <td>Medium</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>182</td>\n      <td>Medium</td>\n    </tr>\n    <tr>\n      <th>15</th>\n      <td>182</td>\n      <td>Medium</td>\n    </tr>\n    <tr>\n      <th>16</th>\n      <td>182</td>\n      <td>Medium</td>\n    </tr>\n    <tr>\n      <th>17</th>\n      <td>48</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>18</th>\n      <td>70</td>\n      <td>Low</td>\n    </tr>\n    <tr>\n      <th>19</th>\n      <td>70</td>\n      <td>Low</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "Let's see the number of vehicles in each bin:\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df[\"horsepower-binned\"].value_counts()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 46,
      "outputs": [
        {
          "execution_count": 46,
          "output_type": "execute_result",
          "data": {
            "text/plain": "Low       153\nMedium     43\nHigh        5\nName: horsepower-binned, dtype: int64"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "Let's plot the distribution of each bin:\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "%matplotlib inline\nimport matplotlib as plt\nfrom matplotlib import pyplot\npyplot.bar(group_names, df[\"horsepower-binned\"].value_counts())\n\n# set x/y labels and plot title\nplt.pyplot.xlabel(\"horsepower\")\nplt.pyplot.ylabel(\"count\")\nplt.pyplot.title(\"horsepower bins\")",
      "metadata": {
        "trusted": true
      },
      "execution_count": 47,
      "outputs": [
        {
          "execution_count": 47,
          "output_type": "execute_result",
          "data": {
            "text/plain": "Text(0.5, 1.0, 'horsepower bins')"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 432x288 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAY8klEQVR4nO3de5RlZX3m8e8DKHjlki4J0kCjQSeN47UlEgdFMUrUsVm5YDOatEqCughjZjQKZiKJkQxexnsSVhsRHBVsE5WeMeMlGOgYL9gogoAEIrQ0Al0CiqKCwG/+2G9vD2VVd1H0Oae66vtZq1bv/b778qs6q89z9u09qSokSQLYadwFSJLmD0NBktQzFCRJPUNBktQzFCRJPUNBktQzFDQvJbkmybPGXcd8l+TwJJu20n9akj8bZU3ase0y7gIkDU9VvWLcNWjH4pGCFrQkC+aDz0L6XTR/GQqazx6f5OIkP0jy0SS7belI8odJrkpyc5J1SR4+0FdJjk9yJXBlOu9IsjnJrUkuSfKYtuyuSd6W5DtJbmynWx7Q+g5PsinJ65N8r53SetHAfnZP8sEkk0k2JvkfSXZqfRuTPKlNv6jVdHCbPzbJJ9v0TklOTPLvSW5KsjbJXq1vWVvv2CTfAT4/0x9qKzWekeRNU36fV7e/xfVJXjqw7HOTXJbkh0muS/Ka+/DaaQdlKGg+Oxo4EjgQeCzwEoAkzwT+Z+vfB9gInD1l3aOAXwOWA88GngY8Cti9rXdTW+7U1v544FeAfYE3DGznl4ElrX01sCbJo1vfe9r2HgE8Hfh9YMub7PnA4W366cC3Ww1b5s9v0ye0Wp8OPBy4BfjrKb/L04FfBZ4z9Q80ixqnW3b3tuyxwF8n2bP1vR94eVU9BHgMWwkhLWBV5Y8/8+4HuAZ48cD8W4DT2vT7gbcM9D0Y+BmwrM0X8MyB/mcC/wY8BdhpoD3AbcAjB9oOBa5u04cDdwIPGuhfC/wZsDNwB7B8oO/lwHlt+lhgXZu+HPgD4Ow2vxF44kDfEQPb2Kf9LrsAy9rv8oit/J1mrLFNnwG8aWDZnwC7DCy7GXhKm/5O+x0eOu7X35/x/XikoPnshoHpH9O9+UP3iXrjlo6q+hHdJ/99B5a/dqD/88B76T6Bb06yJslDgQnggcCFSb6f5PvAp1v7FrdU1W0D8xvb/pcA9xuso01vqeF84LAk+9AFyFrgqUmW0X1Sv6gtdwDwiYH9Xw7cBew93e8yg5lqnM5NVXXnwPzg3/W3gecCG5Ocn+TQbexXC5ChoB3Rd+neTAFI8iDgl4DrBpa5x/C/VfXuqnoS3emkRwF/AnyP7pPzwVW1R/vZvaoePLDqnm37W+zf9v89uk/0B0zpu67t7yq6N9wTgPVVdStdyB0HfKGq7m7rXAv85sD+96iq3apqxt9lGjPVeK9U1VeraiXwMOCTdEGmRcZQ0I7oLOClSR6fZFfgr4CvVNU10y2c5MlJfi3J/ehOF/0UuLu9Mb8PeEeSh7Vl900y9dz9XyS5f5LDgOcDH6uqu+jeNE9J8pAkBwD/HfjQwHrnA3/Ez68fnDdlHuC0to0D2v4nkqycw9/kF2q8Nyu3dV+UZPeq+hlwK3D3ttbTwmMoaIdTVf9Ed17/H4DrgUcCq7ayykPp3vxvoTu1chPw1tb3OuAq4MtJbgX+CRi8SHtDW++7wIeBV1TVt1rfCXQh823gC8BHgNMH1j0feAiwfoZ5gHcB64DPJvkh8GW6C+T3xtZqvDd+D7im/R1eAbxoG8trAUqVX7IjTSfJ4cCHqmrpmEuRRsYjBUlSz1CQJPWGFgpJTm9PTX5zSvsJSb6V5NIkbxloP6k9oXrFNBf6pJGrqvM8daTFZphjqZxBd2/4B7c0JHkGsBJ4XFXdPnDHx3K6C4UH091f/U9JHtXu8JAkjcjQQqGq1rcHdQa9Eji1qm5vy2xu7Svpnva8Hbg6yVXAIcCXtraPJUuW1LJlU3chSdqaCy+88HtVNTFd36hHXXwU3VOep9DdK/6aqvoq3VOgXx5YbhP3fDq1l+Q4ugeA2H///dmwYcNwK5akBSbJxpn6Rn2heRdgL7oxaP4EWJsk92YDVbWmqlZU1YqJiWmDTpI0R6MOhU3Ax6tzAd0Tk0vohgbYb2C5pdxzyAJJ0giMOhQ+CTwDIMmjgPvTjSGzDljVxrY/EDgIuGDEtUnSoje0awpJzqIbqndJuu+QPZluCIDT222qdwCrq3uk+tIka4HL6IYBPt47jyRp9HboYS5WrFhRXmiWpHsnyYVVtWK6Pp9oliT1DAVJUs9QkCT1DAVJUm/UTzTPK8tO/NS4S1iwrjn1eeMuQdIceKQgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoNLRSSnJ5kc/s+5ql9r05SSZYMtJ2U5KokVyR5zrDqkiTNbJhHCmcAR05tTLIf8GzgOwNty4FVwMFtnb9JsvMQa5MkTWNooVBV64Gbp+l6B/BaoAbaVgJnV9XtVXU1cBVwyLBqkyRNb6TXFJKsBK6rqm9M6doXuHZgflNrm24bxyXZkGTD5OTkkCqVpMVpZKGQ5IHA64E33JftVNWaqlpRVSsmJia2T3GSJGC0X8f5SOBA4BtJAJYCX0tyCHAdsN/AsktbmyRphEZ2pFBVl1TVw6pqWVUtoztF9MSqugFYB6xKsmuSA4GDgAtGVZskqTPMW1LPAr4EPDrJpiTHzrRsVV0KrAUuAz4NHF9Vdw2rNknS9IZ2+qiqjtlG/7Ip86cApwyrHknStvlEsySpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknrD/I7m05NsTvLNgba3JvlWkouTfCLJHgN9JyW5KskVSZ4zrLokSTMb5pHCGcCRU9o+Bzymqh4L/BtwEkCS5cAq4OC2zt8k2XmItUmSpjG0UKiq9cDNU9o+W1V3ttkvA0vb9Erg7Kq6vaquBq4CDhlWbZKk6Y3zmsLLgP/XpvcFrh3o29TafkGS45JsSLJhcnJyyCVK0uIyllBI8qfAncCH7+26VbWmqlZU1YqJiYntX5wkLWK7jHqHSV4CPB84oqqqNV8H7Dew2NLWJkkaoZEeKSQ5Engt8IKq+vFA1zpgVZJdkxwIHARcMMraJElDPFJIchZwOLAkySbgZLq7jXYFPpcE4MtV9YqqujTJWuAyutNKx1fVXcOqTZI0vaGFQlUdM03z+7ey/CnAKcOqR5K0bT7RLEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpN7QQiHJ6Uk2J/nmQNteST6X5Mr2754DfScluSrJFUmeM6y6JEkzG+aRwhnAkVPaTgTOraqDgHPbPEmWA6uAg9s6f5Nk5yHWJkmaxtBCoarWAzdPaV4JnNmmzwSOGmg/u6pur6qrgauAQ4ZVmyRpeqO+prB3VV3fpm8A9m7T+wLXDiy3qbX9giTHJdmQZMPk5OTwKpWkRWhsF5qrqoCaw3prqmpFVa2YmJgYQmWStHiNOhRuTLIPQPt3c2u/DthvYLmlrU2SNEKjDoV1wOo2vRo4Z6B9VZJdkxwIHARcMOLaJGnR22VYG05yFnA4sCTJJuBk4FRgbZJjgY3A0QBVdWmStcBlwJ3A8VV117BqkyRNb2ihUFXHzNB1xAzLnwKcMqx6JEnb5hPNkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6s0qFJKcO5s2SdKObatjHyXZDXgg3aB2ewJpXQ9lhi/BkSTtuLY1IN7LgT8GHg5cyM9D4VbgvcMrS5I0DlsNhap6F/CuJCdU1XtGVJMkaUxmNXR2Vb0nya8DywbXqaoPDqkuSdIYzCoUkvxv4JHARcCWL78pwFCQpAVktl+yswJYXlU1zGIkSeM12+cUvgn88jALkSSN32yPFJYAlyW5ALh9S2NVvWAuO03y34A/oDsFdQnwUrpbXz9Kd93iGuDoqrplLtuXJM3NbEPhz7fXDpPsC/xXutNRP0myFlgFLAfOrapTk5wInAi8bnvtV5K0bbO9++j8Iez3AUl+RneE8F3gJODw1n8mcB6GgiSN1GyHufhhklvbz0+T3JXk1rnssKquA94GfAe4HvhBVX0W2Luqrm+L3QDsPZftS5LmbrZHCg/ZMp0kwErgKXPZYRsuYyVwIPB94GNJXjxlf5Vk2judkhwHHAew//77z6UESdIM7vUoqdX5JPCcOe7zWcDVVTVZVT8DPg78OnBjkn0A2r+bZ9j/mqpaUVUrJiYm5liCJGk6s3147bcGZneie27hp3Pc53eApyR5IPAT4AhgA3AbsBo4tf17zhy3L0mao9neffSfB6bvpLtldOVcdlhVX0ny98DX2ra+DqwBHgysTXIssBE4ei7blyTN3WyvKbx0e+60qk4GTp7SfDvdUYMkaUxme/fR0iSfSLK5/fxDkqXDLk6SNFqzvdD8AWAd3fcqPBz4P61NkrSAzDYUJqrqA1V1Z/s5A/DWH0laYGYbCjcleXGSndvPi4GbhlmYJGn0ZhsKL6O7G+gGuqeQfwd4yZBqkiSNyWxvSX0jsHrLqKVJ9qIbquJlwypMkjR6sz1SeOzgMNZVdTPwhOGUJEkal9mGwk5tzCKgP1KY7VGGJGkHMds39v8FfCnJx9r87wKnDKckSdK4zPaJ5g8m2QA8szX9VlVdNryyJEnjMOtTQC0EDAJJWsDu9dDZkqSFy1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSbyyhkGSPJH+f5FtJLk9yaJK9knwuyZXt3z23vSVJ0vY0riOFdwGfrqr/ADwOuBw4ETi3qg4Czm3zkqQRGnkoJNkdeBrwfoCquqOqvg+sBM5si50JHDXq2iRpsRvHkcKBwCTwgSRfT/J3SR4E7F1V17dlbgD2HkNtkrSojSMUdgGeCPxtVT0BuI0pp4qqqoCabuUkxyXZkGTD5OTk0IuVpMVkHKGwCdhUVV9p839PFxI3JtkHoP27ebqVq2pNVa2oqhUTExMjKViSFouRh0JV3QBcm+TRrekIutFX1wGrW9tq4JxR1yZJi924vj3tBODDSe4PfBt4KV1ArU1yLLAROHpMtUnSojWWUKiqi4AV03QdMeJSJEkDfKJZktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJvXF9RzNJdgY2ANdV1fOT7AV8FFgGXAMcXVW3jKs+zT/LTvzUuEtYsK459XnjLkHzxDiPFF4FXD4wfyJwblUdBJzb5iVJIzSWUEiyFHge8HcDzSuBM9v0mcBRIy5Lkha9cR0pvBN4LXD3QNveVXV9m74B2HvURUnSYjfyUEjyfGBzVV040zJVVUDNsP5xSTYk2TA5OTmsMiVpURrHkcJTgRckuQY4G3hmkg8BNybZB6D9u3m6latqTVWtqKoVExMTo6pZkhaFkYdCVZ1UVUurahmwCvh8Vb0YWAesboutBs4ZdW2StNjNp+cUTgV+I8mVwLPavCRphMb2nAJAVZ0HnNembwKOGGc9krTYzacjBUnSmBkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6o08FJLsl+Sfk1yW5NIkr2rteyX5XJIr2797jro2SVrsxnGkcCfw6qpaDjwFOD7JcuBE4NyqOgg4t81LkkZo5KFQVddX1dfa9A+By4F9gZXAmW2xM4GjRl2bJC12Y72mkGQZ8ATgK8DeVXV967oB2HuGdY5LsiHJhsnJydEUKkmLxNhCIcmDgX8A/riqbh3sq6oCarr1qmpNVa2oqhUTExMjqFSSFo+xhEKS+9EFwoer6uOt+cYk+7T+fYDN46hNkhazcdx9FOD9wOVV9faBrnXA6ja9Gjhn1LVJ0mK3yxj2+VTg94BLklzU2l4PnAqsTXIssBE4egy1SdKiNvJQqKovAJmh+4hR1iJJuiefaJYk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVJvHN+nIGmRWHbip8ZdwoJ1zanPG8p2PVKQJPUMBUlSz1CQJPXmXSgkOTLJFUmuSnLiuOuRpMVkXoVCkp2BvwZ+E1gOHJNk+XirkqTFY16FAnAIcFVVfbuq7gDOBlaOuSZJWjTm2y2p+wLXDsxvAn5tcIEkxwHHtdkfJbliRLWN2xLge+MuYrby5nFXMC/sMK+ZrxewA71ecJ9fswNm6phvobBNVbUGWDPuOkYtyYaqWjHuOjR7vmY7Fl+vznw7fXQdsN/A/NLWJkkagfkWCl8FDkpyYJL7A6uAdWOuSZIWjXl1+qiq7kzyR8BngJ2B06vq0jGXNV8sulNmC4Cv2Y7F1wtIVY27BknSPDHfTh9JksbIUJAk9QyFeSDJj8ZdgzpJKsmHBuZ3STKZ5P/ey+2cl2RFm/7HJHts51I1S1P/fyV5SZL3tulXJPn9bazfL78YzKsLzdI8cBvwmCQPqKqfAL/Bfbwtuqqeu10q03ZXVaeNu4b5xiOFeSrJ45N8OcnFST6RZM8kD0tyYet/XPtUu3+b//ckDxxv1QvGPwJbvsHkGOCsLR1JHpTk9CQXJPl6kpWt/QFJzk5yeZJPAA8YWOeaJEuSLEvyzYH21yT58zZ9XpJ3JNnQtvHkJB9PcmWSN43gd16Ukvx5kte06Se3/28XJXnr4GsFPDzJp9vr8ZYxlTsShsL89UHgdVX1WOAS4OSq2gzsluShwGHABuCwJAcAm6vqx+Mrd0E5G1iVZDfgscBXBvr+FPh8VR0CPAN4a5IHAa8EflxVvwqcDDxpDvu9oz1RexpwDnA88BjgJUl+ac6/jR7Q3ugvSnIR8MYZlvsA8PKqejxw15S+xwMvBP4j8MIk+7FAefpoHkqyO7BHVZ3fms4EPtamvwg8FXga8FfAkUCAfxl1nQtVVV2cZBndUcI/Tul+NvCCLZ8ugd2A/elej3cPrH/xHHa95UHNS4BLq+p6gCTfpnvS/6Y5bFPwk/ZGD3TXCIB7DGfRrvk8pKq+1Jo+Ajx/YJFzq+oHbdnL6MYOGhynbcEwFHY86+mOEg6g+zT5OqAAvwx3+1oHvA04HBj8lB7gt6vqHgMxJpnNNu/knkfnu03pv739e/fA9JZ5/6+O1+DrcRcL+PXw9NE81D6R3JLksNb0e8CWo4Z/AV4MXFlVdwM3A88FvjDyQhe204G/qKpLprR/BjghLQWSPKG1rwf+S2t7DN1pp6luBB6W5JeS7Mo9P4lqjKrq+8APk2wZlXnVGMsZqwWbdjuYBybZNDD/dmA1cFq7ePxt4KUAVXVNe0Na35b9ArC0qm4ZZcELXVVtop0OmuIvgXcCFyfZCbia7s39b4EPJLkcuBy4cJpt/izJG4EL6O5o+tZwqtccHQu8L8nddB/CfjDmesbCYS4kCUjy4Kr6UZs+Edinql415rJGziMFSeo8L8lJdO+LG4GXjLec8fBIQZLU80KzJKlnKEiSeoaCJKlnKGhRmDrukKTpGQrSNiTZIe7S21Hq1PxmKGgx2TnJ+5JcmuSzbWTTXxiNFvpRS9+ZZAPwqiS/m+SbSb6RZH1bZuc2muZX2/ovb+2HJ1mf5FNJrkhyWnvQjSTHJLmkbevNre13k7y9Tb+qjXVEkkck+dc2/aQk5ye5MMlnkuwzXZ2j/XNqIfKThRaTg4BjquoPk6wFfht4LXBCVZ3fnjY+Gfjjtvz926ilJLkEeE5VXZeff2HOscAPqurJbdiKf03y2dZ3CLCc7n73TwO/leSLwJvpRlC9BfhskqPohi55bVvvMOCmJPu26fVJ7ge8B1hZVZNJXgicArxsap3SfWUoaDG5uqouatMXAo9k5tFoAT46MP2vwBktTD7e2p4NPDbJ77T53emC5w7ggqra8on/LOA/AT8Dzquqydb+YeBpVfXJJA9O8hC60VA/Qjfq6mFtX4+mG0L7c23IpZ2B62eoU7pPDAUtJlNHutxjG8vftmWiql7RBkt7HnBhkifRjZh6QlV9ZnClJIfTjVw7aFtPiX6RbnyrK+iOHF4GHAq8mm5o7kur6tBt1SndV15T0GK2tdFo7yHJI6vqK1X1BmCS7hP9Z4BXttM7JHlU+8IdgEOSHNiuJbyQbuDCC4Cnp/sWtp3pvq9hcPTb19ANdPh1ui/wub2NmHsFMJHk0Laf+yU5ePv9GaSf80hBi920o9FO461JDqI7OjgX+AZwMbAM+FobuXYSOKot/1XgvcCvAP8MfKKq7m4Drf1z286nquqctvy/0AXN+qq6K8m1tFFUq+qOdorq3e0LmHahG6n10u3yF5AGOPaRtJ2100evqSq/L0E7HE8fSZJ6HilIknoeKUiSeoaCJKlnKEiSeoaCJKlnKEiSev8fv+2Gj297XtMAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<p>\n    Look at the dataframe above carefully. You will find that the last column provides the bins for \"horsepower\" based on 3 categories (\"Low\", \"Medium\" and \"High\"). \n</p>\n<p>\n    We successfully narrowed down the intervals from 59 to 3!\n</p>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h3>Bins Visualization</h3>\nNormally, a histogram is used to visualize the distribution of bins we created above. \n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "%matplotlib inline\nimport matplotlib as plt\nfrom matplotlib import pyplot\n\n\n# draw historgram of attribute \"horsepower\" with bins = 3\nplt.pyplot.hist(df[\"horsepower\"], bins = 3)\n\n# set x/y labels and plot title\nplt.pyplot.xlabel(\"horsepower\")\nplt.pyplot.ylabel(\"count\")\nplt.pyplot.title(\"horsepower bins\")",
      "metadata": {
        "trusted": true
      },
      "execution_count": 48,
      "outputs": [
        {
          "execution_count": 48,
          "output_type": "execute_result",
          "data": {
            "text/plain": "Text(0.5, 1.0, 'horsepower bins')"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 432x288 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYZ0lEQVR4nO3dfbRddX3n8feHoPgA8tBcUyRgIgVnwGUVb1FqVRRH8GEMYysNS52ItNEuZHRGa4NOtdNVZlBbH+pDXbEgWCkYH5DMco2CUYhaAYMijyIpJJAI5Aoo1iqP3/lj72wP13uTS8g55ybn/VrrrLv377f32d/7y8n9nL3POb+TqkKSJIBdhl2AJGn2MBQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQbNSknVJXjzsOma7JEcm2bCF/k8k+ctB1qQd267DLkBS/1TVm4Zdg3Ysnilop5Zkp3niszP9Lpq9DAXNZs9IcmWSnyX5bJLHbO5I8qdJ1ia5M8nKJE/q6askJyW5AbghjQ8m2ZTk7iRXJXlau+1uSf42yc1Jbm8vtzy27TsyyYYk70zyk/aS1mt6jrNnkk8nmUiyPsn/TLJL27c+ybPa5de0NR3arp+Y5Evt8i5JliX51yR3JFmRZJ+2b0G734lJbga+Pt1AbaHGM5P8zaTf523tWNya5ISebV+W5NokP0+yMcnbH8G/nXZQhoJms+OAY4CFwNOB1wMkeRHwf9r+fYH1wLmT9j0WeDZwCPAS4PnAwcCe7X53tNud1rY/A/gdYD/g3T3389vA3LZ9CbA8yVPbvo+09/cU4AXAfwU2/5G9GDiyXX4BcGNbw+b1i9vlk9taXwA8CbgL+Nik3+UFwH8Ejp48QDOocapt92y3PRH4WJK9277TgTdW1R7A09hCCGknVlXevM26G7AOeG3P+vuAT7TLpwPv6+nbHbgPWNCuF/Cinv4XAT8CngPs0tMe4BfAgT1tRwA3tctHAvcDj+/pXwH8JTAHuBc4pKfvjcBF7fKJwMp2+TrgT4Bz2/X1wGE9fUf13Me+7e+yK7Cg/V2esoVxmrbGdvlM4G96tv0lsGvPtpuA57TLN7e/wxOG/e/vbXg3zxQ0m93Ws/zvNH/8oXlGvX5zR1X9G80z//16tr+lp//rwEdpnoFvSrI8yROAMeBxwOVJfprkp8BX2vbN7qqqX/Ssr2+PPxd4VG8d7fLmGi4GnpdkX5oAWQE8N8kCmmfqV7TbPRk4r+f41wEPAPOm+l2mMV2NU7mjqu7vWe8d1z8EXgasT3JxkiO2clzthAwF7Yh+TPPHFIAkjwd+C9jYs81Dpv+tqr+vqmfRXE46GPhz4Cc0z5wPraq92tueVbV7z657t/e/2QHt8X9C84z+yZP6NrbHW0vzB/dkYHVV3U0TckuBb1XVg+0+twAv7Tn+XlX1mKqa9neZwnQ1PixV9d2qWgQ8EfgSTZBpxBgK2hGdA5yQ5BlJdgP+N3BpVa2bauMkv5fk2UkeRXO56FfAg+0f5k8CH0zyxHbb/ZJMvnb/v5I8OsnzgFcAn6uqB2j+aJ6aZI8kTwb+B/CZnv0uBt7Mr18/uGjSOsAn2vt4cnv8sSSLtmFMfqPGh7Nzu+9rkuxZVfcBdwMPbm0/7XwMBe1wquprNNf1vwDcChwILN7CLk+g+eN/F82llTuA97d9fwGsBS5JcjfwNaD3Rdrb2v1+DJwNvKmqftj2nUwTMjcC3wL+GTijZ9+LgT2A1dOsA3wYWAlckOTnwCU0L5A/HFuq8eF4HbCuHYc3Aa/ZyvbaCaXKL9mRppLkSOAzVTV/yKVIA+OZgiSpYyhIkjp9C4UkZ7Sfmrx6UvvJSX6Y5Jok7+tpP6X9hOr1U7zQJw1cVV3kpSONmn7OpXImzXvDP725IckLgUXA71bVPT3v+DiE5oXCQ2neX/21JAe37/CQJA1I30Khqla3H9Tp9WfAaVV1T7vNprZ9Ec2nPe8BbkqyFjgc+M6WjjF37txasGDyISRJW3L55Zf/pKrGpuob9KyLB9N8yvNUmveKv72qvkvzKdBLerbbwEM/ndpJspTmA0AccMABrFmzpr8VS9JOJsn66foG/ULzrsA+NHPQ/DmwIkkezh1U1fKqGq+q8bGxKYNOkrSNBh0KG4AvVuMymk9MzqWZGmD/nu3m89ApCyRJAzDoUPgS8EKAJAcDj6aZQ2YlsLid234hcBBw2YBrk6SR17fXFJKcQzNV79w03yH7HpopAM5o36Z6L7Ckmo9UX5NkBXAtzTTAJ/nOI0kavB16movx8fHyhWZJeniSXF5V41P1+YlmSVLHUJAkdQwFSVLHUJAkdQb9ieZZZcGyLw+7BE2y7rSXD7sEaaR5piBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6vQtFJKckWRT+33Mk/velqSSzO1pOyXJ2iTXJzm6X3VJkqbXzzOFM4FjJjcm2R94CXBzT9shwGLg0HafjyeZ08faJElT6FsoVNVq4M4puj4IvAOonrZFwLlVdU9V3QSsBQ7vV22SpKkN9DWFJIuAjVX1g0ld+wG39KxvaNumuo+lSdYkWTMxMdGnSiVpNA0sFJI8Dngn8O5Hcj9VtbyqxqtqfGxsbPsUJ0kCBvt1nAcCC4EfJAGYD3wvyeHARmD/nm3nt22SpAEa2JlCVV1VVU+sqgVVtYDmEtFhVXUbsBJYnGS3JAuBg4DLBlWbJKnRz7ekngN8B3hqkg1JTpxu26q6BlgBXAt8BTipqh7oV22SpKn17fJRVR2/lf4Fk9ZPBU7tVz2SpK3zE82SpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnq9PM7ms9IsinJ1T1t70/ywyRXJjkvyV49fackWZvk+iRH96suSdL0+nmmcCZwzKS2C4GnVdXTgR8BpwAkOQRYDBza7vPxJHP6WJskaQp9C4WqWg3cOantgqq6v129BJjfLi8Czq2qe6rqJmAtcHi/apMkTW2Yrym8Afh/7fJ+wC09fRvatt+QZGmSNUnWTExM9LlESRotQwmFJO8C7gfOfrj7VtXyqhqvqvGxsbHtX5wkjbBdB33AJK8HXgEcVVXVNm8E9u/ZbH7bJkkaoIGeKSQ5BngH8Mqq+veerpXA4iS7JVkIHARcNsjaJEl9PFNIcg5wJDA3yQbgPTTvNtoNuDAJwCVV9aaquibJCuBamstKJ1XVA/2qTZI0tb6FQlUdP0Xz6VvY/lTg1H7VI0naOj/RLEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpE7fQiHJGUk2Jbm6p22fJBcmuaH9uXdP3ylJ1ia5PsnR/apLkjS9fp4pnAkcM6ltGbCqqg4CVrXrJDkEWAwc2u7z8SRz+libJGkKfQuFqloN3DmpeRFwVrt8FnBsT/u5VXVPVd0ErAUO71dtkqSpDfo1hXlVdWu7fBswr13eD7ilZ7sNbdtvSLI0yZokayYmJvpXqSSNoKG90FxVBdQ27Le8qsaranxsbKwPlUnS6Bp0KNyeZF+A9uemtn0jsH/PdvPbNknSAA06FFYCS9rlJcD5Pe2Lk+yWZCFwEHDZgGuTpJG3a7/uOMk5wJHA3CQbgPcApwErkpwIrAeOA6iqa5KsAK4F7gdOqqoH+lWbJGlqfQuFqjp+mq6jptn+VODUftUjSdo6P9EsSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkzoxCIcmqmbRJknZsW5z7KMljgMfRTGq3N5C26wlM8yU4kqQd19YmxHsj8FbgScDl/DoU7gY+2r+yJEnDsMVQqKoPAx9OcnJVfWRANUmShmRGU2dX1UeS/D6woHefqvp0n+qSJA3BjEIhyT8BBwJXAJu//KYAQ0GSdiIz/ZKdceCQqqp+FiNJGq6Zfk7hauC3+1mIJGn4ZnqmMBe4NsllwD2bG6vqldty0CT/HfgTmktQVwEn0Lz19bM0r1usA46rqru25f4lSdtmpqHwV9vrgEn2A/4bzeWoXyZZASwGDgFWVdVpSZYBy4C/2F7HlSRt3UzffXRxH4772CT30Zwh/Bg4BTiy7T8LuAhDQZIGaqbTXPw8yd3t7VdJHkhy97YcsKo2An8L3AzcCvysqi4A5lXVre1mtwHztuX+JUnbbqZnCntsXk4SYBHwnG05YDtdxiJgIfBT4HNJXjvpeJVkync6JVkKLAU44IADtqUESdI0HvYsqdX4EnD0Nh7zxcBNVTVRVfcBXwR+H7g9yb4A7c9N0xx/eVWNV9X42NjYNpYgSZrKTD+89qqe1V1oPrfwq2085s3Ac5I8DvglcBSwBvgFsAQ4rf15/jbevyRpG8303Uf/uWf5fpq3jC7algNW1aVJPg98r72v7wPLgd2BFUlOBNYDx23L/UuStt1MX1M4YXsetKreA7xnUvM9NGcNkqQhmem7j+YnOS/Jpvb2hSTz+12cJGmwZvpC86eAlTTfq/Ak4P+2bZKknchMQ2Gsqj5VVfe3tzMB3/ojSTuZmYbCHUlem2ROe3stcEc/C5MkDd5MQ+ENNO8Guo3mU8h/BLy+TzVJkoZkpm9J/WtgyeZZS5PsQzNVxRv6VZgkafBmeqbw9N5prKvqTuCZ/SlJkjQsMw2FXdo5i4DuTGGmZxmSpB3ETP+w/x3wnSSfa9dfDZzan5IkScMy0080fzrJGuBFbdOrqura/pUlSRqGGV8CakPAIJCkndjDnjpbkrTzMhQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUGUooJNkryeeT/DDJdUmOSLJPkguT3ND+3Hvr9yRJ2p6GdabwYeArVfUfgN8FrgOWAauq6iBgVbsuSRqggYdCkj2B5wOnA1TVvVX1U2ARcFa72VnAsYOuTZJG3TDOFBYCE8Cnknw/yT8meTwwr6pubbe5DZg3hNokaaQNIxR2BQ4D/qGqngn8gkmXiqqqgJpq5yRLk6xJsmZiYqLvxUrSKBlGKGwANlTVpe3652lC4vYk+wK0PzdNtXNVLa+q8aoaHxsbG0jBkjQqBh4KVXUbcEuSp7ZNR9HMvroSWNK2LQHOH3RtkjTqhvXtaScDZyd5NHAjcAJNQK1IciKwHjhuSLVJ0sgaSihU1RXA+BRdRw24FElSDz/RLEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpM6wvqOZJHOANcDGqnpFkn2AzwILgHXAcVV117Dq03AsWPblYZegSdad9vJhl6ABGuaZwluA63rWlwGrquogYFW7LkkaoKGEQpL5wMuBf+xpXgSc1S6fBRw74LIkaeQN60zhQ8A7gAd72uZV1a3t8m3AvEEXJUmjbuChkOQVwKaquny6baqqgJpm/6VJ1iRZMzEx0a8yJWkkDeNM4bnAK5OsA84FXpTkM8DtSfYFaH9ummrnqlpeVeNVNT42NjaomiVpJAw8FKrqlKqaX1ULgMXA16vqtcBKYEm72RLg/EHXJkmjbjZ9TuE04D8luQF4cbsuSRqgoX1OAaCqLgIuapfvAI4aZj2SNOpm05mCJGnIDAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1Bh4KSfZP8o0k1ya5Jslb2vZ9klyY5Ib2596Drk2SRt0wzhTuB95WVYcAzwFOSnIIsAxYVVUHAavadUnSAA08FKrq1qr6Xrv8c+A6YD9gEXBWu9lZwLGDrk2SRt1QX1NIsgB4JnApMK+qbm27bgPmTbPP0iRrkqyZmJgYTKGSNCKGFgpJdge+ALy1qu7u7auqAmqq/apqeVWNV9X42NjYACqVpNExlFBI8iiaQDi7qr7YNt+eZN+2f19g0zBqk6RRNox3HwU4Hbiuqj7Q07USWNIuLwHOH3RtkjTqdh3CMZ8LvA64KskVbds7gdOAFUlOBNYDxw2hNkkaaQMPhar6FpBpuo8aZC2SpIfyE82SpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqDOP7FCTtQBYs+/KwS9AU1p328r7cr2cKkqSOoSBJ6hgKkqTOrAuFJMckuT7J2iTLhl2PJI2SWRUKSeYAHwNeChwCHJ/kkOFWJUmjY1aFAnA4sLaqbqyqe4FzgUVDrkmSRsZse0vqfsAtPesbgGf3bpBkKbC0Xf23JNdvh+POBX6yHe5nZ+X4TM+xmZ5jM71HPDZ57yM6/pOn65htobBVVbUcWL497zPJmqoa3573uTNxfKbn2EzPsZnebB6b2Xb5aCOwf8/6/LZNkjQAsy0UvgsclGRhkkcDi4GVQ65JkkbGrLp8VFX3J3kz8FVgDnBGVV0zgENv18tROyHHZ3qOzfQcm+nN2rFJVQ27BknSLDHbLh9JkobIUJAkdUYyFJKsS3JVkiuSrGnb9klyYZIb2p97D7vOQUhyRpJNSa7uaZt2LJKc0k5Bcn2So4dT9WBMMzZ/lWRj+9i5IsnLevpGaWz2T/KNJNcmuSbJW9r2kX/sbGFsdozHTlWN3A1YB8yd1PY+YFm7vAx477DrHNBYPB84DLh6a2NBM/XID4DdgIXAvwJzhv07DHhs/gp4+xTbjtrY7Asc1i7vAfyoHYORf+xsYWx2iMfOSJ4pTGMRcFa7fBZw7PBKGZyqWg3cOal5urFYBJxbVfdU1U3AWpqpSXZK04zNdEZtbG6tqu+1yz8HrqOZkWDkHztbGJvpzKqxGdVQKOCCJJe302YAzKuqW9vl24B5wyltVphuLKaahmRLD/ad1ZuTXNleXtp8eWRkxybJAuCZwKX42HmISWMDO8BjZ1RD4Q+q6jCa2VhPSvL83s5qzul8ry6OxRT+ATgQeAZwK/B3Q61myJLsDnwBeGtV3d3bN+qPnSnGZod47IxkKFTVxvbnJuA8mlO125PsC9D+3DS8CoduurEY+WlIqur2qnqgqh4EPsmvT/NHbmySPIrmj97ZVfXFttnHDlOPzY7y2Bm5UEjy+CR7bF4GXgJcTTOdxpJ2syXA+cOpcFaYbixWAouT7JZkIXAQcNkQ6huazX/wWv+F5rEDIzY2SQKcDlxXVR/o6Rr5x850Y7PDPHaG/Ur9oG/AU2he6f8BcA3wrrb9t4BVwA3A14B9hl3rgMbjHJpT2ftormWeuKWxAN5F8+6I64GXDrv+IYzNPwFXAVfS/Gfed0TH5g9oLg1dCVzR3l7mY2eLY7NDPHac5kKS1Bm5y0eSpOkZCpKkjqEgSeoYCpKkjqEgSeoYChoJSRb0znYqaWqGgrQVSWbV19ZOZ0epU7OboaBRMifJJ9s57i9I8tgkz0hySTtJ2XmbJylLclGSD7Xft/GWJK9OcnWSHyRZ3W4zJ8n7k3y33f+NbfuRSVYn+XI7P/4nkuzS9h2f5rs8rk7y3rbt1Uk+0C6/JcmN7fJTkny7XX5WkovbSRy/2jOVxEPqHOxwamfkMwuNkoOA46vqT5OsAP4QeAdwclVdnOSvgfcAb223f3RVjQMkuQo4uqo2Jtmr7T8R+FlV/V6S3YBvJ7mg7TucZp789cBXgFcl+RfgvcCzgLtoZuo9FvhmWwfA84A7kuzXLq9u59H5CLCoqiaS/DFwKvCGyXVKj5ShoFFyU1Vd0S5fTjNj5V5VdXHbdhbwuZ7tP9uz/G3gzDZMNk/+9hLg6Un+qF3fkyZ47gUuq6rNz/jPoZn64D7goqqaaNvPBp5fVV9Ksns7J9f+wD/TfMHP89pjPRV4GnBhM60Oc2im35iqTukRMRQ0Su7pWX4A2Gsr2/9i80JVvSnJs4GXA5cneRYQmrOMr/bulORIfnPK6K3NJ/MvwAk0c998k+Ys4AjgbcABwDVVdcTW6pQeKV9T0Cj7GXBXkue1668DLp5qwyQHVtWlVfVuYILmGf1XgT9rL++Q5OB25l2Aw5MsbF9L+GPgWzQzX74gydwkc4Dje473TeDtwGrg+8ALgXuq6mc0QTGW5Ij2OI9Kcuj2Gwbp1zxT0KhbAnwiyeOAG2merU/l/UkOojk7WEUzy+6VwALge+10yRP8+usnvwt8FPgd4BvAeVX1YJJl7XqAL1fV5qmlv0kTNKur6oEktwA/BKiqe9tLVH+fZE+a/7cfopnlV9qunCVV2s7ay0dvr6pXDLkU6WHz8pEkqeOZgiSp45mCJKljKEiSOoaCJKljKEiSOoaCJKnz/wHFBJTx1+86fQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "The plot above shows the binning result for the attribute \"horsepower\".\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<h2 id=\"indicator\">Indicator Variable (or Dummy Variable)</h2>\n<b>What is an indicator variable?</b>\n<p>\n    An indicator variable (or dummy variable) is a numerical variable used to label categories. They are called 'dummies' because the numbers themselves don't have inherent meaning. \n</p>\n\n<b>Why we use indicator variables?</b>\n\n<p>\n    We use indicator variables so we can use categorical variables for regression analysis in the later modules.\n</p>\n<b>Example</b>\n<p>\n    We see the column \"fuel-type\" has two unique values: \"gas\" or \"diesel\". Regression doesn't understand words, only numbers. To use this attribute in regression analysis, we convert \"fuel-type\" to indicator variables.\n</p>\n\n<p>\n    We will use pandas' method 'get_dummies' to assign numerical values to different categories of fuel type. \n</p>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df.columns",
      "metadata": {
        "trusted": true
      },
      "execution_count": 49,
      "outputs": [
        {
          "execution_count": 49,
          "output_type": "execute_result",
          "data": {
            "text/plain": "Index(['symboling', 'normalized-losses', 'make', 'fuel-type', 'aspiration',\n       'num-of-doors', 'body-style', 'drive-wheels', 'engine-location',\n       'wheel-base', 'length', 'width', 'height', 'curb-weight', 'engine-type',\n       'num-of-cylinders', 'engine-size', 'fuel-system', 'bore', 'stroke',\n       'compression-ratio', 'horsepower', 'peak-rpm', 'city-mpg',\n       'highway-L/100km', 'price', 'city-L/100km', 'horsepower-binned'],\n      dtype='object')"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "Get the indicator variables and assign it to data frame \"dummy_variable\\_1\":\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "dummy_variable_1 = pd.get_dummies(df[\"fuel-type\"])\ndummy_variable_1.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 50,
      "outputs": [
        {
          "execution_count": 50,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   diesel  gas\n0       0    1\n1       0    1\n2       0    1\n3       0    1\n4       0    1",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>diesel</th>\n      <th>gas</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "Change the column names for clarity:\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "dummy_variable_1.rename(columns={'gas':'fuel-type-gas', 'diesel':'fuel-type-diesel'}, inplace=True)\ndummy_variable_1.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 51,
      "outputs": [
        {
          "execution_count": 51,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   fuel-type-diesel  fuel-type-gas\n0                 0              1\n1                 0              1\n2                 0              1\n3                 0              1\n4                 0              1",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>fuel-type-diesel</th>\n      <th>fuel-type-gas</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "In the dataframe, column 'fuel-type' has values for 'gas' and 'diesel' as 0s and 1s now.\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "# merge data frame \"df\" and \"dummy_variable_1\" \ndf = pd.concat([df, dummy_variable_1], axis=1)\n\n# drop original column \"fuel-type\" from \"df\"\ndf.drop(\"fuel-type\", axis = 1, inplace=True)",
      "metadata": {
        "trusted": true
      },
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": "df.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 53,
      "outputs": [
        {
          "execution_count": 53,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   symboling  normalized-losses         make aspiration num-of-doors  \\\n0          3                122  alfa-romero        std          two   \n1          3                122  alfa-romero        std          two   \n2          1                122  alfa-romero        std          two   \n3          2                164         audi        std         four   \n4          2                164         audi        std         four   \n\n    body-style drive-wheels engine-location  wheel-base    length  ...  \\\n0  convertible          rwd           front        88.6  0.811148  ...   \n1  convertible          rwd           front        88.6  0.811148  ...   \n2    hatchback          rwd           front        94.5  0.822681  ...   \n3        sedan          fwd           front        99.8  0.848630  ...   \n4        sedan          4wd           front        99.4  0.848630  ...   \n\n   compression-ratio  horsepower  peak-rpm city-mpg highway-L/100km    price  \\\n0                9.0         111    5000.0       21        8.703704  13495.0   \n1                9.0         111    5000.0       21        8.703704  16500.0   \n2                9.0         154    5000.0       19        9.038462  16500.0   \n3               10.0         102    5500.0       24        7.833333  13950.0   \n4                8.0         115    5500.0       18       10.681818  17450.0   \n\n  city-L/100km  horsepower-binned  fuel-type-diesel  fuel-type-gas  \n0    11.190476                Low                 0              1  \n1    11.190476                Low                 0              1  \n2    12.368421             Medium                 0              1  \n3     9.791667                Low                 0              1  \n4    13.055556                Low                 0              1  \n\n[5 rows x 29 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>symboling</th>\n      <th>normalized-losses</th>\n      <th>make</th>\n      <th>aspiration</th>\n      <th>num-of-doors</th>\n      <th>body-style</th>\n      <th>drive-wheels</th>\n      <th>engine-location</th>\n      <th>wheel-base</th>\n      <th>length</th>\n      <th>...</th>\n      <th>compression-ratio</th>\n      <th>horsepower</th>\n      <th>peak-rpm</th>\n      <th>city-mpg</th>\n      <th>highway-L/100km</th>\n      <th>price</th>\n      <th>city-L/100km</th>\n      <th>horsepower-binned</th>\n      <th>fuel-type-diesel</th>\n      <th>fuel-type-gas</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>3</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>0.811148</td>\n      <td>...</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000.0</td>\n      <td>21</td>\n      <td>8.703704</td>\n      <td>13495.0</td>\n      <td>11.190476</td>\n      <td>Low</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>std</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>0.811148</td>\n      <td>...</td>\n      <td>9.0</td>\n      <td>111</td>\n      <td>5000.0</td>\n      <td>21</td>\n      <td>8.703704</td>\n      <td>16500.0</td>\n      <td>11.190476</td>\n      <td>Low</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>std</td>\n      <td>two</td>\n      <td>hatchback</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>94.5</td>\n      <td>0.822681</td>\n      <td>...</td>\n      <td>9.0</td>\n      <td>154</td>\n      <td>5000.0</td>\n      <td>19</td>\n      <td>9.038462</td>\n      <td>16500.0</td>\n      <td>12.368421</td>\n      <td>Medium</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>fwd</td>\n      <td>front</td>\n      <td>99.8</td>\n      <td>0.848630</td>\n      <td>...</td>\n      <td>10.0</td>\n      <td>102</td>\n      <td>5500.0</td>\n      <td>24</td>\n      <td>7.833333</td>\n      <td>13950.0</td>\n      <td>9.791667</td>\n      <td>Low</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>std</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>4wd</td>\n      <td>front</td>\n      <td>99.4</td>\n      <td>0.848630</td>\n      <td>...</td>\n      <td>8.0</td>\n      <td>115</td>\n      <td>5500.0</td>\n      <td>18</td>\n      <td>10.681818</td>\n      <td>17450.0</td>\n      <td>13.055556</td>\n      <td>Low</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 29 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "The last two columns are now the indicator variable representation of the fuel-type variable. They're all 0s and 1s now.\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "<div class=\"alert alert-danger alertdanger\" style=\"margin-top: 20px\">\n<h1> Question  #4: </h1>\n\n<b>Similar to before, create an indicator variable for the column \"aspiration\"</b>\n\n</div>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "# Write your code below and press Shift+Enter to execute \ndummy_aspiration = pd.get_dummies(df[\"aspiration\"])\ndummy_aspiration.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 54,
      "outputs": [
        {
          "execution_count": 54,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   std  turbo\n0    1      0\n1    1      0\n2    1      0\n3    1      0\n4    1      0",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>std</th>\n      <th>turbo</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": "dummy_aspiration.rename(columns={'std':'aspiration-std', 'turbo': 'aspiration-turbo'}, inplace=True)\ndummy_aspiration.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 55,
      "outputs": [
        {
          "execution_count": 55,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   aspiration-std  aspiration-turbo\n0               1                 0\n1               1                 0\n2               1                 0\n3               1                 0\n4               1                 0",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>aspiration-std</th>\n      <th>aspiration-turbo</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<details><summary>Click here for the solution</summary>\n\n```python\n# get indicator variables of aspiration and assign it to data frame \"dummy_variable_2\"\ndummy_variable_2 = pd.get_dummies(df['aspiration'])\n\n# change column names for clarity\ndummy_variable_2.rename(columns={'std':'aspiration-std', 'turbo': 'aspiration-turbo'}, inplace=True)\n\n# show first 5 instances of data frame \"dummy_variable_1\"\ndummy_variable_2.head()\n\n\n```\n\n</details>\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": " <div class=\"alert alert-danger alertdanger\" style=\"margin-top: 20px\">\n<h1> Question  #5: </h1>\n\n<b>Merge the new dataframe to the original dataframe, then drop the column 'aspiration'.</b>\n\n</div>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "# Write your code below and press Shift+Enter to execute \n\ndf = pd.concat([df, dummy_aspiration], axis=1)\ndf.drop('aspiration', axis = 1, inplace=True)\ndf.head()",
      "metadata": {
        "trusted": true
      },
      "execution_count": 56,
      "outputs": [
        {
          "execution_count": 56,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   symboling  normalized-losses         make num-of-doors   body-style  \\\n0          3                122  alfa-romero          two  convertible   \n1          3                122  alfa-romero          two  convertible   \n2          1                122  alfa-romero          two    hatchback   \n3          2                164         audi         four        sedan   \n4          2                164         audi         four        sedan   \n\n  drive-wheels engine-location  wheel-base    length     width  ...  peak-rpm  \\\n0          rwd           front        88.6  0.811148  0.890278  ...    5000.0   \n1          rwd           front        88.6  0.811148  0.890278  ...    5000.0   \n2          rwd           front        94.5  0.822681  0.909722  ...    5000.0   \n3          fwd           front        99.8  0.848630  0.919444  ...    5500.0   \n4          4wd           front        99.4  0.848630  0.922222  ...    5500.0   \n\n   city-mpg highway-L/100km    price  city-L/100km horsepower-binned  \\\n0        21        8.703704  13495.0     11.190476               Low   \n1        21        8.703704  16500.0     11.190476               Low   \n2        19        9.038462  16500.0     12.368421            Medium   \n3        24        7.833333  13950.0      9.791667               Low   \n4        18       10.681818  17450.0     13.055556               Low   \n\n   fuel-type-diesel  fuel-type-gas  aspiration-std  aspiration-turbo  \n0                 0              1               1                 0  \n1                 0              1               1                 0  \n2                 0              1               1                 0  \n3                 0              1               1                 0  \n4                 0              1               1                 0  \n\n[5 rows x 30 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>symboling</th>\n      <th>normalized-losses</th>\n      <th>make</th>\n      <th>num-of-doors</th>\n      <th>body-style</th>\n      <th>drive-wheels</th>\n      <th>engine-location</th>\n      <th>wheel-base</th>\n      <th>length</th>\n      <th>width</th>\n      <th>...</th>\n      <th>peak-rpm</th>\n      <th>city-mpg</th>\n      <th>highway-L/100km</th>\n      <th>price</th>\n      <th>city-L/100km</th>\n      <th>horsepower-binned</th>\n      <th>fuel-type-diesel</th>\n      <th>fuel-type-gas</th>\n      <th>aspiration-std</th>\n      <th>aspiration-turbo</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>3</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>0.811148</td>\n      <td>0.890278</td>\n      <td>...</td>\n      <td>5000.0</td>\n      <td>21</td>\n      <td>8.703704</td>\n      <td>13495.0</td>\n      <td>11.190476</td>\n      <td>Low</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>two</td>\n      <td>convertible</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>88.6</td>\n      <td>0.811148</td>\n      <td>0.890278</td>\n      <td>...</td>\n      <td>5000.0</td>\n      <td>21</td>\n      <td>8.703704</td>\n      <td>16500.0</td>\n      <td>11.190476</td>\n      <td>Low</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>122</td>\n      <td>alfa-romero</td>\n      <td>two</td>\n      <td>hatchback</td>\n      <td>rwd</td>\n      <td>front</td>\n      <td>94.5</td>\n      <td>0.822681</td>\n      <td>0.909722</td>\n      <td>...</td>\n      <td>5000.0</td>\n      <td>19</td>\n      <td>9.038462</td>\n      <td>16500.0</td>\n      <td>12.368421</td>\n      <td>Medium</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>fwd</td>\n      <td>front</td>\n      <td>99.8</td>\n      <td>0.848630</td>\n      <td>0.919444</td>\n      <td>...</td>\n      <td>5500.0</td>\n      <td>24</td>\n      <td>7.833333</td>\n      <td>13950.0</td>\n      <td>9.791667</td>\n      <td>Low</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2</td>\n      <td>164</td>\n      <td>audi</td>\n      <td>four</td>\n      <td>sedan</td>\n      <td>4wd</td>\n      <td>front</td>\n      <td>99.4</td>\n      <td>0.848630</td>\n      <td>0.922222</td>\n      <td>...</td>\n      <td>5500.0</td>\n      <td>18</td>\n      <td>10.681818</td>\n      <td>17450.0</td>\n      <td>13.055556</td>\n      <td>Low</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 30 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": "<details><summary>Click here for the solution</summary>\n\n```python\n# merge the new dataframe to the original datafram\ndf = pd.concat([df, dummy_variable_2], axis=1)\n\n# drop original column \"aspiration\" from \"df\"\ndf.drop('aspiration', axis = 1, inplace=True)\n\n\n```\n\n</details>\n",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": "df.to_csv('clean_df.csv')",
      "metadata": {
        "trusted": true
      },
      "execution_count": 57,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": "Save the new csv:\n\n> Note : The  csv file cannot be viewed in the jupyterlite based SN labs environment.However you can Click <a href=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DA0101EN-SkillsNetwork/labs/Module%202/DA0101EN-2-Review-Data-Wrangling.ipynb?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDA0101ENSkillsNetwork20235326-2022-01-01\">HERE</a> to download the lab notebook (.ipynb) to your local machine and view the csv file once the notebook is executed.\n",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "### Thank you for completing this lab!\n\n## Author\n\n<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDA0101ENSkillsNetwork20235326-2021-01-01\" target=\"_blank\">Joseph Santarcangelo</a>\n\n### Other Contributors\n\n<a href=\"https://www.linkedin.com/in/mahdi-noorian-58219234/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDA0101ENSkillsNetwork20235326-2021-01-01\" target=\"_blank\">Mahdi Noorian PhD</a>\n\nBahare Talayian\n\nEric Xiao\n\nSteven Dong\n\nParizad\n\nHima Vasudevan\n\n<a href=\"https://www.linkedin.com/in/fiorellawever/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDA0101ENSkillsNetwork20235326-2021-01-01\" target=\"_blank\">Fiorella Wenver</a>\n\n<a href=\"https:// https://www.linkedin.com/in/yi-leng-yao-84451275/ \" target=\"_blank\" >Yi Yao</a>.\n\n## Change Log\n\n| Date (YYYY-MM-DD) | Version | Changed By | Change Description                  |\n| ----------------- | ------- | ---------- | ----------------------------------- |\n| 2022-04-22        | 2.3     | Lakshmi    | Made changes in markdown file       |\n| 2020-10-30        | 2.2     | Lakshmi    | Changed URL of csv                  |\n| 2020-09-09        | 2.1     | Lakshmi    | Updated Indicator Variables section |\n| 2020-08-27        | 2.0     | Lavanya    | Moved lab to course repo in GitLab  |\n\n<hr>\n\n## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n",
      "metadata": {}
    }
  ]
}