Segmentation_Imputer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Guide to RobMel's Segmentation Code\n",
        "The segmentation allows the number of segments we want to impute by, meaning the number of times our datasets will be fractionally split, imputed and recombined. \n",
        "\n",
        "The method is it takes the least NaN value column, splits according to that one, imputes once, and then continues to split on the unaltered column. If the segment has NaNs, they are filled by *most frequent* if categorical and *median* if numerical (the choices for these two are outlined in the code). \n",
        "\n",
        "If there is an overall column in a segmented column that is all Nans, the parent imputed dataframe will be used to fill in the values. Each imputation step returns four versions: one following using the **mean**, **median**, **most frequent** and **KNN**.\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Difference with Sklearn IterativeImputer\n",
        "I found Sklearn's IterativeImputer has some flexibility in terms of by what attributes you want to divide by, **however** there appear some limitations, namely all NaN columns are dropped, and the imputation type is one by one (these are challenges which may be overcome). Link to documentation is [here](https://scikit-learn.org/stable/modules/generated/sklearn.impute.IterativeImputer.html)\n",
        "\n",
        "I have been focusing on building the custom imputation code mentioned above, and leave this to experimentation after we get a working imputer using the above.\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "## Perceived Benefits\n",
        "\n",
        "The perceived benefits of such a code is that it focuses on splitting and imputing the data on features heuristically selected to maximize data amount used as an input for imputation.\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "# Non-Disclosure Note\n",
        "\n",
        "Due to the non-disclosure required by the original project this was from, some parts of the module have been redacted and are still in the process of being generalized and published."
      ],
      "metadata": {
        "id": "EgusHwefnn4d"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "ZZc2gwIFnLpt"
      },
      "outputs": [],
      "source": [
        "#@title Imports\n",
        "import os\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import statsmodels.api as sm\n",
        "import scipy\n",
        "from scipy import interpolate\n",
        "from itertools import product\n",
        "import  matplotlib.pyplot as plt\n",
        "from sklearn.impute import SimpleImputer, KNNImputer\n",
        "from sklearn.preprocessing import OneHotEncoder,LabelEncoder"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "filepath = \"PATH TO DF\"\n",
        "df = pd.read_excel(filepath)"
      ],
      "metadata": {
        "id": "8ikEx2PRnYVA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Module"
      ],
      "metadata": {
        "id": "JZbrClnmqSEe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Sample Imputer: Feel free to change based on your needs and data distribution\n",
        "imp = KNNImputer(missing_values=np.nan, n_neighbors=1)"
      ],
      "metadata": {
        "id": "f362ileOpDmD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class SegmentImputer:\n",
        "  \"\"\"\n",
        "  Custom Segmentation Imputer Class\n",
        "  \"\"\"\n",
        "  def __init__(self,imputer,segments = [], skip = [], drop = [], threshold = 1,backup=None):\n",
        "    \"\"\"\n",
        "    imputer             - any imputer with the same API as sklearn imputers\n",
        "    segments (optional) - priority list of segments to take place\n",
        "    skip (optional)     - features to not be included when choosing a segmenting feature\n",
        "    drop (optional)     - features to be dropped if spotted in dataset\n",
        "    threshold (default) - maximum percentage of NaN values in a DF before looking outside for \n",
        "    backup (optional)   - precalculated baseline backup dataset if imputer to be used with same dataset\n",
        "    \"\"\"\n",
        "    self.imputer = imputer\n",
        "    self.segments = segments\n",
        "    self.skip = skip\n",
        "    self.drop = drop\n",
        "    self.threshold = threshold\n",
        "    self._available= [] # segments partitioned by (moves from more to less, aka specific to general)\n",
        "    self.__labelencoderlist = []\n",
        "    self._dummydict = {} #used for temporary storage of dummy columns\n",
        "    self._baselinebackup = backup # to save computation time\n",
        "    self.__checksegmentlist()\n",
        "\n",
        "  def preprocess(self,df,method=\"default\",deep=True,fillna=False,cat_cols=[],dummies=[]):\n",
        "    \"\"\"\n",
        "    Preprocess DataFrame to be compatible for imputation\n",
        "    \n",
        "    df      - dataframe to be preprocessed\n",
        "    method  - method to be used for preprocessing\n",
        "      * default: uses LabelEncoder to transform categorical variables to numeric (optimal KNN if n_neighbor=1)\n",
        "      * missforest: non-parametric preprocessing [coming soon]\n",
        "      * none: no preprocessing\n",
        "      * onehot: apply one hot encoding (increasing dimension sizes in process)\n",
        "    \"\"\"\n",
        "\n",
        "    # UPDATE NOTE: DOES NOT HANDLE MISSING VALUES YET FOR CATEGORICAL ENCODING\n",
        "    if method==\"none\":\n",
        "      return df\n",
        "      \n",
        "    if method=='onehot':\n",
        "      encdf = pd.DataFrame(index=df.index)\n",
        "      cols = []\n",
        "      for col in df.columns:\n",
        "        if col in cat_cols:\n",
        "          cols.append(pd.get_dummies(df[col],prefix=col))\n",
        "        else:\n",
        "          cols.append(df[col])\n",
        "      \n",
        "      encdf = pd.concat(cols,axis = 1)\n",
        "      return encdf      \n",
        "\n",
        "    if method=='default':\n",
        "      encdf = df.copy(deep=deep)\n",
        "      encdf.drop(columns=self.drop,inplace=True)\n",
        "      if not len(cat_cols):\n",
        "        return encdf\n",
        "\n",
        "      try:\n",
        "        i=0 # index counter variable\n",
        "        for col in cat_cols:\n",
        "          if col in self.skip or col in self.drop:\n",
        "            continue\n",
        "          \n",
        "          self.__labelencoderlist.append(LabelEncoder())\n",
        "          encdf[col] = self.__labelencoderlist[i].fit_transform(encdf[col])\n",
        "          i+=1 # update index\n",
        "\n",
        "      except (TypeError,IndexError) as e:\n",
        "        print(f\"For col {col}, the following error appeared\\n{e}\")\n",
        "        print(f\"Col type {type(col)}\")\n",
        "      except:\n",
        "        raise Exception()\n",
        "    else:\n",
        "      raise Exception(\"No Method Selected\")\n",
        "    \n",
        "    if len(dummies)>0:\n",
        "      for dummy in dummies:\n",
        "        dummyset = pd.get_dummies(encdf[dummy])\n",
        "        self._dummydict[dummy] = dummyset.columns\n",
        "        for col in dummyset.columns:\n",
        "          encdf[col] = dummyset[col]\n",
        "        encdf = encdf.drop(columns=dummy)\n",
        "    return encdf\n",
        "\n",
        "  def segment(self,df,n,segmentcriteria=\"leastnan\", splitcriteria=\"unique\",preprocess=True,preprocess_method=\"default\",reset_available=True,cat_cols=[],dummies=[]):\n",
        "    \"\"\"\n",
        "    Allows for segmentation with backup to take place\n",
        "\n",
        "    df      - raw dataframe to be segmented\n",
        "    n       - number of segmentations to take place\n",
        "    \"\"\"\n",
        "    if preprocess:\n",
        "      self._baselinebackup = df\n",
        "      df = self.preprocess(df, method=preprocess_method,cat_cols=cat_cols,dummies=dummies)\n",
        "    splits = self.get_splits(df,n,criteria=segmentcriteria,reset_available=reset_available)\n",
        "\n",
        "    counter = 0\n",
        "    imputed_splits = []\n",
        "    for split in splits:\n",
        "      subset = df\n",
        "      for i in range(n):\n",
        "        attr = self.available[i]\n",
        "        subset = subset[subset[attr]==split[i]]\n",
        "      counter+=len(subset)\n",
        "      imputed_subset = self.impute(subset)\n",
        "      if imputed_subset is not None:\n",
        "        imputed_splits.append(imputed_subset)\n",
        "      else:\n",
        "        print(f\"Current Split {split}\")\n",
        "        print(f\"Split Size: {len(subset)} ({len(subset)*100/len(df):.2f}%)\")\n",
        "        print(f\"{counter} of {len(df)} points processed ({counter*100/len(df):.2f}%)\")\n",
        "        continue\n",
        "    \n",
        "    imputed_df = pd.concat(imputed_splits)\n",
        "    if imputed_df.isnull().sum().sum()>0:\n",
        "      print(f\"moving one layer up to {n-1} segments from {n}\")\n",
        "      self.available.pop()\n",
        "      return self.segment(imputed_df,n-1,segmentcriteria=segmentcriteria,splitcriteria=splitcriteria,preprocess=False,reset_available=False,cat_cols=cat_cols,dummies=dummies)\n",
        "    \n",
        "    print(\"Segmentation Complete\")\n",
        "\n",
        "    print(\"Post Processing...\")\n",
        "    post_process_df = self._postprocess(imputed_df,cat_cols=cat_cols,dummies=dummies)\n",
        "    return post_process_df\n",
        "\n",
        "  def impute(self,subset):\n",
        "    \"\"\"\n",
        "    Subset to be imputed\n",
        "    \"\"\"\n",
        "    # check if subset exists\n",
        "    if len(subset)==0:\n",
        "      return None\n",
        "    elif len(subset)==1:\n",
        "      return subset # cannot be imputed alone\n",
        "    \n",
        "    if subset.isnull().sum().sum()==0:\n",
        "      return subset\n",
        "    \n",
        "    # save nan columns:\n",
        "    nan_bools = subset.isnull().all()\n",
        "    nan_cols = list(nan_bools[nan_bools==True].index)\n",
        "    order = subset.columns # in case columns get dropped\n",
        "    temp_order = []\n",
        "    for col in order:\n",
        "      if col in nan_cols:\n",
        "        continue\n",
        "      else:\n",
        "        temp_order.append(col)\n",
        "\n",
        "    # impute\n",
        "    print(\"imputing...\")\n",
        "    imputed_subset = self.imputer.fit_transform(subset)\n",
        "    imputed_subset = pd.DataFrame(imputed_subset,columns=temp_order)\n",
        "    \n",
        "    # put back nan columns\n",
        "    for col in nan_cols:\n",
        "      imputed_subset[col]=np.nan\n",
        "    \n",
        "    imputed_subset = imputed_subset.reindex(columns = order)\n",
        "    \n",
        "    return imputed_subset\n",
        "\n",
        "\n",
        "  def get_splits(self,df,n,criteria='leastnan',reset_available=False):\n",
        "    \"\"\"\n",
        "    Gets subsets for given number of splits\n",
        "\n",
        "    df      - preprocessed dataframe to be segmented\n",
        "    n       - number of segments\n",
        "    criteria (optional) - method to be used when choosing a new segment\n",
        "    reset_available (optional) - whether to recalculate partitions to be used\n",
        "    \"\"\"\n",
        "    if reset_available:\n",
        "      self._get_set_segments(df,n,criteria=criteria)\n",
        "    \n",
        "    vars = [list(df[col].unique()) for col in self.available]\n",
        "    splits = product(*vars) # iterator\n",
        "    \n",
        "    return splits\n",
        "\n",
        "\n",
        "  def _get_set_segments(self,df,n,criteria=\"leastnan\"):\n",
        "    \"\"\"\n",
        "    Gets and Sets Segments to be used\n",
        "\n",
        "    df      - preprocessed dataframe to be segmented\n",
        "    n       - number of segments\n",
        "    criteria (optional) - method to be used when choosing new segment\n",
        "    \"\"\"\n",
        "    self.available = []\n",
        "    for i in range(n):\n",
        "      if i<len(self.segments):\n",
        "        self.available.append(self.segments[i])\n",
        "      else:\n",
        "        self.available.append(self.next_segment(df,used_segments = self.available,criteria=criteria))\n",
        "    return self.available\n",
        "    \n",
        "  def _updatesplitcount():\n",
        "    pass\n",
        "\n",
        "  def _check_threshold(self,df,split):\n",
        "    nans = self._count_nans(df)[split]\n",
        "    pass\n",
        "\n",
        "  def _count_nans(self,df):\n",
        "    return ((df.isnull().sum()).sort_values(ascending=False)/len(df))\n",
        "\n",
        "  ### BEG OUTDATED ###\n",
        "      #split,splitbackup = splitter(df,seg)\n",
        "      #backup.extend(splitbackup)\n",
        "      #dfs.extend(split)\n",
        "    # while 'n'\n",
        "      # choose segment\n",
        "      # split data based on unique values of segment\n",
        "      # impute, if not available use backup data to impute\n",
        "      # return, merge, and sort data\n",
        "  ### END OUTDATED ###\n",
        "\n",
        "  def next_segment(self,df,used_segments=[],criteria=\"leastnan\"):\n",
        "    \"\"\"\n",
        "    df            - data frame to find next segmenting attribute\n",
        "    used_segments - list of used segments to be skipped over\n",
        "    criteria      - criteria which to choose next segment attribute\n",
        "      * leastnan: least number of NaNs (default)\n",
        "      * mostnan: most number of Nans\n",
        "      * alphabetical: alphabetically by attribute name\n",
        "      * leastunique: least number of unique values\n",
        "      * mostunique: most number of unique values\n",
        "    \"\"\"\n",
        "    # preliminary check for priority segments\n",
        "    for seg in self.segments:\n",
        "      if seg not in used_segments:\n",
        "        return seg\n",
        "\n",
        "    # else resume normal flow\n",
        "    if criteria==\"leastnan\":\n",
        "      attrs = list((df.isnull().sum()).sort_values(ascending=False).index)\n",
        "    elif criteria==\"mostnan\":\n",
        "      attrs = list((df.isnull().sum()).sort_values(ascending=True).index)\n",
        "    elif criteria==\"alphabetical\":\n",
        "      attrs = list(df.columns.sort_values(ascending=False))\n",
        "    elif criteria==\"leastunique\":\n",
        "      attrs = list(df.nunique().sort_values(ascending=False).index)\n",
        "    elif criteria==\"mostunique\":\n",
        "      attrs = list(df.nunique().sort_values(ascending=True).index)\n",
        "    else:\n",
        "      raise Exception(f\"Segmentation Criteria {criteria} not found.\\nChoose from 'leastnan','mostnan','alphabetical','leastunique','mostunique'.\")\n",
        "    \n",
        "    attr = attrs[-1]\n",
        "    while attr in used_segments or attr in self.skip:\n",
        "      attrs.pop()\n",
        "      attr = attrs[-1]\n",
        "    \n",
        "    return attrs[-1]\n",
        "\n",
        "  def split(self,df,segment,criteria=\"unique\",interval=None):\n",
        "    \"\"\"\n",
        "    df        - dataframe to be split\n",
        "    segment   - segment as to split on\n",
        "    criteria  - criteria of which to apply splits\n",
        "      * unique: by each unique value (default)\n",
        "      * range: by intervals of values (only works on numeric, else defaults to unique)\n",
        "    \"\"\"\n",
        "    if criteria==\"range\":\n",
        "      if not interval:\n",
        "        print(\"Interval not supplied, defaulting to unique\")\n",
        "      else:\n",
        "        raise Exception(\"Range Not Built Yet\")\n",
        "    elif criteria not in (\"range\",\"unique\"):\n",
        "      raise Exception(f\"Supplied criteria {criteria} not valid.\\nCriteria must be: 'unique','range'.\\nIf 'range' supply 'interval' value.\")\n",
        "    \n",
        "    # default\n",
        "    return df[segment].unique()\n",
        "\n",
        "  def _postprocess(self,df,method=\"none\",cat_cols = [],dummies = []):\n",
        "    \"\"\"\n",
        "    Post-processing of data\n",
        "    \n",
        "    df      - dataframe to be processed\n",
        "    method  - method to be used\n",
        "      * default: for REDACTED Project\n",
        "      * none: no processing\n",
        "    \"\"\"\n",
        "\n",
        "    if method==\"none\":\n",
        "      return df\n",
        "\n",
        "    ### BEG OUTDATED ###\n",
        "    #df = df.apply(lambda series: pd.Series(\n",
        "    #self.__labelencoder.inverse_transform(series[series.notnull()]),\n",
        "    #index=series[series.notnull()].index))\n",
        "    ### END OUTDATED ###\n",
        "\n",
        "    if len(dummies)>0:\n",
        "      # get max\n",
        "      for dummy in dummies:\n",
        "        cols = self._dummydict[dummy]\n",
        "        df[dummy] = df[cols].idxmax(axis=1)\n",
        "        df = df.drop(columns=cols)  \n",
        "\n",
        "    if len(cat_cols)>0:\n",
        "      for i,col in enumerate(cat_cols):\n",
        "        df[col] = pd.Series(self.__labelencoderlist[i].inverse_transform(df[col].astype(int)))\n",
        "\n",
        "    # sort columns\n",
        "    cols = self._baselinebackup.columns\n",
        "    df = df.reindex(cols,axis=1)\n",
        "\n",
        "    return df\n",
        "  \n",
        "  def __checksegmentlist(self):\n",
        "    for seg in self.segments:\n",
        "      if seg in self.skip:\n",
        "        self.segments.remove(seg)"
      ],
      "metadata": {
        "id": "ciqSgxyOpVJF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model Run and Visualization"
      ],
      "metadata": {
        "id": "NA6PKANTqWR1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# May be skewed, but using KMeans on the nonans dataset\n",
        "from sklearn.cluster import KMeans\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "def fitting(df):\n",
        "    Sum_of_squared_distances = []\n",
        "    K = range(1,20)\n",
        "    for k in K:\n",
        "        km = KMeans(n_clusters=k)\n",
        "        s = StandardScaler()\n",
        "        s.fit(df)\n",
        "        s_df = s.transform(df)\n",
        "        km = km.fit(s_df)\n",
        "        Sum_of_squared_distances.append(km.inertia_)\n",
        "    plt.plot(K, Sum_of_squared_distances, 'bx-')\n",
        "    plt.xlabel('k')\n",
        "    plt.ylabel('Sum_of_squared_distances')\n",
        "    plt.title('Elbow Method For Optimal k')\n",
        "    plt.show()\n",
        "    return \n",
        "\n",
        "seg = SegmentImputer(KNNImputer())\n",
        "fitting(seg.preprocess(df))"
      ],
      "metadata": {
        "id": "YkyaqMs0qLrF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# End Notes:\n",
        "\n",
        "With further updates in Numpy/Pandas, there remains an opportunity to further parallelize many of the operations in the original module."
      ],
      "metadata": {
        "id": "K07twf3wqhra"
      }
    }
  ]
}