differential_preserver_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Using DPLib\n",
    "\n",
    "The following example will give an insight on how to use this module. Make sure to read the **README.md** \n",
    "file beforehand.\n",
    "\n",
    "1. Create a Spark Session. Make sure to enable PyArrow configuration."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "\n",
    "spark = SparkSession.builder \\\n",
    "    .master('local') \\\n",
    "    .appName('differential_privacy') \\\n",
    "    .config('spark.some.config.option', 'some-value') \\\n",
    "    .getOrCreate()\n",
    "\n",
    "spark.conf.set('spark.sql.execution.arrow.enabled', 'true')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "2. Create a Spark DataFrame (sdf). \n",
    "\n",
    "Generate an sdf with random values. It is better to manually specify the **schema** of sdf so as to avoid \n",
    "any *TypeErrors*.\n",
    "\n",
    "Here I will generate an sdf with 3 columns: *'Numeric'*, *'Rounded_Numeric'*, *'Boolean'* and 10,000 rows \n",
    "to show 3 ways of using DPLib."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "+---------+------------------+-------+\n|   Number|    Rounded_Number|Boolean|\n+---------+------------------+-------+\n|54912.523| 69594.76397557047|    yes|\n|35090.402|   40837.245314592|    yes|\n|23256.068|13188.899709749729|     no|\n| 20674.72|59698.587525683884|     no|\n| 63732.16| 50144.16482347039|     no|\n+---------+------------------+-------+\nonly showing top 5 rows\n\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "from random import random, randint, choice\n",
    "from pyspark.sql.types import *\n",
    "\n",
    "# generate a row with random numbers of range(0, 100000) and random strings of either 'yes' or 'no'\n",
    "def generate_rand_tuple():\n",
    "    number_1 = randint(0, 100000) + random()\n",
    "    number_2 = randint(0, 100000) + random()\n",
    "    string = choice(['yes', 'no'])\n",
    "    return number_1, number_2, string\n",
    "\n",
    "data = [generate_rand_tuple() for _ in range(100000)]\n",
    "\n",
    "schema = StructType([\n",
    "    StructField('Number', FloatType()),\n",
    "    StructField('Rounded_Number', DoubleType()),\n",
    "    StructField('Boolean', StringType())\n",
    "])\n",
    "\n",
    "sdf = spark.createDataFrame(data=data, schema=schema)\n",
    "sdf.show(n=5)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "3. Setup and configure **DPLib**\n",
    "\n",
    "DPLib can work with numbers and binary strings. To anonymize a number based column, you have to setup the column \n",
    "category as *'numeric'*. To anonymize a string based column, you have to setup the column category as *'boolean'*.\n",
    "\n",
    "3.1 Initializing the module\n",
    "\n",
    "The module takes in 3 optional parameters when initializing: *Spark DataFrame*, *epsilon* and *delta*. Module can also \n",
    "be initialized without any parameters and they can be added later."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "outputs": [],
   "source": [
    "from spark_privacy_preserver.differential_privacy import DPLib\n",
    "\n",
    "epsilon = 0.00001\n",
    "delta = 0.5\n",
    "sensitivity = 10\n",
    "\n",
    "# method 1\n",
    "dp = DPLib(global_epsilon=epsilon, global_delta=delta, sdf=sdf)\n",
    "dp.set_global_sensitivity(sensitivity=sensitivity)\n",
    "\n",
    "# method 2\n",
    "dp = DPLib()\n",
    "dp.set_sdf(sdf=sdf)\n",
    "dp.set_global_epsilon_delta(epsilon=epsilon, delta=delta)\n",
    "dp.set_global_sensitivity(sensitivity=sensitivity)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Note:** The reason behind the word *global* in above functions\n",
    "\n",
    "Suppose the user want to anonymize 3 columns of a DataFrame with same epsilon, delta and sensitivity and another \n",
    "column with different parameters. Now all the user has to do is to set up global parameters for 3 columns and \n",
    "local parameters for 4th column. \n",
    "\n",
    "This will simplify when multiple columns of a DataFrame have to be processed with same parameters."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "3.2 Configuring columns\n",
    "\n",
    "User can configure columns with column specific parameters. Column specific parameters will be given higher priority \n",
    "over global parameters if explicitly specified.\n",
    "\n",
    "parameters that can be applied to method *set_column()*:\n",
    "1. column_name: name of column as string -> compulsory\n",
    "2. category: category of column. can be either *'numeric'* or *'boolean'* -> compulsory\n",
    "3. epsilon: column specific value -> optional\n",
    "4. delta: column specific value -> optional\n",
    "5. sensitivity: column specific value -> optional\n",
    "6. lower_bound: set minimum number a column can have. can only be applied to category *'numeric'* -> optional\n",
    "7. upper_bound: set maximum number a column can have. can only be applied to category *'numeric'* -> optional\n",
    "8. label1: string label for a column. can only be applied to category *'binary'* -> optional\n",
    "9. label2: string label for a column. can only be applied to category *'binary'* -> optional\n",
    "10. round: value by which to round the result. can only be applied to category *'numeric'* -> optional\n",
    "\n",
    "You can see the results with "
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "outputs": [],
   "source": [
    "dp.set_column(column_name='Number', \n",
    "              category='numeric')\n",
    "# epsilon, delta, sensitivity will be taken from global parameters and applied.\n",
    "\n",
    "dp.set_column(column_name='Rounded_Number', \n",
    "              category='numeric',\n",
    "              epsilon=epsilon * 10,\n",
    "              sensitivity=sensitivity * 10,\n",
    "              lower_bound=round(sdf.agg({'Rounded_Number': 'min'}).collect()[0][0]) + 10000,\n",
    "              upper_bound=round(sdf.agg({'Rounded_Number': 'max'}).collect()[0][0]) - 10000,\n",
    "              round=2)\n",
    "# epsilon, sensitivity will be taken from user input instead of global parameters\n",
    "# delta will be taken from global parameters.\n",
    "\n",
    "dp.set_column(column_name='Boolean',\n",
    "              category='boolean',\n",
    "              label1='yes',\n",
    "              label2='no',\n",
    "              delta=delta if 0 < delta <= 1 else 0.5)\n",
    "# sensitivity will be taken from user input instead of global parameters\n",
    "# epsilon will be taken from global parameters.\n",
    "# 'boolean' category does not require delta"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "3.2.1 To view existing configuration for the class, use following method"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "Global parameters\n-----------------\n\nEpsilon      1e-05\nDelta        0.5\nSensitivity  10.0\n\n\nColumn specific parameters\n--------------------------\n\n| Column name    | Column category   | Epsilon   | Delta   | Sensitivity   | Lower bound   | Upper bound   | Round   | Label 1   | Label 2   |\n|----------------|-------------------|-----------|---------|---------------|---------------|---------------|---------|-----------|-----------|\n| Number         | numeric           | 1e-05     | 0.5     | 10.0          | -inf          | inf           | --      | --        | --        |\n| Rounded_Number | numeric           | 0.0001    | 0.5     | 100           | 10002         | 89998         | 2       | --        | --        |\n| Boolean        | boolean           | 1e-05     | 0.5     | --            | --            | --            | --      | yes       | no        |\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "dp.get_config()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "3.2.2 To drop a column or to drop all columns use the *drop_column()* method. \n",
    "To drop all columns use '*' as input parameter"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "outputs": [],
   "source": [
    "# Methods are commented out to show the expected output by execute() method.\n",
    "\n",
    "# dp.drop_column('Rounded_Number', 'Number')\n",
    "# \n",
    "# dp.drop_column('*')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "3.3 Executing"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "+---------+------------------+-------+\n|   Number|    Rounded_Number|Boolean|\n+---------+------------------+-------+\n|54912.523| 69594.76397557047|    yes|\n|35090.402|   40837.245314592|    yes|\n|23256.068|13188.899709749729|     no|\n| 20674.72|59698.587525683884|     no|\n| 63732.16| 50144.16482347039|     no|\n| 69335.56|15584.497621045835|     no|\n|3425.1135| 40487.41520427121|     no|\n| 7046.105|24430.790320596756|     no|\n| 14452.13|   91870.280643966|    yes|\n|99262.375| 79575.21126624933|     no|\n|15766.421|18318.521522660874|    yes|\n|  90014.6|28330.082632902333|    yes|\n| 87265.21| 44303.34359314786|    yes|\n|36002.305| 21976.41587165995|     no|\n|26693.273|30043.504766771344|     no|\n|36500.125| 96298.48997401856|     no|\n|55626.527| 72690.89554990188|    yes|\n|29266.404| 26632.67830042936|    yes|\n| 40381.26| 2583.275670165855|    yes|\n|66289.695| 85074.72392196131|    yes|\n+---------+------------------+-------+\nonly showing top 20 rows\n\n",
      "+------------------+--------------+-------+\n|            Number|Rounded_Number|Boolean|\n+------------------+--------------+-------+\n| 54902.54527921043|      69609.44|    yes|\n| 35092.71174878052|      40697.38|    yes|\n|23253.861758415853|      13356.09|     no|\n| 20676.01723684414|      59609.33|     no|\n|   63735.508949082|      50598.17|     no|\n| 69316.27292882366|      15490.02|    yes|\n|3420.9184953495987|      40916.92|     no|\n| 7050.111334070098|      24380.35|    yes|\n|14418.563432544235|       89998.0|    yes|\n| 99279.07030038328|      79733.41|     no|\n| 15759.42757256932|       18317.7|    yes|\n| 90015.98635798543|      28107.14|    yes|\n|  87250.1961851628|      44005.98|     no|\n| 36004.51814183073|      22028.49|    yes|\n|  26683.6904876414|      29590.43|     no|\n| 36497.23340340989|       89998.0|     no|\n| 55614.13091646135|      73166.72|    yes|\n| 29243.34247065875|      26403.26|    yes|\n|  40398.8176834737|       10002.0|    yes|\n| 66304.38452463523|      85103.56|    yes|\n+------------------+--------------+-------+\nonly showing top 20 rows\n\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "# gets first 20 rows of DataFrame before anonymizing and after anonymizing\n",
    "sdf.show()\n",
    "\n",
    "dp.execute()\n",
    "\n",
    "dp.sdf.show()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As you can see, there is a clear difference between original DataFrame and anonymized DataFrame.\n",
    "\n",
    "1. Column *'Number'* is anonymized but the values are not bound to a certain range. The algorithm produces the result \n",
    "with maximum precision as it can achieve.\n",
    "\n",
    "2. Column *'Rounded_Number'* is both anonymized and bounded to the values set by user. As you can see, the values \n",
    "never rise above upper bound and never become lower than lower bound. Also they are rounded to 2nd decimal place as set.\n",
    "\n",
    "3. Column *'Boolean'* undergoes through a mechanism that randomly decides to flip to the other binary value or not, \n",
    "in order to satisfy differential privacy.\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n",
     "is_executing": false
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "pycharm-b2afa515",
   "language": "python",
   "display_name": "PyCharm (spark-privacy-preserver)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}