Lecture2A-part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 2A (Part 2): Bayesian Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Bayesian network (BN) is composed of random variables (nodes) and their conditional dependencies (arcs) which, together, form a directed acyclic graph (DAG). They have become a widely used method in the modelling of uncertain knowledge. A conditional probability table (CPT) is associated with each node. It contains the conditional probability distribution of the node given its parents in the DAG:\n",
    "\n",
    "<img src='images/Wetgrass.png' style='width: 450px;' >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically, each node represents a random variable, which is decribed by a probability distribution over its parents' nodes. The biggest advantage of the Bayesian network is its compact and modular strucute. Humans do not have access to all the probability distributions and all variables of the world. For this reason, in order to make probabilisitc inferences, humans need to combine different sources of evidence in order to come up with an answer. This is precisely what Bayesin Networks do, using elaborate probabilistic formulas based on the Naive Bayes that we jsut saw. It is not relevant to go through the mathematics. Bayesian Networks are graphical structures that enable any non-expert to use them in daily decision making tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Bayesian Network in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the Bayesian Network, which describes the following decision scenario. \n",
    "\n",
    "This network trys to expresss the probability of a person having wither Tuberculosis, Lung Cancer or Bronchitis, given some symptoms, Shortness in Breath (Dispnea), exames (like a positive xray result) and some historical information: visits to Asia and Smoking.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='images/asia.png' style='width: 450px;' >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The random variables of this network are:\n",
    "* Visit to Asia\n",
    "* Tuberculosis\n",
    "* Either Tuberculosi or Lung Cancer\n",
    "* Positive X-Ray\n",
    "* Dispnea\n",
    "* Bronchitis\n",
    "* Smoker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**QUESTION. Which of the above random variables are root notes of the Network? (A root node is a node that does not descend from any other node)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** \n",
    "- Visit to Asia\n",
    "- Smoker\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Network Structure in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by importing into Python the necessary libraries to work in this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:47:25.480058Z",
     "start_time": "2020-03-31T00:47:25.470790Z"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from pylab import *\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#Python Library that deals with Bayesan Networks (BNs)\n",
    "import pyAgrum as bn_graphs\n",
    "import pyAgrum as gum\n",
    "import pyAgrum.lib.notebook as gnb\n",
    "\n",
    "from pyAgrum.lib.bn2roc import showROC\n",
    "\n",
    "import seaborn as sns\n",
    "sns.set()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define our Bayesian Network. As wyou can see, it is an empty structure (for now...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:47:26.139366Z",
     "start_time": "2020-03-31T00:47:26.135729Z"
    }
   },
   "outputs": [],
   "source": [
    "bn = bn_graphs.BayesNet('CancerBN') #Creates an empty network called CancerBN\n",
    "print(bn)                           #Prints the created BN\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code builds a general network structure, but with no nodes or edges or conditional probability tables.\n",
    "Our next step will be precisely to specify these variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Random Variables (the nodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a random variable, we need to use pyAgrum's function *LabelizedVariable*, which is a variable whose domain is a finite set of labels. You can do it in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:47:27.695718Z",
     "start_time": "2020-03-31T00:47:27.692176Z"
    }
   },
   "outputs": [],
   "source": [
    "#The function LabelizedVariable( id_name, label, cardinality), receives the follwoing arguments\n",
    "#name: is a string that uniquely identifies the node\n",
    "#cardinality: is an integer which specifies the amount of different values that the \n",
    "#             the random varible will have. We will set this for 0 now\n",
    "\n",
    "id_name = 'LungCancer'\n",
    "label = 'LungCancer'\n",
    "LungCancer = bn_graphs.LabelizedVariable(id_name, label, 0)\n",
    "print(LungCancer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now specify the labels \"true\" and \"false\" of our random variable. For this, we use the method *addLabel()*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:47:28.445044Z",
     "start_time": "2020-03-31T00:47:28.441401Z"
    }
   },
   "outputs": [],
   "source": [
    "#In our example, we have a random variable 'LungCancer' which can have the values \n",
    "LungCancer.addLabel('present')   #'present' if LungCancer occured or \n",
    "\n",
    "LungCancer.addLabel('absent')    #'absent' if LungCancer did not occur.\n",
    "print(LungCancer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now add the created random variable to our network by using the method .add()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:47:29.213430Z",
     "start_time": "2020-03-31T00:47:29.209474Z"
    }
   },
   "outputs": [],
   "source": [
    "bn.add( LungCancer )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:47:29.627089Z",
     "start_time": "2020-03-31T00:47:29.624218Z"
    }
   },
   "outputs": [],
   "source": [
    "#Create a list with the names of the nodes\n",
    "nodes_lst = ['VisitAsia','Smoker', 'Tuberculosis', 'Bronchitis', 'Dispnea', 'PositiveXray', 'TubercOrLungCan']\n",
    "print(nodes_lst)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:47:30.097111Z",
     "start_time": "2020-03-31T00:47:30.092194Z"
    }
   },
   "outputs": [],
   "source": [
    "#Creates a new node for each of the random variables in nodes_lst\n",
    "\n",
    "# node is a variable  that will go through each entry of the list nodes_lst\n",
    "# the iterations are performed by the function *for* in the following way:\n",
    "# node = 'Visit Asia' ............. iteration #1\n",
    "# node = 'Smoker' ................. iteration #2\n",
    "# node = 'Tuberculosis' ........... iteration #3\n",
    "# node = 'Bronchitis' ............. iteration #4\n",
    "# node = 'Dispnea' ................ iteration #5\n",
    "# node = 'Positive Xray' .......... iteration #6\n",
    "# node = 'TubercOrLungCan' ........ iteration #7\n",
    "for node in nodes_lst:\n",
    "    print(node)\n",
    "    var = bn_graphs.LabelizedVariable(node, node, 0)  #creates random variable\n",
    "    var.addLabel('present')                           #adds the label 'true'\n",
    "    var.addLabel('absent')                            #adds the label 'false'\n",
    "    bn.add(var)                                       #adds the created var to the network\n",
    "\n",
    "print(bn)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Edges (the arcs between nodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined our nodes, we need to define the arcs between them. For this, we use the method addArc( sourceNode, targetNode ):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:47:31.212010Z",
     "start_time": "2020-03-31T00:47:31.209123Z"
    }
   },
   "outputs": [],
   "source": [
    "# Arc between \n",
    "bn.addArc('VisitAsia', 'Tuberculosis')\n",
    "print(bn)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:47:31.633885Z",
     "start_time": "2020-03-31T00:47:31.630175Z"
    }
   },
   "outputs": [],
   "source": [
    "arc_lst = [ ('Tuberculosis', 'TubercOrLungCan'), ('LungCancer', 'TubercOrLungCan'), ('Smoker','LungCancer' ), ('Smoker', 'Bronchitis'), ('Bronchitis', 'Dispnea'), ('TubercOrLungCan', 'Dispnea'), ('TubercOrLungCan', 'PositiveXray' ) ]\n",
    "print( arc_lst )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:47:32.072701Z",
     "start_time": "2020-03-31T00:47:32.069230Z"
    }
   },
   "outputs": [],
   "source": [
    "#Creates a new edge for each of the random variables in arc_lst\n",
    "\n",
    "# arc is a variable that will go through each entry of the list arc_lst\n",
    "# the iterations are performed by the function *for* in the following way:\n",
    "# arc = ('Tuberculosis', 'TubercOrLungCan') ............ iteration #1\n",
    "# arc = ('Lung Cancer', 'TubercOrLungCan') ............. iteration #2\n",
    "# arc = ('Smoker', 'Lung Cancer') ............. iteration #3\n",
    "# arc = ('Smoker', 'Bronchitis') ............. iteration #4\n",
    "# arc = ('Bronchitis', 'Dispnea') ............. iteration #5\n",
    "# arc = ('TubercOrLungCan', 'Dispnea') ............. iteration #6\n",
    "# arc = ('TubercOrLungCan', 'Positive Xray') ............. iteration #3\n",
    "for arc in arc_lst:\n",
    "    bn.addArc( arc[0],  arc[1] )      #adds the created arc to the network\n",
    "\n",
    "print(bn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display your Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:15.883305Z",
     "start_time": "2020-03-31T00:48:15.462699Z"
    }
   },
   "outputs": [],
   "source": [
    "gnb.showBN(bn, size=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the Conditional Probability Tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Onde we have the structure of the network, we need to specify the conditional probability tables (CPTs). In Python, each CPT is referred to as a *Potential*.\n",
    "\n",
    "There are several ways to fill these CPTs. In this workshop, we will show you some of them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Low Level Method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filling the conditional probability table of the root node: VisitAsia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:47.656177Z",
     "start_time": "2020-03-31T00:48:47.651483Z"
    }
   },
   "outputs": [],
   "source": [
    "#Fill the conditional probability table of the variable \n",
    "#VisitAsia according to Figure 1: Pr(VisitAsia=present)  = 0.01\n",
    "#                                 Pr(VisitAsia=absent) = 1 - 0.01 = 0.99\n",
    "bn.cpt('VisitAsia').fillWith( [0.01, 1-0.01] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filling the conditional probability table of the root node: Smoker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:48.258058Z",
     "start_time": "2020-03-31T00:48:48.253400Z"
    }
   },
   "outputs": [],
   "source": [
    "# Fill the conditional probability table of the variable \n",
    "# Smoker according to Figure 1: Pr(Smoker=present)  = 0.5\n",
    "#                               Pr(Smoker=absent) = 1 - 0.5 = 0.5\n",
    "bn.cpt('Smoker').fillWith( [0.5, 1-0.5] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most convinient way to fill conditional probability tables is by using dictionaries in Python. This is done in the following way for variable TubercOrLungCan:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:48.968869Z",
     "start_time": "2020-03-31T00:48:48.961815Z"
    }
   },
   "outputs": [],
   "source": [
    "bn.cpt( 'TubercOrLungCan' )[ {'LungCancer': 'present',  'Tuberculosis': 'present'}  ] = [1, 0]\n",
    "bn.cpt( 'TubercOrLungCan' )[ {'LungCancer': 'present',  'Tuberculosis': 'absent'} ] = [1, 0]\n",
    "bn.cpt( 'TubercOrLungCan' )[ {'LungCancer': 'absent', 'Tuberculosis': 'present'}  ] = [1, 0]\n",
    "bn.cpt( 'TubercOrLungCan' )[ {'LungCancer': 'absent', 'Tuberculosis': 'absent'} ] = [0, 1]\n",
    "\n",
    "bn.cpt('TubercOrLungCan')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:49.365531Z",
     "start_time": "2020-03-31T00:48:49.360391Z"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "bn.cpt( 'Tuberculosis' )[ {'VisitAsia': 'present'}  ] = [ 0.05, 1 - 0.05  ]\n",
    "bn.cpt( 'Tuberculosis' )[ {'VisitAsia': 'absent'} ] = [ 0.01, 1 - 0.01 ]\n",
    "\n",
    "bn.cpt('Tuberculosis')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try it yourself! Can you write down the conditional probability tables for the node LungCancer according to the probabilities in Figure 1?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:50.005937Z",
     "start_time": "2020-03-31T00:48:50.001816Z"
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE:\n",
    "\n",
    "\n",
    "\n",
    "# ---------------------\n",
    "bn.cpt('LungCancer')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:50.366158Z",
     "start_time": "2020-03-31T00:48:50.360739Z"
    }
   },
   "outputs": [],
   "source": [
    "#Bronchitis\n",
    "bn.cpt( 'Bronchitis' )[ {'Smoker': 'present'}  ] = [ 0.6, 1 - 0.6  ]\n",
    "bn.cpt( 'Bronchitis' )[ {'Smoker': 'absent'} ] = [ 0.3, 1 - 0.3 ]\n",
    "\n",
    "bn.cpt('Bronchitis')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:50.709389Z",
     "start_time": "2020-03-31T00:48:50.701897Z"
    }
   },
   "outputs": [],
   "source": [
    "bn.cpt( 'Dispnea' )[ {'Bronchitis': 'present',  'TubercOrLungCan': 'present'}  ] = [0.9, 1-0.9]\n",
    "bn.cpt( 'Dispnea' )[ {'Bronchitis': 'present',  'TubercOrLungCan': 'absent'} ] = [1, 0]\n",
    "bn.cpt( 'Dispnea' )[ {'Bronchitis': 'absent', 'TubercOrLungCan': 'present'}  ] = [0.7, 1-0.7]\n",
    "bn.cpt( 'Dispnea' )[ {'Bronchitis': 'absent', 'TubercOrLungCan': 'absent'} ] = [0.8, 1-0.8]\n",
    "\n",
    "bn.cpt('Dispnea')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:51.057705Z",
     "start_time": "2020-03-31T00:48:51.052275Z"
    }
   },
   "outputs": [],
   "source": [
    "# PositiveXray\n",
    "bn.cpt( 'PositiveXray' )[ {'TubercOrLungCan': 'present'}  ] = [ 0.6, 1 - 0.6  ]\n",
    "bn.cpt( 'PositiveXray' )[ {'TubercOrLungCan': 'absent'} ] = [ 0.3, 1 - 0.3 ]\n",
    "\n",
    "bn.cpt('PositiveXray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:52.906114Z",
     "start_time": "2020-03-31T00:48:51.432813Z"
    }
   },
   "outputs": [],
   "source": [
    "gnb.showInference( bn )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving your Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well done! Your network is now complete! We can now save it in different formats. In this unit, we will use the format *.net* because it is the one that is widey used in the scientific community"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:57.003556Z",
     "start_time": "2020-03-31T00:48:56.998082Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "bn_graphs.saveBN( bn, os.path.join('data', 'Asia.net'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To open the saved file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:57.624768Z",
     "start_time": "2020-03-31T00:48:57.614505Z"
    }
   },
   "outputs": [],
   "source": [
    "bn_saved = bn_graphs.loadBN(os.path.join('data','Asia.net'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:58.442912Z",
     "start_time": "2020-03-31T00:48:58.023657Z"
    }
   },
   "outputs": [],
   "source": [
    "bn_saved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inferences in Bayesian Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Probabilistic inference is the task of deriving the probability of one or more random variables taking a specific value or a specific set of values. For instance, we can use the Bayesian Network to *infer* the probability of the Lung Cancer being present given that a person Smokes:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Pr( LungCancer = present | Smokes = present ) =~?$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To do this, we need to choose an algorithm to perform probabilistic inferences. There are two was to accomplish this in python:\n",
    "- An exact method: **LazyPropagation**, which is usually applied for small networks\n",
    "- An approximate method: **Gibbs**, which is usually applied for large networks.\n",
    "\n",
    "In this unit, we will apply exact probabilistic methods, so we will use the **LazyPropagation** method. We can use it in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:48:59.782197Z",
     "start_time": "2020-03-31T00:48:59.778675Z"
    }
   },
   "outputs": [],
   "source": [
    "inference = bn_graphs.LazyPropagation(bn_saved)\n",
    "print(inference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference Without Evidence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inferenceswithout evidence are inferences in which you do not know anything about your decision scenario. All you variables are *unknown*. In other words, they are **not observed**. These are inferences of the type: whart is the probability of a person having Dispnea?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Pr( Dispnea = true ) =~?$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do this in Python in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:04.703148Z",
     "start_time": "2020-03-31T00:49:04.697508Z"
    }
   },
   "outputs": [],
   "source": [
    "inference.makeInference()\n",
    "inference.posterior('Dispnea')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "If you want to access these values individually, in Python, you proceed like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:05.637603Z",
     "start_time": "2020-03-31T00:49:05.633838Z"
    }
   },
   "outputs": [],
   "source": [
    "# Pr( Dispnea = present)\n",
    "pr_Dispnea = inference.posterior('Dispnea')[0]\n",
    "print('Pr( Dispnea = prsent ) = ' + str(pr_Dispnea))\n",
    "\n",
    "# You can round this number to 4 decimal places\n",
    "print('Pr( Dispnea = present ) = ' + str(round(pr_Dispnea,4)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:06.137156Z",
     "start_time": "2020-03-31T00:49:06.005452Z"
    }
   },
   "outputs": [],
   "source": [
    "gnb.showProba(inference.posterior('Dispnea'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TRY IT YOURSELF** \n",
    "Can you answer the following queries?\n",
    "$$Pr( Bronchitis = present ) =~  $$\n",
    "$$Pr( Tuberculosis = absent ) =~  $$\n",
    "$$Pr( VisitAsia = present ) =~$$\n",
    "$$Pr( PositiveXray = absent ) =~ $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T01:01:12.034564Z",
     "start_time": "2020-03-31T01:01:12.032645Z"
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:07.809593Z",
     "start_time": "2020-03-31T00:49:07.386936Z"
    }
   },
   "outputs": [],
   "source": [
    "gnb.showProba(inference.posterior('Bronchitis'))\n",
    "gnb.showProba(inference.posterior('Tuberculosis'))\n",
    "gnb.showProba(inference.posterior('VisitAsia'))\n",
    "gnb.showProba(inference.posterior('PositiveXray'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference with Eviddence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bayesian Networks also allow us to make more complex questions (or queries) to the network. For instance, let's imagine that we know that a person resently visited Asia. What is now the probability of that person tuberculosis given this additional piece of information (i.e. this piece of evidence)?\n",
    "\n",
    "$$Pr( Tuberculosis = present~|~VisitAsia = present ) =~?$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:08.454418Z",
     "start_time": "2020-03-31T00:49:08.448230Z"
    }
   },
   "outputs": [],
   "source": [
    "# When we observe that an event occured, then we have a piece of evidence to give to our network.\n",
    "# We can specify this by using the function setEvidence() and by specifying the observed variable and its state:\n",
    "inference.setEvidence({'VisitAsia':'present'})\n",
    "\n",
    "# Then, we just make the inference as presented before\n",
    "inference.makeInference()\n",
    "inference.posterior('VisitAsia')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:08.969024Z",
     "start_time": "2020-03-31T00:49:08.843976Z"
    }
   },
   "outputs": [],
   "source": [
    "gnb.showProba(inference.posterior('Tuberculosis'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question** What happened to the probabilities? Knowing that a person went to Asia, what impact did this information cause in, for instance, the person getting Tuberculosis?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer**\n",
    "Before we observed that a person Visited Asia, the probability of the person having tuberculosiswas:\n",
    "$$Pr( Tuberculosis = present ) = 1\\%$$\n",
    "\n",
    "After observing that a person has been in Asia, the probability of Tuberculosis increased to:\n",
    "$$Pr( Tuberculosis = present | VisitAsia = present ) = 5\\%$$\n",
    "Which is not very significant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Try it yourself** Knowing that a person has been in Asia recently and is showing signs of shortness in breath (Dispnea), what happened to the probability distributions in the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer:\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing All Inferences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python also allows us to have a full visualizatin of the inferences of all variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:25.011579Z",
     "start_time": "2020-03-31T00:49:23.618713Z"
    }
   },
   "outputs": [],
   "source": [
    "# Showing the full network when no variables are observed\n",
    "gnb.showInference( bn_saved )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:26.433696Z",
     "start_time": "2020-03-31T00:49:25.035534Z"
    }
   },
   "outputs": [],
   "source": [
    "# Showing the full network when we observe that John Called the police\n",
    "gnb.showInference( bn_saved, inference, {'Tuberculosis':'present'} )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Bayesian Network Using Existing Data - Back to the Breast Cancer Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The conditional probability tables can be manually inderted into the Bayesian Network if we have this knowledge (which usually is acquired from experts and general statistics). However, most of the times, we have a dataset and we need to fill these conditional probability tables using that dataset. In this section, we will guide you on how to achieve this. Note that whether we manually fill these CPTs or if we learn them using existing data, the topology of the network must always be defined before hand!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part of the notebook, we will show how one could have used a Bayesian Network to model the Breast Cancer dataset presented in part 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:29.956188Z",
     "start_time": "2020-03-31T00:49:29.953050Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas\n",
    "import os\n",
    "import math\n",
    "import pyAgrum as gum\n",
    "import pyAgrum.lib.notebook as gnb\n",
    "from pyAgrum.lib.bn2roc import showROC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:31.192296Z",
     "start_time": "2020-03-31T00:49:31.186274Z"
    }
   },
   "outputs": [],
   "source": [
    "# this is a very helpful function to discretise continous data\n",
    "# Bayesian networks only support discrete values, they are not able to deal with continuous variables\n",
    "# given that our breast cancer dataset has continous variables, we need to discretise them\n",
    "def discretize_dataframe( df, num_bins, class_var ):\n",
    "    r=np.array(range(num_bins+1))/(1.0*num_bins)\n",
    "  \n",
    "    # quantiles are building using pandas.qcut\n",
    "    # The \"class\" column is just copied.\n",
    "    l=[]\n",
    "    \n",
    "    for col in df.columns.values:\n",
    "        \n",
    "        if col!=class_var:\n",
    "            l.append( pd.DataFrame( pd.qcut( df[col],r, duplicates='drop',precision=2),columns=[col]))\n",
    "        else:\n",
    "            l.append( pd.DataFrame( np.round(df[col],2),columns=[col]))\n",
    "    \n",
    "    treated=pd.concat(l, join='outer', axis=1)\n",
    "    return treated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to load our dataframe with the breast cancer data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:32.052007Z",
     "start_time": "2020-03-31T00:49:32.016795Z"
    }
   },
   "outputs": [],
   "source": [
    "filepath = \"data/breast_data_bn.csv\"\n",
    "data = pd.read_csv(filepath)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataframe has continuous variables which need to discretized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:32.828766Z",
     "start_time": "2020-03-31T00:49:32.768851Z"
    }
   },
   "outputs": [],
   "source": [
    "num_bins = 5\n",
    "\n",
    "data_discr = discretize_dataframe( data, 5, \"diagnosis\" )\n",
    "data_discr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:33.125896Z",
     "start_time": "2020-03-31T00:49:33.105061Z"
    }
   },
   "outputs": [],
   "source": [
    "# save discretized dataframe to file:\n",
    "file_discr = \"data/breast_data_discr.csv\"\n",
    "data_discr.to_csv(file_discr, index=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:33.810869Z",
     "start_time": "2020-03-31T00:49:33.795591Z"
    }
   },
   "outputs": [],
   "source": [
    "data_discr = pd.read_csv(file_discr)\n",
    "data_discr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learning a Bayesian network from a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:38.883126Z",
     "start_time": "2020-03-31T00:49:38.356730Z"
    }
   },
   "outputs": [],
   "source": [
    "# call a BN learner\n",
    "learner = gum.BNLearner( file_discr )\n",
    "# specify the learning algorithm. For the purposes of this unit, one can use the Hill Climbing algorithm\n",
    "learner.useGreedyHillClimbing()\n",
    "# learn the data\n",
    "bn = learner.learnBN()\n",
    "bn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's ask some questions to the network by calling the inference engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:39.134032Z",
     "start_time": "2020-03-31T00:49:39.127251Z"
    }
   },
   "outputs": [],
   "source": [
    "features = data_discr.drop([\"diagnosis\"], axis=1).columns.tolist()\n",
    "features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:40.589514Z",
     "start_time": "2020-03-31T00:49:39.513732Z"
    }
   },
   "outputs": [],
   "source": [
    "inference = bn_graphs.LazyPropagation(bn)\n",
    "\n",
    "gnb.showInference(bn,inference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can already see that the dataset is highly unlabanced in terms of the diagnosis code: there 62% of negative cases vs 37% of positive cases. Let's continue to ask questions to the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:41.805253Z",
     "start_time": "2020-03-31T00:49:40.630914Z"
    }
   },
   "outputs": [],
   "source": [
    "# what are the general characteristics of a person diagnosed with breast cancer?\n",
    "gnb.showInference( bn, inference, {'diagnosis':1} )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T00:49:42.908460Z",
     "start_time": "2020-03-31T00:49:41.846605Z"
    }
   },
   "outputs": [],
   "source": [
    "# what happens if you observe a positive diagnosis and a radius mean between \"(12.73, 14.06]\"?\n",
    "# what happened to the texture_mean? Can you explain why?\n",
    "value = '\"(12.73, 14.06]\"'\n",
    "gnb.showInference( bn, inference, {'diagnosis':1, 'radius_mean' : value} )\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have demonstradted with different classification techniques using Bayesian Networks. In the first approach, we mangaged to model a classifier without using any training set and relying solely on prior knowledge. In the second approach we used only machine learning techniques. Finally, in the third example we assumed the naive bayes hypothesis and obtained a model combined"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try it yourlself!\n",
    "\n",
    "Try to model the following network and come up with some analysis.\n",
    "\n",
    "Scenario: You have a burglar alarm that is sometimes set off by minor earthquakes. You have two neighbours, John and Mary, who promised to call you if they hear the alarm.\n",
    "\n",
    "Example of an inference task: suppose Mary calls you, but John does not, what is the probability that a burglary occured in your house?\n",
    "\n",
    "<img src = \"images/burglar_bn.png\" width=\"500px\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create empty network\n",
    "\n",
    "\n",
    "# Create nodes\n",
    "\n",
    "\n",
    "# Define edges\n",
    "\n",
    "\n",
    "# conditional probability tables\n",
    "\n",
    "\n",
    "# Show probaility inference in full uncertainty\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question: suppose Mary calls you, but John does not, what is the probability that a burglary occured in your house?\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happened to the probability distribution of the network?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question: suppose John calls you, but Mary does not, what is the probability that a burglary occured in your house?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable John does not play a role as important as Mary in determining whether an Alarm was triggered or not. What about when both are observed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question: suppose John and Mary call you, what is the probability that a burglary occured in your house?\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why the variables Burglar and Earthquake did not show a big change comapared to the Alarm variable?\n",
    "- This is because Burglar and Earthquake are both rare events in this decision scenario. Small changes in these probability distributions should be weighted differently when planning a decision strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}