notebooks/smoothed_vision_transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "58063edd",
   "metadata": {},
   "source": [
    "# Certification of Vision Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0438abb9",
   "metadata": {},
   "source": [
    "In this notebook we will go over how to use the PyTorchSmoothedViT tool and be able to certify vision transformers against patch attacks!\n",
    "\n",
    "### Overview\n",
    "\n",
    "This method was introduced in Certified Patch Robustness via Smoothed Vision Transformers (https://arxiv.org/abs/2110.07719). The core technique is one of *image ablations*, where the image is blanked out except for certain regions. By ablating the input in different ways every time we can obtain many predicitons for a single input. Now, as we are ablating large parts of the image the attacker's patch attack is also getting removed in many predictions. Based on factors like the size of the adversarial patch and the size of the retained part of the image the attacker will only be able to influence a limited number of predictions. In fact, if the attacker has a $m x m$ patch attack and the retained part of the image is a column of width $s$ then the maximum number of predictions $\\Delta$ that could be affected are: \n",
    "\n",
    "<p style=\"text-align: center;\"> $\\Delta = m + s - 1$ </p>\n",
    "\n",
    "Based on this relationship we can derive a simple but effective criterion that if we are making many predictions for an image and the highest predicted class $c_t$ has been predicted $k_t$ times and the second most predicted class $c_{t-1}$ has been predicted $k_{t-1}$ times then we have a certified prediction for $c_t$ if: \n",
    "\n",
    "\n",
    "<p style=\"text-align: center;\"> $k_t -  k_{t-1} > 2\\Delta$  </p>\n",
    "\n",
    "Intuitivly we are saying that even if $k$ predictions were adversarially influenced and those predictions were to change, then the model will *still* have predicted class $c_t$.\n",
    "\n",
    "### What's special about Vision Transformers?\n",
    "\n",
    "The formulation above is very generic and it can be applied to any nerual network model, in fact the original paper which proposed it (https://arxiv.org/abs/2110.07719) considered the case with convolutional nerual networks. \n",
    "\n",
    "However, Vision Transformers (ViTs) are well siuted to this task of predicting with vision ablations for two key reasons: \n",
    "\n",
    "+ ViTs first tokenize the input into a series of image regions which get embedded and then processed through the neural network. Thus, by considering the input as a set of tokens we can drop tokens which correspond to fully masked (i.e ablated)regions significantly saving on the compute costs. \n",
    "\n",
    "+ Secondly, the ViT's self attention layer enables sharing of information globally at every layer. In contrast convolutional neural networks build up the receptive field over a series of layers. Hence, ViTs can be more effective at classifying an image based on its small unablated regions.\n",
    "\n",
    "Let's see how to use these tools!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "aeb27667",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "sys.path.append(\"..\")\n",
    "from torchvision import datasets\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# The core tool is PyTorchSmoothedViT which can be imported as follows:\n",
    "from art.estimators.certification.derandomized_smoothing import PyTorchDeRandomizedSmoothing\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "80541a3a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "# Function to fetch the cifar-10 data\n",
    "def get_cifar_data():\n",
    "    \"\"\"\n",
    "    Get CIFAR-10 data.\n",
    "    :return: cifar train/test data.\n",
    "    \"\"\"\n",
    "    train_set = datasets.CIFAR10('./data', train=True, download=True)\n",
    "    test_set = datasets.CIFAR10('./data', train=False, download=True)\n",
    "\n",
    "    x_train = train_set.data.astype(np.float32)\n",
    "    y_train = np.asarray(train_set.targets)\n",
    "\n",
    "    x_test = test_set.data.astype(np.float32)\n",
    "    y_test = np.asarray(test_set.targets)\n",
    "\n",
    "    x_train = np.moveaxis(x_train, [3], [1])\n",
    "    x_test = np.moveaxis(x_test, [3], [1])\n",
    "\n",
    "    x_train = x_train / 255.0\n",
    "    x_test = x_test / 255.0\n",
    "\n",
    "    return (x_train, y_train), (x_test, y_test)\n",
    "\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = get_cifar_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2ac0c5b3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['vit_base_patch8_224',\n",
       " 'vit_base_patch16_18x2_224',\n",
       " 'vit_base_patch16_224',\n",
       " 'vit_base_patch16_224_miil',\n",
       " 'vit_base_patch16_384',\n",
       " 'vit_base_patch16_clip_224',\n",
       " 'vit_base_patch16_clip_384',\n",
       " 'vit_base_patch16_gap_224',\n",
       " 'vit_base_patch16_plus_240',\n",
       " 'vit_base_patch16_rpn_224',\n",
       " 'vit_base_patch16_xp_224',\n",
       " 'vit_base_patch32_224',\n",
       " 'vit_base_patch32_384',\n",
       " 'vit_base_patch32_clip_224',\n",
       " 'vit_base_patch32_clip_384',\n",
       " 'vit_base_patch32_clip_448',\n",
       " 'vit_base_patch32_plus_256',\n",
       " 'vit_giant_patch14_224',\n",
       " 'vit_giant_patch14_clip_224',\n",
       " 'vit_gigantic_patch14_224',\n",
       " 'vit_gigantic_patch14_clip_224',\n",
       " 'vit_huge_patch14_224',\n",
       " 'vit_huge_patch14_clip_224',\n",
       " 'vit_huge_patch14_clip_336',\n",
       " 'vit_huge_patch14_xp_224',\n",
       " 'vit_large_patch14_224',\n",
       " 'vit_large_patch14_clip_224',\n",
       " 'vit_large_patch14_clip_336',\n",
       " 'vit_large_patch14_xp_224',\n",
       " 'vit_large_patch16_224',\n",
       " 'vit_large_patch16_384',\n",
       " 'vit_large_patch32_224',\n",
       " 'vit_large_patch32_384',\n",
       " 'vit_medium_patch16_gap_240',\n",
       " 'vit_medium_patch16_gap_256',\n",
       " 'vit_medium_patch16_gap_384',\n",
       " 'vit_small_patch16_18x2_224',\n",
       " 'vit_small_patch16_36x1_224',\n",
       " 'vit_small_patch16_224',\n",
       " 'vit_small_patch16_384',\n",
       " 'vit_small_patch32_224',\n",
       " 'vit_small_patch32_384',\n",
       " 'vit_tiny_patch16_224',\n",
       " 'vit_tiny_patch16_384']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# There are a few ways we can interface with PyTorchSmoothedViT. \n",
    "# The most direct way to get setup is by specifying the name of a supported transformer.\n",
    "# Behind the scenes we are using the timm library (link: https://github.com/huggingface/pytorch-image-models).\n",
    "\n",
    "\n",
    "# We currently support ViTs generated via: \n",
    "# https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/vision_transformer.py\n",
    "# Support for other architectures can be added in. Consider raising a feature or pull request to have \n",
    "# additional models supported.\n",
    "\n",
    "# We can see all the models supported by using the .get_models() method:\n",
    "PyTorchDeRandomizedSmoothing.get_models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e8bac618",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Running algorithm: salman2021\n",
      "INFO:root:Converting Adam Optimiser\n",
      "WARNING:art.estimators.certification.derandomized_smoothing.pytorch: ViT expects input shape of: (3, 224, 224) but (3, 32, 32) specified as the input shape. The input will be rescaled to (3, 224, 224)\n",
      "INFO:art.estimators.classification.pytorch:Inferred 9 hidden layers on PyTorch classifier.\n",
      "INFO:art.estimators.certification.derandomized_smoothing.pytorch:PyTorchViT(\n",
      "  (patch_embed): PatchEmbed(\n",
      "    (proj): Conv2d(3, 384, kernel_size=(16, 16), stride=(16, 16))\n",
      "    (norm): Identity()\n",
      "  )\n",
      "  (pos_drop): Dropout(p=0.0, inplace=False)\n",
      "  (patch_drop): Identity()\n",
      "  (norm_pre): Identity()\n",
      "  (blocks): Sequential(\n",
      "    (0): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (1): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (2): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (3): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (4): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (5): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (6): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (7): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (8): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (9): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (10): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (11): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "  )\n",
      "  (norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "  (fc_norm): Identity()\n",
      "  (head_drop): Dropout(p=0.0, inplace=False)\n",
      "  (head): Linear(in_features=384, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import timm\n",
    "\n",
    "# We can setup the PyTorchSmoothedViT if we start with a ViT model directly.\n",
    "\n",
    "vit_model = timm.create_model('vit_small_patch16_224')\n",
    "optimizer = torch.optim.Adam(vit_model.parameters(), lr=1e-4)\n",
    "\n",
    "art_model = PyTorchDeRandomizedSmoothing(model=vit_model, # Name of the model acitecture to load\n",
    "                                         loss=torch.nn.CrossEntropyLoss(), # loss function to use\n",
    "                                         optimizer=optimizer, # the optimizer to use: note! this is not initialised here we just supply the class!\n",
    "                                         input_shape=(3, 32, 32), # the input shape of the data: Note! that if this is a different shape to what the ViT expects it will be re-scaled\n",
    "                                         nb_classes=10,\n",
    "                                         ablation_size=4, # Size of the retained column\n",
    "                                         replace_last_layer=True, # Replace the last layer with a new set of weights to fine tune on new data\n",
    "                                         load_pretrained=True) # if to load pre-trained weights for the ViT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "353ef5a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Running algorithm: salman2021\n",
      "INFO:timm.models._builder:Loading pretrained weights from Hugging Face hub (timm/vit_small_patch16_224.augreg_in21k_ft_in1k)\n",
      "INFO:timm.models._hub:[timm/vit_small_patch16_224.augreg_in21k_ft_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.\n",
      "WARNING:art.estimators.certification.derandomized_smoothing.pytorch: ViT expects input shape of: (3, 224, 224) but (3, 32, 32) specified as the input shape. The input will be rescaled to (3, 224, 224)\n",
      "INFO:art.estimators.classification.pytorch:Inferred 9 hidden layers on PyTorch classifier.\n",
      "INFO:art.estimators.certification.derandomized_smoothing.pytorch:PyTorchViT(\n",
      "  (patch_embed): PatchEmbed(\n",
      "    (proj): Conv2d(3, 384, kernel_size=(16, 16), stride=(16, 16))\n",
      "    (norm): Identity()\n",
      "  )\n",
      "  (pos_drop): Dropout(p=0.0, inplace=False)\n",
      "  (patch_drop): Identity()\n",
      "  (norm_pre): Identity()\n",
      "  (blocks): Sequential(\n",
      "    (0): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (1): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (2): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (3): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (4): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (5): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (6): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (7): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (8): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (9): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (10): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "    (11): Block(\n",
      "      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (attn): Attention(\n",
      "        (qkv): Linear(in_features=384, out_features=1152, bias=True)\n",
      "        (q_norm): Identity()\n",
      "        (k_norm): Identity()\n",
      "        (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
      "        (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls1): Identity()\n",
      "      (drop_path1): Identity()\n",
      "      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "      (mlp): Mlp(\n",
      "        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n",
      "        (act): GELU(approximate='none')\n",
      "        (drop1): Dropout(p=0.0, inplace=False)\n",
      "        (norm): Identity()\n",
      "        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n",
      "        (drop2): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ls2): Identity()\n",
      "      (drop_path2): Identity()\n",
      "    )\n",
      "  )\n",
      "  (norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)\n",
      "  (fc_norm): Identity()\n",
      "  (head_drop): Dropout(p=0.0, inplace=False)\n",
      "  (head): Linear(in_features=384, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Or we can just feed in the model name and ART will internally create the ViT.\n",
    "\n",
    "art_model = PyTorchDeRandomizedSmoothing(model='vit_small_patch16_224', # Name of the model acitecture to load\n",
    "                                        loss=torch.nn.CrossEntropyLoss(), # loss function to use\n",
    "                                        optimizer=torch.optim.SGD, # the optimizer to use: note! this is not initialised here we just supply the class!\n",
    "                                        optimizer_params={\"lr\": 0.01}, # the parameters to use\n",
    "                                        input_shape=(3, 32, 32), # the input shape of the data: Note! that if this is a different shape to what the ViT expects it will be re-scaled\n",
    "                                        nb_classes=10,\n",
    "                                        ablation_size=4, # Size of the retained column\n",
    "                                        replace_last_layer=True, # Replace the last layer with a new set of weights to fine tune on new data\n",
    "                                        load_pretrained=True) # if to load pre-trained weights for the ViT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7a4255f",
   "metadata": {},
   "source": [
    "Creating a PyTorchSmoothedViT instance with the above code follows many of the general ART patterns with two caveats: \n",
    "+ The optimizer would (normally) be supplied initialised into the estimator along with a pytorch model. However, here we have not yet created the model, we are just supplying the model architecture name. Hence, here we pass the class into PyTorchDeRandomizedSmoothing with the keyword arguments in optimizer_params which you would normally use to initialise it.\n",
    "+ The input shape will primiarily determine if the input requires upsampling. The ViT model such as the one loaded is for images of 224 x 224 resolution, thus in our case of using CIFAR data, we will be upsampling it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "44975815",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of the ablated image is (10, 4, 224, 224)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fc35ccb6980>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAESCAYAAABdK7eSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABCAklEQVR4nO3de3QV9bk38O/Mvue2cyM3AsgdVECLGqPW10oOIXZ5tPCHeFg9qCxZ5QTWUmxtc5bFS9uVql2t1UNxnXMs4FpSWs8SfbUtFlHCsQaqqbxeUCqUCkgSLiG3nX2d+b1/xGwJmWcgkOwb389ao2Q/e/b+zczekycz88yjKaUUiIiIiBJET/YAiIiI6OLC5IOIiIgSiskHERERJRSTDyIiIkooJh9ERESUUEw+iIiIKKGYfBAREVFCMfkgIiKihGLyQURERAnF5IOIiIgSiskHEaWMtWvX4pJLLoHX60VVVRX+8pe/JHtIRDQKmHwQUUr47W9/i9WrV+Phhx/GX//6V8yZMwe1tbU4duxYsodGRCNMY2M5IkoFVVVVuPrqq/Ef//EfAADTNDFu3DisWrUKP/jBD5I8OiIaSc7ReuG1a9fiySefRFtbG+bMmYNnnnkG11xzzVnnM00TR48eRW5uLjRNG63hEZENpRR6enpQUVEBXR/9A6SRSAQtLS1oaGiIP6brOmpqatDc3Gw5TzgcRjgcjv9smiY6OjpQVFTEfQdREgxrv6FGwebNm5Xb7Va//vWv1ccff6zuvfdelZ+fr9rb28867+HDhxUATpw4pcB0+PDh0dhFDPHFF18oAOqdd94Z9Pj3vvc9dc0111jO8/DDDyd9/XDixGnodC77jVE58vHzn/8c9957L+6++24AwLPPPovf//73+PWvf33Ww6e5ubkAgLnXVMHpHDq8rq5T4rwe3RRjBW4lxioLssRYcaEcK/JnizG37hJjDo9PjPU/wSGGTnV2ibFoTF7GfL9fjOlGVIyFI2ExFgrJMa/PI8YMGGIsGAyIsTx/rhiDkl8zEpGXz2Fz8M9hsx1ysnPksQDIzpI/N06XV4yFwhExpjSbvyR0eTkiEfk1Y8r6CEEoHMEPn34h/n1MRQ0NDVi9enX8566uLowfPz6JIzp3//JPV2LmJWP6d9VnbAKH5oBD2NaawwndKexbNA3Q9P7XG9gVfPnvD/cfwcbX/jxCoyeydy77jRFPPoZ7+PTMQ6c9PT39A3M6LZMPu18IDl0+1Op0yL+Y3S75NT0ueRV53XKC4XbIMadHjgEAHPJ7Bm3eU9flZfTavKcu/96GBjmhgynPaLduDJvrnE3DZn3brTclv6YOeb04YJMk2nzWfGfZhj6vW4y5XHLM7mzB+SYfDpvXlJKPr8aTmNMXxcXFcDgcaG9vH/R4e3s7ysrKLOfxeDzweOQkN5W5XQ7xO+LQHHAIh6z1syYfX35mBxIQbeD9Ru0MO9EQ57LfGPGTuSdOnIBhGCgtLR30eGlpKdra2oY8v7GxEX6/Pz6NGzdupIdERCnO7XZj7ty52L59e/wx0zSxfft2VFdXJ3FkaUbDV0dSeNkLpbCkl9o2NDSgq6srPh0+fDjZQyKiJFi9ejX+67/+Cxs3bsQnn3yCFStWIBAIxE/fkgX5oB5RShvxY3HDPXyazodOiWjk3HHHHTh+/DjWrFmDtrY2XHHFFdi6deuQo6h0Gh7doDQ14snH6YdPb7/9dgBfHT5duXLlOb/Op59+As3ivGfniRPiPIXydXzQiuRgsSFfHKP5SsRYwOwQY72G/CeJ0uRz/gDQF5IvEOwLyhd5Rg35+owTNif+vU55rLGY/JoOm+sM7BLKvpB8UWnMlJddCxWJMV2+PAPRsLzOfE75c9Frc/FnhxGT3xBAVpZ8MbJmczGyZnOtEGxK1/pC8kW1sajNBbdO6+0Ujtov32hZuXLlsPYT6SrcexyBDuvPpTKVeERDwfHVdR1niBkGIlHr67COHe0+r3ESjZZRuQpp9erVWLp0Ka666ipcc801eOqpp3j4lIhoFNndLlLx/AylmFFJPnj4lIiIiCSjVn91sRw+JSIiouFh8TcRUYqzOmkycBsPnlChdMTkg4goxQ1cLq4sHmPBC6WjpN/ng4iIRg6PhFA6SNkjH16nBt3qduk2twSZYFNOe0mp3NukZEyhGPPZlUza3EI2GA6JsVBULv0EAGXzum6fTV8Ym94uypTf02/TvyYWtbstvTwWw+aW7Q63vBHDEXm9RWPyesmyeU1ntjxOr818MU0uCdaVzW3nAcRs/h61u915Tra8LXoDfWIsGpPLaW26DqCn27pXkFSySSND0z3QHNafy5gRhRGzXv8xw0TMtC6DVoB4f377TytR4qVs8kFElMnkP15Ov0f6YIppBGUInnYhIkoqniihiw+TDyKipOIlo3TxYfJBRERECcXkg4goqXjahS4+TD6IiJKKtwqji0/KVrt4NQO6NvTK7txcecjTxhaIsSKf3PbUZcrlnb0dcmdTw5Rzt2Cf3BVUt29qi7z8HDHmtCkN7ezqkeez2dKFuXJ5Z0+3XG4aselOG7TpsqpsznHnZMulzdFIUIzphryALpsOu4Yhj9NpUxMbDsvzAYDbJW9kXSiVBIBw7yn5RW06JXtsuvrGTLlCoitgXYIdselmTBeur6cXXUKD7kjEgCGVzTsATbeOudwueH3WnzuXXX03URKkbPJBRJSplFJQQhtaZcoxzfYAiRpSvjtwC3abWwcRJQVPuxARZSjmHJSqmHwQESXBaFzlwStHKF0w+SAiSoLROCrBIx2ULph8EBGlmvM8hKHOf1aihGLyQUSURJbJwnkewuCRD0oXKVvtku9xwKEPzY18NmWTfpvupWPyXGLMMOUOnna9PR1Om/pGi7EPCJv2ZZpOm7pYp003VSMsl6IqhzyeY8c65de06W7a0yd3We0z5BLlHF+eGENYfj+HTVMt3aYMwOGRux0HA3KZdZZLHqdTqEYYEArJyx+MyqW2ps3frZ298lg7++TPVK9N2Xcoav25iBkstR1NTocOl8t63Wu6Bmn1G8qAKXwPdKcOh8N6n6Tb7I+IkiFlkw8iooylyV1tNU3+28U0NSho8duSDX4FzfI11cAbEqUQpsNEREk2nOs0tDP+f67PJ0olTD6IiJKMCQJdbJh8EBFlKFa+UKpi8kFElMGYgFAq4gWnREQZiqdzKFWNePLxyCOP4NFHHx302PTp0/Hpp58O63WK/V44LcpDc11yeavXK8d0h5z/+3xyiW40Jpd+mjZfbaXkUsuI1LHyS0ZELps0lU23WJvyVuWUu6z2ROTutIYhr9M+m3JMu1LNnoC8DF90yGNx6fJr5vXK2yLaJrQPBRDsksuFxxdPEWMlJZViDAC03C4xFj51Uoz19srL39Ujl9qe6JLLrP9xWB6L4bDeBZhnKSWmC+Ny6PA4hVJbGGInYlMpGEJ3Y2WYMIQybmXY3TSAKPFG5cjHZZddhjfeeOOrN7Hr505EdJHpL6e1KbUVcun+QluBUv2TZYjJJKWWUckKnE4nysrKRuOliYiIKM2NygWnn332GSoqKjBp0iQsWbIEhw4dGo23ISIiojQ04kc+qqqqsGHDBkyfPh2tra149NFH8fWvfx0fffQRcnNzhzw/HA4jHA7Hf+7u7h7pIRERZZyhdzglSh8jnnzU1dXF/z179mxUVVVhwoQJ+N3vfodly5YNeX5jY+OQC1SJiC52Z0sumHhQOhv1+3zk5+dj2rRp2L9/v2W8oaEBXV1d8enw4cOjPSQiIiJKolEvQ+nt7cWBAwfw7W9/2zLu8XjgsehUW1acBbdF19g8t9yhMydLLifVbEpU7W7Do9l0kQ0H5TJN3ebvkqJcv81YgOxsuQtrd5dcNurPk7uw9oTk5f/8C/k1e8Nyqa3bpvHp2Cybzrwum7LQk51iLKzksbhsutr684ae7htw3aVXibHuVrk8UfXZVw/4i+UuyuE+ed309sp/D3hc8muOK5OXsaSkVIy1d1uX78YME4c+OiLORxfGiEYQDVvvP5QJaMq6qsXjdMHrtv4cuJwOeDzWMWkeomQZ8eTju9/9Lm699VZMmDABR48excMPPwyHw4E777xzpN+KiCgtKdOAaXPvDe3L1OPMUy8OTYPudFomJk6nAy7htgYOqU0uUZKMePJx5MgR3HnnnTh58iTGjBmDG264Abt27cKYMWNG+q2IiDJUf9phlWTwWg/KBCOefGzevHmkX5KI6CLzVYrRn4awtoUyC4/FERGlMO20/xJlCiYfRDTqHnnkEWiaNmiaMWNGPB4KhVBfX4+ioiLk5ORg0aJFaG9vT+KIiWg0MfkgooS47LLL0NraGp/efvvteOz+++/Hq6++ihdffBFNTU04evQoFi5cmMTREtFoStmObwU5PngsOtg6I53iPB6XvDhZniwxFg7KZahRUy7tzc8vEGN2jZwihn3OF43K3UuzcnLE2NHjYTF24HO5s+nxHnkZ++QQJvjk0tfbv36FGKssl5fhf1r+Lsaa97eJsZgpd/R16vK26Ok8Lsb6euX1mZt7ltJFQz5M7vXK87ptOjNnafJ8MUPeUOPHVYix3I4ey8cjUQM7R7jUVur51NXVheeeew6bNm3CzTffDABYv349Zs6ciV27duHaa68d0XGkArfHC59P3ifJn1gHlCZ9thRiMet9mWmyqy2lFh75IKKEkHo+tbS0IBqNoqamJv7cGTNmYPz48Whubk7WcEeVy+mM3+PozMntdsMjTA6nE7quWU6AglKGMNnclIcoCVL2yAcRZQ67nk9tbW1wu93Iz88fNE9paSna2uSjXewLRZS+mHwQ0aiz6/nk8/nO6zXZF4ooffG0CxEl3Ok9n8rKyhCJRNDZ2TnoOe3t7ZbXiAxgXyii9MXkg4gSbqDnU3l5OebOnQuXy4Xt27fH4/v27cOhQ4dQXV0tvobH40FeXt6giYjSA0+7ENGos+v55Pf7sWzZMqxevRqFhYXIy8vDqlWrUF1dnZGVLkSUwsnHmIJCeN1DhxfskMtQdc2mW2ifXE4bjMhlik5NLn3si8rla3aHlIJRuSwUAPIL5L/gIoZchPf3I0fFWEe3TYdWp9wN2OGQlyTPK79midO6hBMAvB1yCevUPPkwe2uhPJb2zmNiLNwnr+/3//Y3MabH5AqBaPZZ/sr2y51kocufU79fLr/MNeVtH4rIn28VkS/EvGRMtvB6NjXW5+FsPZ9+8YtfQNd1LFq0COFwGLW1tfjVr341omNIJTFTIWJYf740ABBK9WNm/7xWnC4HnEJjOc0h78eIkiFlkw8iyhxn6/nk9Xqxdu1arF27NkEjSi5DKTGJ0CDfTD1mGuJ8OnToTofl3Dq72lKK4SeSiChjsAcMpQcmH0RERJRQTD6IiIgooZh8EBEljXwRMVEmY/JBRJQ0vEaDLk6sdiEiSjAFDUpKPDRN7Fyr6w44lRBzOmHAuvOxyV09pZiU/UTmFxXD5xn6RSrIkftA6Lrccryz+5QYiwZ65dc05HtZmJDvA6Fc8qrNyfGKMQCIQo5/8nf5vhSBcECMeb0eOWZxP5UBvmz5vhMFDvleEC3728VYLCK/X9gv3+djTIG8XjTI992IxuR7w/RFgmIs0CcfEo/E7O+Dodndy8Xmj12XLgeVLt+rwSXc3wEAYmH5vipKuG+M9DiNDAUdSriHkHLo0M4ojR3YGk6HE26H9bZWcMAUkw/5Xj5EyZCyyQcREfXjyRnKNLzmg4gorfEoFaUfJh9ERGmNx0Uo/TD5ICIiooRi8kFEREQJNezkY+fOnbj11ltRUVEBTdPw8ssvD4orpbBmzRqUl5fD5/OhpqYGn3322UiNl4go7TmcTjhdbsvJ4XRBdziFyQXd6bacoDkRi8FyMuXCPKKkGHa1SyAQwJw5c3DPPfdg4cKFQ+JPPPEEnn76aWzcuBETJ07ED3/4Q9TW1mLv3r3weu1LTAfRnYBF6azmkstp7Xi88nxZsG4rDgBOm/zMrlNk1KYM1+PzizEAONEmt6PvOyGXDE8qlNdvWK42hdemnHb65LFiTLd50ZhDXt/dNmXPTkeXGMt1y9upqGCyGJs8dbwYO3joXTH26d++EGNup1y+CgBKyeXbsZj8tdOdckmkyy2vU9Pmt4tpc02Apll/hqXHaWS4vVnwZVuXh8dgipeQOpweOF3W3/NgXwThgPXnMmpfGU6UcMNOPurq6lBXV2cZU0rhqaeewkMPPYTbbrsNAPD888+jtLQUL7/8MhYvXnxhoyUiIqK0N6J/3hw8eBBtbW2oqamJP+b3+1FVVYXm5uaRfCsiIiJKUyN6k7G2tjYAQGlp6aDHS0tL47EzhcNhhE+7A2N3d/dIDomIiIhSTNJP7DY2NsLv98encePGJXtIRERENIpGNPkoK+vvy9HePrivR3t7ezx2poaGBnR1dcWnw4cPj+SQiIiIKMWM6GmXiRMnoqysDNu3b8cVV1wBoP80yu7du7FixQrLeTweDzweuekZEVGmMQwTkah108qYMmAo6+olNxzQHNbzKdOAUkIjTOH1iJJl2MlHb28v9u/fH//54MGD2LNnDwoLCzF+/Hjcd999+PGPf4ypU6fGS20rKipw++23D+t9QqEYYNE6WovKXUgBuZ4sEJCvJYlE5QNAMV0uX+3tk0tiu21iY8fZr3YVk+edUCyXTU6ukEsx+0LyfGOnzRFjbiWX057qiooxX36RGMNJuTvruLJyMdYZkLv2TpoxVYzlFcilxHkFM8XYqePydjjVJZcEA4DLpixYV3KyHTVtuijb/P4wbGopbRrlQimhq63wOI2M3r4wOnv6LGPBcAhRw/q7lZuXhzy/9QaNRsOAYf19VaZ9aThRog07+XjvvffwjW98I/7z6tWrAQBLly7Fhg0b8OCDDyIQCGD58uXo7OzEDTfcgK1btw7vHh9ERDQimEZSKhp28nHTTTfZ/lWkaRoee+wxPPbYYxc0MCIiIspMSa92ISKi0cOet5SKmHwQERFRQjH5ICLKQLzWg1LZiJbaEhHR2cUMExGhQskwTRiGdWmTYRiIxawrYYxYDEbUOmYachUVUTKkbPJhaAYMi86aypBLCu0uhPV5fWIsJ1cuxTx6XC7tPXjkuBhzuuSxuNuPijEACLXLrzu1RC6nnXeTXG564IsOMZY7dowYKy6yvjkcABw73i7G8vNtSk1NeRnculyGe+y43GXW6e0UY8c7W8XYF61y91mXS/5c5OfZ3zchGJS3v3LKBxw1m7pY06YMV9dsOtfadF82+OdxUpzs7IFHs/4M6Q4FaZOZykRMuN1ArC+IWI/15znQZdPWmigJeNqFiChTMbmkFMXkg4goGRKRGLDUhVIUkw8iomTQ0J+AnGsSwqMYlEFS9poPIqKMN5wjEzyKQRmERz6IiIgooXjkg4gowZRSMIVOgZoOSIV7ylQwYlJXWyWemuFBE0o1KZt8+P3Z8HndQx6POeVS295euZxMCe2rAaCrR+5Q+vkhuZy0t1cu0/R55YNKrQflDrsAUGqx3APGjp0gxvIrJooxV49NaahXLn2tnHONPFubXPrqi8nlwgbk7RQIyLHyLLkkOCLcFwEAtOwcMVaZXSHGcvPlMuOek21iDACOtZ8UY1FNXt+hiE33UV0+6Z/tkRs3RoI25cRu67EY/HU1qsKhMPqsm9rC5dTgdAqdawMB9AhJi8+hI89lvUt3WHQIJ0omnnYhIkoHw73glBeoUgpj8kFElA6Ge/BioJqGKAUx+SAiylQ820IpiskHERERJRSTDyKiTKPO+D9RimHyQUQXZOfOnbj11ltRUVEBTdPw8ssvD4orpbBmzRqUl5fD5/OhpqYGn3322aDndHR0YMmSJcjLy0N+fj6WLVtmW02W9mxOh2haf1ia5Jc8bUZ98P81m8aDRMmQsqW2vV0diIWGlgE6Iz3iPC6LLrhxcrNUOB1ysK9XLsMtyJU7t+Zny6WPwVP2pbYlFUVibOzs/yPGPjoSEWN/2y/HrisvFGOdnfJ8pZPniDEdQh0hgEhYLsPNV3LJbPcxuXzVF7FuJQ4A5YU2y2d4xJhrdoEYC9p0ygWAP//h/4qxI4fl5XcIpa/95F8gNk10EbX5G0MXWrCHhHbvVgKBAObMmYN77rkHCxcuHBJ/4okn8PTTT2Pjxo2YOHEifvjDH6K2thZ79+6F19v/PVmyZAlaW1uxbds2RKNR3H333Vi+fDk2bdp0zuNIJzleDwqEbto+txMeoWQ2GgwhFrL+TubmeFBcaF1WfkJ1ATh1XmMlGg0pm3wQUXqoq6tDXV2dZUwphaeeegoPPfQQbrvtNgDA888/j9LSUrz88stYvHgxPvnkE2zduhXvvvsurrrqKgDAM888g1tuuQU/+9nPUFEh34uFiNITT7sQ0ag5ePAg2traUFNTE3/M7/ejqqoKzc3NAIDm5mbk5+fHEw8AqKmpga7r2L17t/ja4XAY3d3dgyYiSg9MPoho1LS19d8JtrS0dNDjpaWl8VhbWxtKSkoGxZ1OJwoLC+PPsdLY2Ai/3x+fxo0bN8KjJ6LRwuSDiNJSQ0MDurq64tPhw4eTPSQiOkdMPoho1JSV9ffHaW8f3COpvb09HisrK8OxY8cGxWOxGDo6OuLPseLxeJCXlzdoIqL0wOSDiEbNxIkTUVZWhu3bt8cf6+7uxu7du1FdXQ0AqK6uRmdnJ1paWuLPefPNN2GaJqqqqhI+5kRQ6L8Y12rCWWISzaYayi5GlAzDrnbZuXMnnnzySbS0tKC1tRVbtmzB7bffHo/fdddd2Lhx46B5amtrsXXr1mG9j64BDovvi2HToVPZfMF0yKWDhiaX2p6SKzjR3S3vDFRYLlEt98slugBw9Te+IcYqp18rxl5a/2sxVmbT2dURCYqxL/5+QH7NSZeKMW/RFDGWreRy6b6OY2LMZ8qlr5GgXNp7okeO5Y+ROwEXlV0ixoK99n9l6zZhwy137tV0+TMcjcqfKU1osw4AmpJjsZhQ0mmc+92pent7sX///vjPBw8exJ49e1BYWIjx48fjvvvuw49//GNMnTo1XmpbUVER32/MnDkTCxYswL333otnn30W0WgUK1euxOLFizO20qWrT0GopkVuMAafw7rkPMfjQEGe9f6jcuJ4zLziMvSnNoM/R5H/tx/Y/fkFjJhoZA07+ThbTT8ALFiwAOvXr4//7PHI91IgovT23nvv4RunJcyrV68GACxduhQbNmzAgw8+iEAggOXLl6OzsxM33HADtm7dGr/HBwC88MILWLlyJebNmwdd17Fo0SI8/fTTCV+W9MdbmlJ6GHbyYVfTP8Dj8dieqyWizHHTTTfZnhLQNA2PPfYYHnvsMfE5hYWFGXtDscQ6s5Xt0KMgRKlgVK752LFjB0pKSjB9+nSsWLECJ0/Kd6ZkrT4R0UjShH8TpY4RTz4WLFiA559/Htu3b8fjjz+OpqYm1NXVwTCszzuzVp+I6MLwZAulmxG/vfrixYvj/541axZmz56NyZMnY8eOHZg3b96Q5zc0NMTPEQP9V8IzASEiIspco15qO2nSJBQXFw+6Gv50rNUnIrowPLlC6WbUG8sdOXIEJ0+eRHl5+bDm01T/dCZD6MIJAJou51JOmzRLBW1eU26yisIi666UAFCWJZf2fu2qafKLAph5nVxOe+qYXGrsickdeCdVVoox02Yhy0rGiLFYSF7GPptuuJGYPF80KH8kDcjlwge+OCLGPvzoPTF23bXyOIvK5O7C3T1ySTAAuOSPBoovkUutTZvPsBGxKZm1Ke3uOt4pxsI91gMNR+X3oguXk+NBgd9nGSvwmMgWSm0rxhajXOhCXT55GiZfcbVl7POQDuAP5zVWotEw7OTDrqa/sLAQjz76KBYtWoSysjIcOHAADz74IKZMmYLa2toRHTgRUTrTNJubgl1AbOBvtoG6Fx4VoVQ07OTDrqZ/3bp1+OCDD7Bx40Z0dnaioqIC8+fPx49+9CPe64OIaKTZZBdMPCiVDTv5OFtN/+uvv35BAyIiutidU+Iw8KQznqwBUOrLh5h9UIpibxciohRzTjmDdsb/v6S+fEwx8aAUxuSDiCiDaOABD0p9TD6IiJLlzDPY53u3MIv5mIBQKhv1UtvzZcYMmI6huVEwLJeFum06tzqdLjHm0OUyxSllcidVr0/O3S6ZIN8obc4NctdaACifPluM7WleL8bGj5PHWnbZLDHmHjNZjDmz/GKsLySX/Qa75c617UcPi7FT7XLJrBGVu9P6cr1irLhY3vaHj74vxkrLx4qxWJ+87ACggmExpgVOiTFDyR2GlVXt+Zd8HnkZ3WVyrNtj/SsqFOGvrtFU4NNQmmu9jkuKCpGXY12GO2n6NFwyzbpjdH7FeJRMse40XfBp+/kNlGiUpGzyQUSUyUar1FYInPO4iBKBp12IiDLFmQ1tiVIUkw8iokyhfZVzsNqFUhmTDyKiDCJU4BKlFCYfRERElFBMPoiIiCihUrbaxeVwwuUYOrxTPXK5pRGSDzT6sqxL1wDAoctXZpXYdK493NopxiZ/bYEYq5wlx/rJJbPRnoAY8+fKZbFjpl0hxgJO6y6ZAPDx+++KsXBQHkt3d6cYO/HFITHmMOSyZ69X/riOnSiXxc4WShMBIOaQO8y6HPlyzC13QgYAZygkxvo+/0KMmTGbzrU2fyr0OhxiLKtIXsbSCuvOvcEQu9qOpr6+MLq6rL8/Uy6di4lTZ1rGxk+ZioopUy1jLq8XQP+2HnrndfkzQJQMKZt8EBFlKvtCFA26bp1MarpDjmk6BlINXvdBqY6nXYiIMhlLbikFMfkgIspkPPxBKYjJBxFRJuGRDkoDTD6IiDIJj3RQGmDyQURERAmVstUukVAYujm03C/LIw9Z88rlhi49JsaUIcd8OfJr/vMd/yzGrqubJ8byikvFGAC0//0TMeawWY7Oni4xdvwf+8TY0R65rHLHyy+LsRyf3C01FJa7vpaVyiXBeblySeDBI3I33IjNeimsuESMTZs1V4zB8Iihjk65+y4A9NmUfZ8KymPVlPz5DgXljs69Sj7Wrnrlst+Z+cJ7yRXPNAK+VvU1fO3SCZaxmV+7DmMnTrOM+XIL4MuxLsUP9XSh48hBy1jvyWPnN1CiUZKyyQcRUaZyuZ3weD04/U4c8ZjHDZfHbTmfw+mErssHrJVpnaBKjxMlC0+7EBElDS/QoIsTkw8ioqRRGInyFBa4ULrhaRcioqQZmSMfPH5C6YZHPoiIMhSPiFCqYvJBRJSBTm8uR5RqhnXapbGxES+99BI+/fRT+Hw+XHfddXj88ccxffr0+HNCoRAeeOABbN68GeFwGLW1tfjVr36F0lL78tIzmSoCU1lcoW1RfjtAi8lXdMeU3IVU0+Svp9eTJ8aumCuXaXpcchnq3j3vizEAOHX0gBgLh+WyyZ5THWLs8P69YqxXyR1/XYb8fjlOuQw5zyuXzI4pkEttW9vbxFgsKm/Dvh65tPfwQbmLLvCxGOnt7RFjXqf9Lj3mKRFjJ2PyZ8rn84qxrFx5O/mccllwT1+3GIuZ1mW/MZvvGV24yvETMOOyyyxjRcXFcDmtd81dJ0/iiNAVOdTbhb7O45ax4+2t5zdQolEyrCMfTU1NqK+vx65du7Bt2zZEo1HMnz8fgcBXraHvv/9+vPrqq3jxxRfR1NSEo0ePYuHChSM+cCKidOVyueD1ei0np8MBXdMsJ8OIIRIOW0+RMKLRiOVkxOR7yxAlw7COfGzdunXQzxs2bEBJSQlaWlpw4403oqurC8899xw2bdqEm2++GQCwfv16zJw5E7t27cK11147ciMnIiKitHRB13x0dfXfUbOwsBAA0NLSgmg0ipqamvhzZsyYgfHjx6O5udnyNcLhMLq7uwdNRERElLnOO/kwTRP33Xcfrr/+elx++eUAgLa2NrjdbuTn5w96bmlpKdrarM/lNzY2wu/3x6dx48ad75CIiIgoDZx38lFfX4+PPvoImzdvvqABNDQ0oKurKz4dPiz37yAiIqL0d143GVu5ciVee+017Ny5E5WVlfHHy8rKEIlE0NnZOejoR3t7O8rKyixfy+PxwOORr9QnIroYDe36QpQ5hpV8KKWwatUqbNmyBTt27MDEiRMHxefOnQuXy4Xt27dj0aJFAIB9+/bh0KFDqK6uHubQzC+nMx6Nye02na4sMWbE5NLBCOQrwUv91h0kAeD1//uaGCsslUs4S8rtTy1F+uTutC6XnKjlZMslnE5dLovNtikLLispEmPBnlNizOeQx3ny+AkxFo3I2ynXK5eaRnrlUtvP3n9PjLV++jcxFo4FxRhc8voEAMNufVfKZcjIlj/fukcue/YKJbMAUAB5vc28bKLl433BKID/J853up07d+LJJ59ES0sLWltbsWXLFtx+++3x+F133YWNGzcOmqe2tnbQBewdHR1YtWoVXn31Vei6jkWLFuGXv/wlcnJyzmkM6Wbf3r3INgOWsZz8EniyrL/LLo8PLrf19jQiIURD1q8ZCfad30CJRsmwko/6+nps2rQJr7zyCnJzc+PXcfj9fvh8Pvj9fixbtgyrV69GYWEh8vLysGrVKlRXV7PShShDBQIBzJkzB/fcc49YVr9gwQKsX78+/vOZRzuXLFmC1tbWeAn/3XffjeXLl2PTpk2jOvZk6esNoKuz0zIWNRzwBq3vaZOdk4fsXOvjIUY0DNOwTkJN3reFUsywko9169YBAG666aZBj69fvx533XUXAOAXv/hF/C+X028yRkSZqa6uDnV1dbbP8Xg84qnXTz75BFu3bsW7776Lq666CgDwzDPP4JZbbsHPfvYzVFRUjPiYiSi5hn3a5Wy8Xi/Wrl2LtWvXnvegiCiz7NixAyUlJSgoKMDNN9+MH//4xygq6j+l19zcjPz8/HjiAQA1NTXQdR27d+/Gt771LcvXDIfDCIfD8Z9Zpk+UPtjbhYhG1YIFC/D8889j+/btePzxx9HU1IS6ujoYRv+pgLa2NpSUDL4dvdPpRGFhoViiD7BMnyidnVe1CxHRuVq8eHH837NmzcLs2bMxefJk7NixA/PmzTvv121oaMDq1avjP3d3dzMBIUoTPPJBRAk1adIkFBcXY//+/QD6S/SPHTs26DmxWAwdHR3idSJA/3UkeXl5g6bMo4R/E6W3lD3yYZoaTHPoVd1um06qXqfc1Ra6XDGvHHLpoxmRO6meOCEfEu49Lsd8Uftz0ybkZSwskEtf8yvGiLGYERZjXxyVx6psdni6Ln98IjaNrByaXNqb7ZXLpW2aFsNhF7TpWmxE5LJm3eLzN6C7Ty4zBoCIRy7Tza2Qt0XA1ynGeky5DDcUkP+OKMqbJMaKhVLqQEB+rwt15MgRnDx5EuXl5QCA6upqdHZ2oqWlBXO/7BT95ptvwjRNVFVVjdo4kqmvpxPdJ623WainWyypjxUUQSsstowpU1l3AgcQ65NL0YmSIWWTDyJKD729vfGjGABw8OBB7NmzB4WFhSgsLMSjjz6KRYsWoaysDAcOHMCDDz6IKVOmoLa2FgAwc+ZMLFiwAPfeey+effZZRKNRrFy5EosXL87YSpdYJIxIyDpBNaMxxBxOWN1mLMvthOGzTkwUNCjhYLYZk/+IIkoGnnYhogvy3nvv4corr8SVV14JAFi9ejWuvPJKrFmzBg6HAx988AH++Z//GdOmTcOyZcswd+5c/O///u+ge3288MILmDFjBubNm4dbbrkFN9xwA/7zP/8zWYuUAs5MPHjKhTILj3wQ0QW56aabbMvwX3/99bO+RmFhYcbeUOz8nHnKjzdap8zCIx9ERGmFR0Eo/TH5ICJKKzwKQumPyQcRERElVMpe86FrHuja0OF5PXKHTmXTnTbbJ5dwZudal64BQF9U7iRalOsWY06bsUS62sUYAJi6/Lp9LrmktLTUukMpAJgRuXRy+uxKMfbOW9vFWETJnTJdmvzXWbBXni8vV75Xg9spf1wdmrxeekPyNjzYKpfMdnbK2zCsWXcPHTBmmpzXj8236c6r5G1/6oS83twhm/LlsTadifusG44Fg2xENpq8Lh3ZHuuSeocOODTr9e8wQ4j1WZfqK02H0oRqF5v9GFEypGzyQUSUqVxOHV63dfKhQ0EXruvQzSjM6GkluqcVxSjoUMK9d1hqS6mGp12IiNIVL/+gNMXkg4goE7EohlIYkw8ioqQZxQxh4KgIkxBKQUw+iIiSJgHnTXhqhlIQkw8ioqTioQm6+KRstYvLqcHtHJob9YXljqAOr013Wod1MyYA6IvKHUgdLnnH4HHLJZMulzwWd5ZfjAGAP0+et+24XKbbN1YumS0ZN0WMfXHshBi77OrrxVjv8aNi7O9/+1iMBXo7xZjTIW8Lv18uw9Ugl9q2fiGP89DnNl1tPfJ2yCuVS7cBYEyhzVhtSn+1Dvk9C07JX9exJYVirDJf/lzs32vd0TgYYnXEaHI4HHAJpeO65oAulMw63T44hI63usMJp8u65Nrrlfd/RMmQsskHEVGmcjoccDmtEwWHww1dd1k1tf0y+fBazud2OeETOt56PfL9Y4iSgaddiIhS0Yhdq8HTOpR6mHwQEWU0XnFKqYfJBxERESUUkw8iIiJKKCYfREQZhdd4UOobVvLR2NiIq6++Grm5uSgpKcHtt9+Offv2DXrOTTfdBE3TBk3f+c53RnTQRETpTJmAKUxK6ejfNQ+dTMNELBqxnIxYDEbMgBEzv/z/V5NpyqXoRMkwrFLbpqYm1NfX4+qrr0YsFsO///u/Y/78+di7dy+ys7+6P8G9996Lxx57LP5zVpb9PRGslBTpyPIOzY2iJ0+K8wQN+QsWsOmArnS5fbjTpo17Xp7cqtwt1NsDQDBg3RJ7gM9ls1kicuy9d94RY5Omy/cHOXLE+l4PAKDr8sVqWR55GR0291Xx+eR7WQR65ft8BINyLBaLiLEcofwQAK67cpoY8+bK9+qIOWJiDACMaJ8YCx6W7/Oh91iXUQJASVauGLty2mXyfPmlYqyl9aDl46GI/fLRhTGVBtMUvltKhwbrjremYSCqrPdXOhTcwv7KjMn7OKJkGFbysXXr1kE/b9iwASUlJWhpacGNN94YfzwrKwtlZWUjM0IiIiLKKBd0zUdXV//dIQsLB99d8YUXXkBxcTEuv/xyNDQ0oK9P/iswHA6ju7t70EREROeIl3hQGjrvO5yapon77rsP119/PS6//PL44//yL/+CCRMmoKKiAh988AG+//3vY9++fXjppZcsX6exsRGPPvro+Q6DiOjixtt4UBo67+Sjvr4eH330Ed5+++1Bjy9fvjz+71mzZqG8vBzz5s3DgQMHMHny5CGv09DQgNWrV8d/7u7uxrhx4853WERERJTiziv5WLlyJV577TXs3LkTlZVy0yoAqKqqAgDs37/fMvnweDzweNj0iIgozqKvC1EmGVbyoZTCqlWrsGXLFuzYsQMTJ0486zx79uwBAJSXl5/XAImIMo1pGojFrDsH67oOBaGiRQM0KSlRJpRpfQFIKCR3AydKhmElH/X19di0aRNeeeUV5Obmoq2tv0TT7/fD5/PhwIED2LRpE2655RYUFRXhgw8+wP33348bb7wRs2fPHtbAKivdyPENLeX0a3Ip4v7D8oWt7cflq7IihnzkJSdHXkWBPrkdu2H2ijHHWa7z7TgulxP39MolkKGoPB6HkmO5OQVirL2tQ4wdCcglo6aS/2wrHSOXKGum3Mr9VOcpMebJlrdhvl8uUXU75G0RjtiUJwodSQcEwvLrRnrlebNNeb4p4+QKsooyeZ0ePiKXWZ88bv2dCUdZmjmaItEQwhHrdW+YQTid1p8DDdqXe4+h36+AocOMfVmie8aRk1On5O8xUTIMK/lYt24dgP4biZ1u/fr1uOuuu+B2u/HGG2/gqaeeQiAQwLhx47Bo0SI89NBDIzZgIqKLm1Vir87+FKIUMuzTLnbGjRuHpqamCxoQERENF7MNSi/s7UJEREQJxeSDiIiIEorJBxERESXUed9kjIiIzpOSr6FTSoMplMxqkOczDUDqrWkqdrWl1JKyyUdevgs5WUNLEoNCaSAAFJRYd4IEAGTLnXVPtMs18KGI3C3V6Za7ntrMBvMsZYxRQx5PV1AuN8226d4a6pPLYoOhE2IsYjNWwyamlLwtervlbZiX57OJ+cVYMCi/5omT8jrLyZE77Gq6fGBQi9lffO12ysvhkavF4XbL6+2SKZeIsWCfPJ6dO/eKsQ/+dszy8ZhNh2i6cKZpwjCk74+CUtafPdMAIGwaw4zBFBKTqE3XZ6Jk4GkXIiIiSigmH0RERJRQTD6IiIgooZh8EBERUUIx+SAiIqKEStlqFyKijGVzN3RN06Fp1lVPugZIPRs13QSkChqxFS5RcqRs8uHwOuH0Dh2eN88tzlOYIx/IcQbl8lWXTy4r7D5ls4oM+f183hJ5Npd9GaMR7hRj7ix5PC6nvG4cDrnUOGxzD4BIVC7RUzadazWbSlQVkct+DTkEl10nWbdcZtx5Si61DUbkLrr+fLmU2mlThgsAus226IPcmbj9RI8YO2XT0bgnIHctfmPHp/L7CRXK0n0mrDQ2NuKll17Cp59+Cp/Ph+uuuw6PP/44pk+fHn9OKBTCAw88gM2bNyMcDqO2tha/+tWvUFpaGn/OoUOHsGLFCrz11lvIycnB0qVL0djYCKczZXdT5013aHC6rT9DbmcOXE7r76tpmpBabEWNMFQsaBnTHDa3ISBKAp52IaIL0tTUhPr6euzatQvbtm1DNBrF/PnzEQgE4s+5//778eqrr+LFF19EU1MTjh49ioULF8bjhmHgm9/8JiKRCN555x1s3LgRGzZswJo1a5KxSBnk3JNIokTKvD8piCihtm7dOujnDRs2oKSkBC0tLbjxxhvR1dWF5557Dps2bcLNN98MAFi/fj1mzpyJXbt24dprr8Wf/vQn7N27F2+88QZKS0txxRVX4Ec/+hG+//3v45FHHoHbLR9JIjs83UKpiUc+iGhEdXX1nwIqLCwEALS0tCAajaKmpib+nBkzZmD8+PFobm4GADQ3N2PWrFmDTsPU1taiu7sbH3/8cQJHT0SJwCMfRDRiTNPEfffdh+uvvx6XX345AKCtrQ1utxv5+fmDnltaWoq2trb4c05PPAbiAzEr4XAY4fBX13J1d3eP1GKkNAUez6D0xyMfRDRi6uvr8dFHH2Hz5s2j/l6NjY3w+/3xady4caP+nqmAiQdlAiYfRDQiVq5ciddeew1vvfUWKisr44+XlZUhEomgs7Nz0PPb29tRVlYWf057e/uQ+EDMSkNDA7q6uuLT4cOHR3BpRpeuK2ia9QTNhFKGMJkwlWE5KWVCmcLE604pxaTsaZdArxOaaVFa6cgR58nJlus0XT7525dt02bU75fLUHu7rcva+mPtcqzvLF1tQ3I8110kxrwuuRQ1FpZLjZ1OOQcVqgEBAC6PXL6nafKMWTnyx063+UTGDLnU1O2TZ8zLl8uMOzrk0tYemxLkvEJ5OwBAn00X0c/+cVKMffqh/Au0tFAu/S2tlJcRurwcxf5cy8cN08Tnp2zqnk+jlMKqVauwZcsW7NixAxMnThwUnzt3LlwuF7Zv345FixYBAPbt24dDhw6huroaAFBdXY2f/OQnOHbsGEpK+svUt23bhry8PFx66aWW7+vxeODxyCXWqSwrN4r8Yuv1GwqaiEYDlrFw1EQ0ar0vM2ImTKHbcjQif/+JkiFlkw8iSg/19fXYtGkTXnnlFeTm5sav0fD7/fD5fPD7/Vi2bBlWr16NwsJC5OXlYdWqVaiursa1114LAJg/fz4uvfRSfPvb38YTTzyBtrY2PPTQQ6ivr0/bBGPk8fAFZQ4mH0R0QdatWwcAuOmmmwY9vn79etx1110AgF/84hfQdR2LFi0adJOxAQ6HA6+99hpWrFiB6upqZGdnY+nSpXjssccStRhpQMPQBISXn1J6YvJBRBdEncMFBV6vF2vXrsXatWvF50yYMAF/+MMfRnJoFwEmHpSeeMEpERERJRSTDyKiZLA6gzJqL06UWoaVfKxbtw6zZ89GXl4e8vLyUF1djT/+8Y/xeCgUQn19PYqKipCTk4NFixYNKZ8jIrrYaZqCdma5rU357enP6U8shk79jWsHfsYZcaLUMqxrPiorK/HTn/4UU6dOhVIKGzduxG233Yb3338fl112Ge6//378/ve/x4svvgi/34+VK1di4cKF+POf/zzsgR09DGRZVMCGO+Wy2Nwxcimm12fTvVSu3kVhobyKegNCS1AAnZ1y7NRJ+z4Vp+RKTDhMubzVtDn3bkittgHAlGN22ammy+ebHTadSIM23YCVvAnhMuVtGOvrEGNGUN4Whk2n3M5eeb6IfbU0OmzKsP+xX97AnSetSywBIBKQ37TMb30vDACYOWGsGJOGGTVM/PUf8jqlC1M8JoJxE6xXfl9fEBGhUjvY50AoaL0PCIWAoPDxcXvl0m+iZBhW8nHrrbcO+vknP/kJ1q1bh127dqGysvKszaOIiIiIzvuaD8MwsHnzZgQCAVRXV59T8ygiIiKiYZfafvjhh6iurkYoFEJOTg62bNmCSy+9FHv27Dlr8ygrF2tzKCIioovVsI98TJ8+HXv27MHu3buxYsUKLF26FHv37j3vAVyszaGIiIguVsNOPtxuN6ZMmYK5c+eisbERc+bMwS9/+ctzah5lJZ2bQxEREdHwXfB9PkzTRDgcHtQ8asCZzaOseDyeeOnuwERERMNjf69T3gmVUsuwrvloaGhAXV0dxo8fj56eHmzatAk7duzA66+/fk7No4bDcBXBcA1tKBV1XyXOEzblzo167IQY8/rlL2b+GLm0t0CX60IL++ROop0dPjEGAJ0n5HLaYEDeZEbMpoRXyXmmGZPHGgrKnU3dbvn9HE55GXpC8vsFe206Eyu5XDBXt+7OCgCmLl9HFI3K69OTLZcuey0+m6fLd8tjnYR8MTZrTrYYmz57jhi7ZMoUMXbNtXLJ8JGjvZaPhyMx4K//EOejC1M8thLjppVYxsJhEzGhqjrY50Q4ZF0eHg5qCAasv+fHzS8AHD2foRKNimElH8eOHcO//uu/orW1FX6/H7Nnz8brr7+Of/qnfwJw9uZRRERERMNKPp577jnb+Lk0jyIioq+wLy1djNjbhYgoiZh40MWIyQcRERElFJMPIiIiSqhh3+F0tKkvm6P1hayrBYLC4wCgueTGY6YpV6boffKBT2dAfk3ocqOvQFCu6AgEbV4TQJ9dNUhIrsCwWUTY5Zm21S5heayGsmksZ9PILhiWlyEUkd9PKTnmtKk8CkXkWNhunWnyOB1KruYBgHBUfuGIVMoAwGUzn/SdAIDegFwlFLTZhmFh3QyMX9k0K0w16TTWYMhAb+DLdYyvTr0oAJGwCekrGepTCAtFfeGghlDI+nsubWei0XAu30VNpdg39siRI7zLKVGKOHz4MCorK5M9jHPy97//HZMnT072MIgueuey30i55MM0TRw9ehS5ubnQNA3d3d0YN24cDh8+zBuQnYbrRcZ1Y20460UphZ6eHlRUVEDX0+PsbGdnJwoKCnDo0CH4/f5kD+eCZMpnmMuRekZzWYaz30i50y66rltmTLz7qTWuFxnXjbVzXS/p9gt8YGfn9/szZrtnymeYy5F6RmtZznW/kR5/0hAREVHGYPJBRERECZXyyYfH48HDDz8Mj8e+l8bFhutFxnVjLdPXSyYtX6YsC5cj9aTKsqTcBadERESU2VL+yAcRERFlFiYfRERElFBMPoiIiCihmHwQERFRQqV08rF27Vpccskl8Hq9qKqqwl/+8pdkDynhdu7ciVtvvRUVFRXQNA0vv/zyoLhSCmvWrEF5eTl8Ph9qamrw2WefJWewCdTY2Iirr74aubm5KCkpwe233459+/YNek4oFEJ9fT2KioqQk5ODRYsWob29PUkjTpx169Zh9uzZ8ZsIVVdX449//GM8nqnrJd32F4888gg0TRs0zZgxIx5P1e00Evukjo4OLFmyBHl5ecjPz8eyZcvQ29ubwKXod7Zlueuuu4ZsowULFgx6Tiosy0jtDw8dOoRvfvObyMrKQklJCb73ve8hFhudvkApm3z89re/xerVq/Hwww/jr3/9K+bMmYPa2locO3Ys2UNLqEAggDlz5mDt2rWW8SeeeAJPP/00nn32WezevRvZ2dmora1FKCQ3GssETU1NqK+vx65du7Bt2zZEo1HMnz8fgUAg/pz7778fr776Kl588UU0NTXh6NGjWLhwYRJHnRiVlZX46U9/ipaWFrz33nu4+eabcdttt+Hjjz8GkJnrJV33F5dddhlaW1vj09tvvx2Ppep2Gol90pIlS/Dxxx9j27ZteO2117Bz504sX748UYsQd7ZlAYAFCxYM2ka/+c1vBsVTYVlGYn9oGAa++c1vIhKJ4J133sHGjRuxYcMGrFmzZnQGrVLUNddco+rr6+M/G4ahKioqVGNjYxJHlVwA1JYtW+I/m6apysrK1JNPPhl/rLOzU3k8HvWb3/wmCSNMnmPHjikAqqmpSSnVvx5cLpd68cUX48/55JNPFADV3NycrGEmTUFBgfrv//7vjF0v6bi/ePjhh9WcOXMsY+mync5nn7R3714FQL377rvx5/zxj39UmqapL774ImFjP9OZy6KUUkuXLlW33XabOE+qLsv57A//8Ic/KF3XVVtbW/w569atU3l5eSocDo/4GFPyyEckEkFLSwtqamrij+m6jpqaGjQ3NydxZKnl4MGDaGtrG7Se/H4/qqqqLrr11NXVBQAoLCwEALS0tCAajQ5aNzNmzMD48eMvqnVjGAY2b96MQCCA6urqjFwv6by/+Oyzz1BRUYFJkyZhyZIlOHToEID0/fyeyz6pubkZ+fn5uOqqq+LPqampga7r2L17d8LHfDY7duxASUkJpk+fjhUrVuDkyZPxWKouy/nsD5ubmzFr1iyUlpbGn1NbW4vu7u74UdORlJLJx4kTJ2AYxqCVAAClpaVoa2tL0qhSz8C6uNjXk2mauO+++3D99dfj8ssvB9C/btxuN/Lz8wc992JZNx9++CFycnLg8Xjwne98B1u2bMGll16akeslXfcXVVVV2LBhA7Zu3Yp169bh4MGD+PrXv46enp603U7nsk9qa2tDSUnJoLjT6URhYWHKLduCBQvw/PPPY/v27Xj88cfR1NSEuro6GIYBIDWX5Xz3h21tbZbbbSA20lKuqy3RcNXX1+Ojjz4adL78Yjd9+nTs2bMHXV1d+J//+R8sXboUTU1NyR4Wnaauri7+79mzZ6OqqgoTJkzA7373O/h8viSOjAYsXrw4/u9Zs2Zh9uzZmDx5Mnbs2IF58+YlcWSydNkfpuSRj+LiYjgcjiFX4ra3t6OsrCxJo0o9A+viYl5PK1euxGuvvYa33noLlZWV8cfLysoQiUTQ2dk56PkXy7pxu92YMmUK5s6di8bGRsyZMwe//OUvM3K9ZMr+Ij8/H9OmTcP+/fvTdjudyz6prKxsyIXAsVgMHR0dKb1sADBp0iQUFxdj//79AFJvWS5kf1hWVma53QZiIy0lkw+32425c+di+/bt8cdM08T27dtRXV2dxJGllokTJ6KsrGzQeuru7sbu3bszfj0ppbBy5Ups2bIFb775JiZOnDgoPnfuXLhcrkHrZt++fTh06FDGrxsrpmkiHA5n5HrJlP1Fb28vDhw4gPLy8rTdTueyT6qurkZnZydaWlriz3nzzTdhmiaqqqoSPubhOHLkCE6ePIny8nIAqbMsI7E/rK6uxocffjgomdq2bRvy8vJw6aWXjsqgU9LmzZuVx+NRGzZsUHv37lXLly9X+fn5g67EvRj09PSo999/X73//vsKgPr5z3+u3n//ffX5558rpZT66U9/qvLz89Urr7yiPvjgA3XbbbepiRMnqmAwmOSRj64VK1Yov9+vduzYoVpbW+NTX19f/Dnf+c531Pjx49Wbb76p3nvvPVVdXa2qq6uTOOrE+MEPfqCamprUwYMH1QcffKB+8IMfKE3T1J/+9CelVGaul3TcXzzwwANqx44d6uDBg+rPf/6zqqmpUcXFxerYsWNKqdTdTiOxT1qwYIG68sor1e7du9Xbb7+tpk6dqu68886UWpaenh713e9+VzU3N6uDBw+qN954Q33ta19TU6dOVaFQKKWWZST2h7FYTF1++eVq/vz5as+ePWrr1q1qzJgxqqGhYVTGnLLJh1JKPfPMM2r8+PHK7Xara665Ru3atSvZQ0q4t956SwEYMi1dulQp1V/a9sMf/lCVlpYqj8ej5s2bp/bt25fcQSeA1ToBoNavXx9/TjAYVP/2b/+mCgoKVFZWlvrWt76lWltbkzfoBLnnnnvUhAkTlNvtVmPGjFHz5s2LJx5KZe56Sbf9xR133KHKy8uV2+1WY8eOVXfccYfav39/PJ6q22kk9kknT55Ud955p8rJyVF5eXnq7rvvVj09PSm1LH19fWr+/PlqzJgxyuVyqQkTJqh77713SEKbCssyUvvDf/zjH6qurk75fD5VXFysHnjgARWNRkdlzNqXAyciIiJKiJS85oOIiIgyF5MPIiIiSigmH0RERJRQTD6IiIgooZh8EBERUUIx+SAiIqKEYvJBRERECcXkg4iIiBKKyQcRERElFJMPIiIiSigmH0RERJRQTD6IiIgoof4/N1Cc/V7M0REAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# We can see behind the scenes how PyTorchDeRandomizedSmoothing processes input by passing in the first few CIFAR\n",
    "# images into art_model.ablator.forward along with a start position to retain pixels from the original image.\n",
    "original_image = np.moveaxis(x_train, [1], [3])\n",
    "\n",
    "ablated = art_model.ablator.forward(torch.from_numpy(x_train[0:10]).to(device), column_pos=6)\n",
    "ablated = ablated.cpu().detach().numpy()\n",
    "\n",
    "# Note the shape:\n",
    "# - The ablator adds an extra channel to signify the ablated regions of the input.\n",
    "# - The input is reshaped to be 224 x 224 to match the image shape that the ViT is expecting\n",
    "print(f\"The shape of the ablated image is {ablated.shape}\")\n",
    "\n",
    "ablated_image = ablated[:, 0:3, :, :]\n",
    "\n",
    "# shift the axis to disply\n",
    "ablated_image = np.moveaxis(ablated_image, [1], [3])\n",
    "\n",
    "# plot the figure: Note the axis scale!\n",
    "f, axarr = plt.subplots(1,2)\n",
    "axarr[0].imshow(original_image[0])\n",
    "axarr[1].imshow(ablated_image[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7253ce1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can now train the model. This can take some time depending on hardware.\n",
    "from torchvision import transforms\n",
    "\n",
    "scheduler = torch.optim.lr_scheduler.MultiStepLR(art_model.optimizer, milestones=[10, 20], gamma=0.1)\n",
    "art_model.fit(x_train, y_train, \n",
    "              nb_epochs=30, \n",
    "              update_batchnorm=True, \n",
    "              scheduler=scheduler,\n",
    "              transform=transforms.Compose([transforms.RandomHorizontalFlip()]))\n",
    "torch.save(art_model.model.state_dict(), 'trained.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "046b8168",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Normal Acc 0.902 Cert Acc 0.703: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 79/79 [02:06<00:00,  1.61s/it]\n"
     ]
    }
   ],
   "source": [
    "# Perform certification\n",
    "art_model.model.load_state_dict(torch.load('trained.pt'))\n",
    "acc, cert_acc = art_model.eval_and_certify(x_test, y_test, size_to_certify=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a2683f52",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Running algorithm: salman2021\n",
      "INFO:timm.models._builder:Loading pretrained weights from Hugging Face hub (timm/vit_small_patch16_224.augreg_in21k_ft_in1k)\n",
      "INFO:timm.models._hub:[timm/vit_small_patch16_224.augreg_in21k_ft_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.\n",
      "INFO:art.estimators.classification.pytorch:Inferred 9 hidden layers on PyTorch classifier.\n",
      "INFO:root:Running algorithm: salman2021\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of the ablated image is (10, 4, 224, 224)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:timm.models._builder:Loading pretrained weights from Hugging Face hub (timm/vit_small_patch16_224.augreg_in21k_ft_in1k)\n",
      "INFO:timm.models._hub:[timm/vit_small_patch16_224.augreg_in21k_ft_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.\n",
      "INFO:art.estimators.classification.pytorch:Inferred 9 hidden layers on PyTorch classifier.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of the ablated image is (10, 4, 224, 224)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnYAAACbCAYAAADvEdaMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA71klEQVR4nO29e5Bc9XXvu/aj39OPefWMRho9kYSExMNCjwEfh8S6yJjEJpbrxLdcAae4pkxG3MJKOYlSjl2hUlEdn5wyZUeGuqkEkrqhcHFywY6MyeFKtmSMACODQW+EHjN6zGhGMz09/e7e+3f+6NFev9UI2xLz6On5fqpUWr336r1/e8/u1b/+rZehlFIEAAAAAABmPeZMDwAAAAAAAEwOmNgBAAAAADQImNgBAAAAADQImNgBAAAAADQImNgBAAAAADQImNgBAAAAADQImNgBAAAAADQImNgBAAAAADQImNgBAAAAADQImNgBAAAAADQImNiBhmLXrl20ePFiCgaDtHHjRnrjjTdmekgAADAlwN6BqzFlEzs8cGC6+f73v0/bt2+nb37zm/TLX/6SbrnlFtqyZQtdunRppocGGhzYOzDdwN6BD8NQSqnJPuj3v/99uv/+++nJJ5+kjRs30uOPP07PPfccHT9+nJLJ5K99r+u6dOHCBYpGo2QYxmQPDUwiSikaHx+nrq4uMs2ZX/zduHEjrV+/nv7hH/6BiKrPUnd3Nz3yyCP0l3/5l7/2vXjuZg/19tzB3s0N6u25g72bG1zXc6emgA0bNqje3l7vteM4qqurS+3cufM3vre/v18REf7Non/9/f1T8RhdE8ViUVmWpZ5//nmx/f7771ef+cxnPqBfKBTU2NiY9+/IkSMzfh/xb/Y9d0rB3s21f/Xw3MHezb1/1/Lc2TTJlEolOnjwIO3YscPbZpombd68mQ4cOPAB/WKxSMVi0XutJhYQ123YSLZt09jYqNAPmK4nN/uVJy9oDgu9thZ+3RqPeLLf9Ak9KxDSXlieOJoaE3rlCp8rEY/ztTlleT0lvpZCgeVgKCD0HHI8OZ/Pin2xeJRfKNYrleS5LOI/n6WNvSnSJPQiYb4Xti/I4yuWhJ4ytF8DJh+7VJJ6FWV47//r7/wbRaNRmmmGh4fJcRzq6OgQ2zs6OujYsWMf0N+5cyf9zd/8zXQND0wB9fDcTZa9+2//9AwFw2G6cOJtoT989rgnOw5/JpMLVgi9BUtWenKiY4EnB0PSxJ88+ron95065MmVjLRBlnauaCLmyXZA2tl1m+7w5KU38JgKaWm3jx55x5NdV9qTcqXgyceOHvHk8bHLQk+3rZWyZqtH8kIvk+PjVRw+V1tbs9BLNPP3gqsy/J6KUKNCvvo3Kpcr9PJ/7q+L5w72bu5xLc/dpE/sJuuBs22bbNsWExYiIsvkZWPb4smW3yf1Aj6+tKCfJ3N+S07s7ID22uL35P1SzzT5XEHtPaYj1MggnniSyzuDNcdztPBG15F/Bv34pFjPJCX0LGI9/T6FAvJcoaDfk30+lmtX4D9sYmfV6F2Z2PFxZt9S/o4dO2j79u3e63Q6Td3d3TM4InCt1MNzN1n2LhgOUygcoUAwKLb7/fx51Sd2tXoh7cdbWPthVzuxC4b4h2wgwD82zdofjfq5ND07KH+ghiM8OWrSvnhsVx4vHObzuq601aUy/x0DAb7eYo3NVJptNYiPYdvyXLatXbPBNtjnk/fCrx3f0dY4ah8rpyLtbj08d9cK7N3s51qeu0mf2F0rH/bAHTt2lAzTpNTwsNBv0eyZ0cov2hw5mzVCHNuSdUc8OePID6ky2JDkCvzrLpcvCr2yw0ZlWJvpBG15vEqF9Szz6saxei7+hVyp+QVrFFo92dRsYLkoxxSy+foz2urbiCN/cobDbHwNbcXSqJnkkua/zxXYWFbKNUbfrl5LsVzz03YGaWtrI8uyaHBwUGwfHBykzs7OD+gHAoEP/E0AmGo+zN6Np0apXCxSa6JF6Kt2njAqm1fO5i1cKvQcbSJlujlPdnPyM1oY5VUwleeVrfltMhZwYfcNntx9wyJP7pq/QOglkzw+n48/T5WEXNnrXsCfwUpF2rtCgVfcUqO8cjY8PCL0bL9u/NkwNrfKz3Ewwscb01YOA0H5decqvjc+m4+RHksJvVKxauMrsHdgljDpEaDX88DFYjHxD4Brxe/307p162jPnj3eNtd1ac+ePdTT0zODIwONDOwdmAlg78CvY9IndnjgwEyxfft2+sd//Ef6l3/5Fzp69Cg9/PDDlM1m6U/+5E9memigQYG9AzMF7B34MKbEFbt9+3Z64IEH6Pbbb6cNGzbQ448/jgcOTDl/9Ed/RENDQ/SNb3yDBgYG6NZbb6WXXnrpA/FPAEwmsHdgJoC9Ax/GlEzsJuOBC9oGmaZBVBMWsEiLq1vcwdmpyXYZmxLS48q0oMN8sSD0CmUtQ03T82tBxkREpAXQKpffE2+RsSSVsp7QwcdwapIsLD9fWLEkx1Su8DjCmp4dkWMKavsqBsfsmcoVehXi4+mJEE0ROfZMlmNzyhUtZqcmZnM8Xc0YLpVrLqoO2LZtG23btm2mhwHmEJPyBVsuE9llKhVlPGsux/Foi1fM9+RMVmaxlspsQ1ra2C7aPumUWb6cM1fv2HS7J8/vkLFz8Xg7D83mz3m4JnlCDzE2tHTSfDYj9IpanG44JO1Oc4Lj+5YtXe3JR48eF3pk8DGKRbZV8ZjMdtXyw2gszS5yRTK2z3V58KOjfD/zORnLfKXSa8Wpnxi7K8DegasxZckTeOAAAHMF2DsAQL0w8+WzAQAAAADApDDj5U4+jKDhkGm4FI3KIa6Yz8vurSFOefe50p2ZGeFld8fl+Wu+Jv3f1JbtYwmu/2T7pcshNTbO+7QhtUSlW2E8zUv6Ja2kSb4gXSxKc482abWgiIjKJU7XN7V6Ur6adHVHK45saz7WYo07x6/5JkyXr7+YkUVESSsFE9DKrFRc6dody1ZdFaWK3A4AuD4qhQJVDIOMigxvCPg5/GJMK/3U2ildpwtv4vIkye4uT9brVlZPxLZBFAa+KIsB504NsZ7JtvT4u78SeutXsev0ExvWe7Kq6VSZTnPB976zF8Q+v1Y03e/nLOG29vlCr6//PdYLst3N1BR4T6f5Ptk+touxmLTV+Ty7c3Uva6XGrnm19eQlAVC3YMUOAAAAAKBBwMQOAAAAAKBBqFtXbCJgkWWaFKpxP8a1zND2mNYSxpUuDP2VZWt+RVPOZYtaxXa9FY1dk1nqFNk9qiw+xqVLKamnZYqO53ipP+fIjKymkFaYtCjHbmmtc0yD1/+tgGwjlM+yKyXs0/o51rhBClpHjbxWPd2t8S2kMny8VI7vS6a2en25ev0VB65YACaDYj5HhnKpKSQ/47EWzk792C23enL30uVCb1zLSD1+qt+T05oNIiLKpFKefDnF7teLAzIsI6ZlxZLJWaK7v//vQs/3X9kW/k7Px3m7T4aDdHaye5iU7CaUGuUwl1++xT1lbZ+0/ZEo27iKFjZSyqSEnmaeqV2rluDU2ODLIzwOk7R+2rb8WkwkqlnG5ZoOPADUK1ixAwAAAABoEDCxAwAAAABoEDCxAwAAAABoEOo2xq4tHiTbMinqs8T2YJBfmxbHWYRqOkWUtbIBrlZaRCkZZ1HSOko4JY6hcFVNeRItPkPZXEJgvCRT7R2Hx5fTYtBq49HGs3z88yPyGD6TdWMZHnt5QMam5Mc4fmZhm1buIClLIRhRLjVQHOW4mkxGnndsnGPshsc4pvBM/5jQc6zqY+Mq5P8DMBkEAjYFAj4qW1GxPR/iEkyn0/yZfPuVN4TeyGXu9HD+Andb8FmybYxuW4oVtml6HC4R0bx2/mq4NHDWk2MBWT5lPJX25BOnT/P757XJ8/r4ePO6O8W+Lu113wDHBx5/t1/oJedx3N+ZPs0WlqVtdUv82tG6ZgRrSlgFbI7RzhdYLxaLCT3brr5PuVgHAbMDPKkAAAAAAA0CJnYAAAAAAA1C3bpiO9vC5LctivllqY2mMLsCDOEulW5BQytXUtQqjJskXROtUW6YHYlwqYH0mHR7xrXl+XGti8TZ81IvU2RXrF/zEMwPy1tt+zRX5+WU2FdUWkcNrdxJPCbdNHes5ibe6YvsSlA5eS/ibexyKOZ4HJmMnNcHfKzX3cnnSiZlM/PBdNVlW3Fc6jt0jgAAH41QKEmhUJgupaS9O9nP7sgjhw95sumT9sTRus3kxznEwjKlmzJfZNdpapzl8WxG6J05d9STIyG2BSuXrZQD19y5P//ZTz150ZIlQm3FyhWe3NoaF/sCQb6WeIzdpWZFhoBki3oHIS7Bkk+NCz3H4ZCSYIhtWiYt9WJa+ZSAFuJTKskwnNxEyZhyWf5tAKhXsGIHAAAAANAgYGIHAAAAANAg1K0rtrkpRAGfRXYpJbYHNBdEOMDVwot5uXxe1prdJxLNnlzbnLrk8Ny2XNY6OTQ1Cb0LQ7z0//5ZdhEMjcvleb1Jw6IQL+/f919uFXoL5vHx/+fBU2LfgZMDnlxx2dVhm3Ls4ylu1J3L8PiiUZ/QI4fdz8Eg7/MHZcZx2OB9Fa0r9kKtqTgRUXSk6tIolR3aD1csAB+ZRHMrhcIROtl/Qmy/eIYzTcM+/oyPZWWniEz6kicbLrtfU+PSxZrKs42zA/x5b+tICr2QFqIyf/EtntxdYzNO/+qAJ1sG26qyI7vpDA1zNv7atavEvhuWL+Xja5mvTZtuE3rvHOvz5GKBw2aKvpqsWGIXq6vYjg0MXBB6fq2rUbxZv35ZLSCfr4bNwBULZgtYsQMAAAAAaBAwsQMAAAAAaBAwsQMAAAAAaBDqNsauvbmFgn6b8iMFsd00tHIdOS3FvyTjH2xD6wBR5niP2plsvsxxIYlmjs0oOTKe7dQ5js8YSWulRWxZid2y+AyxIOslbZlqHxzheJnlMVmJ/WILH2MwxbEzxZysDv/WCY7HMSscZ1KOyMrpFNfKlZhaaYF4WKhFXb7mgpbyr0ppobe4PTKhg5gTACaD06cPUiAYpGPvnxTbL1x835MdrYxJNB4ReiuXL/bkNavWePLFobzQOzvEx2jvZLuwaJksTxJt5ZizwVF+jxo+LfT6znLc21CK4+hWrRZq9H+s4Li6bEaOydXC8VSJbdzh1w4IveUrb/XkjvkJT37tjf1Cb2CQ7ZUeF1fIS/s5Oso2OdTEx3OVjNnL5qrXX6nIuEEA6hWs2AEAAAAANAiY2AEAAAAANAh164pNtLZRKOCj5qaQ2G6anKKfSnPKf7mmcrqppdu7xEvrqqZie1MTp82XieWjp2TZgWyR3RHBIKfJB/3yeKEIuzebLXYDHDw5KPQqJX5fMS5dse3NPA5DS90vV6RbOldil0ZW6zZRqkgXqaG5m/XGGz5TduFQptbxwubxVYpFqTfhplY17moAwPXxi5//hGyfTXaH7OywbNVaTw5pze1XrV4u9FauWODJToE/x8qUbs8scacc28d2xrISQq9cYRuXHR/x5HhN+EVFswF9l9geB5vOC714jEtOLV22WOxT2vpCPsVdgo69/rbUy/P1r9nyKU9ee/NSoZd/k12x758848nhsCxhFU+0aq/4+yKdlqVkisXqmOCKBbMFrNgBAAAAADQImNgBAAAAADQIdeuKJdMmMn1k+HwfqhLQuiiESWaJ2dqc1TS17hIkM54CIa6wPjzAWVK5Ybkcv7RFq3SueUSDEZlZunLZfD6vplix5HXoy/22JZtdR/18La3Nyzx52fKFQu903y88+dgJdn347RrXqWI3daXCf3KzJqPX5+cxulr1epeky9YwTPE/AOCjMXT+MlmWRbfdcq/YHghwJ4YWrenDvC6Z+T6SYtvVf5JdpyU3IPRMg92Jls2fcUdJm0GanXCK7M5VjrSfTfE2T76c4XAV0y/tsSs6/tSEcGiHbArydS3u6hZqQYvfZxLbtLVrZEZvIpHw5B/m/5cnD1yUNn1+kjvqOAbbal9NuE46XXXtVjNsZYgOAPUIvpkBAAAAABoETOwAAAAAABoETOwAAAAAABqEuo2xKxQqRMogo5yv2cPp9tksp7WXynKOWjE5Ji6T4/iTdE52gJjfzbdAVXjfojYZV7asi+PPcgXeN3/FLULPrzhWY3SMuzeERGo9EV3mgJnuznliVyrLsSpLb+SyBrFmGc8Xa+Zq7qNDPPbRMRmz59PiXUzFMTdlV6bva2F15GgV22uqopCaiJdRCuVOAJgMQpFmsm2bfDUfqZTWeSbQkvDkXEXGuhW0uN9Qc5Tf49Z8eAt61xxtczkn1IIhLRbX4HJJrllTLqqV49T8imP7rFCz0FN+tneuIc9lOJp9svj4voiMAQ418etKke3d5fOylFRrhOMSP/vpLZ785q/OCL2M1omiUBzy5GJefuckogkiIipp3XgAqGewYgcAAAAA0CBgYgcAAAAA0CDUrSvWMRxyDJOUIyud6+6/UJC7UjRFpZvygtb8+vQ5Xma3a3wd/sELnlwYZL3lSVme5JN3sUv0/fPscojObxd6ba3cReLSELsIEgmZ/m+6fHy/1vGh+j4uXWIHU548lLoo9M5f5JR/n4+vPxGTbpp8nq9Z2TyXN2p8rK7mmjUNQ9OT8380nABgcunsXkQ+n/8Dn7VCgcNNBtNsrv2JNqFXrrCbUi8Rlc/Ijjxlxce3bQ7LqFiyLEo4xmVHkq0pT1Yj0k1Z0kI2DJePHQrVdAzSTJyrpE13tC5Bpk/rmmHJe5HJsvvV0OJGAjX3LK3Z3VC4xZM/0XOz0Dv+/llPPnRkgM+Tzgo9/0SHjnJZjhuAegUrdgAAAAAADQImdgAAAAAADULdumLj8QiFgn6q2HL5O5Ph9C9V5iX8sXGZCXq2b1B7D7sjQkE5l714ml0dHUF2Z8yfv0joJbq4urlvXHN1BqXLdsEtG3jXALtUQ5UhoecQX0c2WxD75oXZvVvSKr0bEdnEekGEM9KiCXYBj18eEHqXBi97ctng8RZKNdXmTfaxRgKcVVzKS3fOlQ4VTk1HCgDA9aEMi5RhfcDdlxtn92NAc2+Op0eEXqnAn+Vcmt/jq/mIRiPscm1vZjdlrEWGirQn+FyOzd158gE5vpFFbIOKjhYqUpNl61S0zNqaTF3H1Gyc5opNtMjMWtfhY+pZ+/G4dPv6DbZjqfGUJ6uytGO3rmKbmYjyfdm9+38JvaHBYSIiqlRkFQEA6hWs2AEAAAAANAiY2IEZZ//+/fQHf/AH1NXVRYZh0AsvvCD2K6XoG9/4Bs2bN49CoRBt3ryZ3nvvPaEzMjJCX/ziFykWi1EikaAHH3xQrNQCAEA9AHsHphpM7MCMk81m6ZZbbqFdu3Zddf+3vvUt+s53vkNPPvkkvf766xSJRGjLli1U0KqyfvGLX6TDhw/Tyy+/TLt376b9+/fTQw89NF2XAAAAvxWwd2CqqdsYu8zYCFUKPrJLslOEz9DmoloKvW3JkiG5DMfcNUc5fiQRCQq9/CjH2CW7uDvE/Jt/R+gdOscxIidOsnzHvBahl0rxvo5l3JXCJBlzUtIqnSeULE+SvsQxcSGt2vm8lppzORwX4ruZ41HyNWVRfv7iDz35XD+f1/LL+EDSYua0CilUrpn/m+XqmAqTlP5/zz330D333HPVfUopevzxx+nrX/86ffaznyUion/913+ljo4OeuGFF+gLX/gCHT16lF566SX6xS9+QbfffjsREX33u9+lT3/60/T3f//31NXV9YHjFotFKhY5LimdTn9AB4Bpo1IiMohstyQ2xzVz1R3nz+eNSxNCr0kr/WRpNjKbTgm9Qo7tYijCtmXlcmlbuhct8GTTx/HGmZQ8Xvc87pqz8jR3yYi1SDvb0szlU2xbdpRwNVujNDMejMgSVpWC1g1He4+vtkQM8ee6tY3jkjM5aYOzKY5Fnt/Occ33/cHdQu+FH/3/RDR55U5g78BUgxU7UNecPn2aBgYGaPPmzd62eDxOGzdupAMHDhAR0YEDByiRSHhGjoho8+bNZJomvf7661c97s6dOykej3v/uru7p/ZCAADgNwB7ByYDTOxAXTMwUP1V3dHRIbZ3dHR4+wYGBiiZTIr9tm1TS0uLp1PLjh07aGxszPvX398/BaMHAIDfHtg7MBnUrSvWNIgsg8ipKbWhNHehSbw07hjSFTuq9WtOp7XOC0Xp6pgXZzft+t/9XU9esHKT0Pv/nvpnT+7Uyo5YJVmJ/fyp91lv6WpPDrbeIPQiil3MuZFLYl/IZbdqKc/ug+Fx6UpItHMJltbOxZ6cz8SEnqm9dPwcp1HbeaJc5ntjaKn9hpJp/pVK9bEpz+IWFIFAgAKBwG9WBGAauHPDrRQKhmjp6lvE9gvnuWTS/C52l65YvkzodbbzF72l+HM9rpX7ICIqamVI9M9/U0SWO2lqYleq5Wc3r6/GVZzPcmjHx9awy3bxisVCr+yyQVY16wkVl+24snhMlk9+PZULbG9czS1q2vJ4RlCza9q+Yrks9GyLQ1GcUsqT29tkWamP/5f1RESULxTp+R/+hGYjsHdzC6zYgbqms7Naa2pwcFBsHxwc9PZ1dnbSpUtyclypVGhkZMTTAQCAegf2DkwGmNiBumbJkiXU2dlJe/bs8bal02l6/fXXqaenh4iIenp6KJVK0cGDBz2dvXv3kuu6tHHjxmkfMwAAXA+wd2AyqFtXrKGq/5ya5XO9Sba+Aq/yNXpaomlLK2dXdYZlZtPHbl/hyavuYPfr6CXpAg5UOJts6QLOGHMNmdHameTsKj2LK5eSLoxShfeV8/LP4BC7At4/f86T3z30ptC7YxMfs7WTM3rT4/LXnE9LLmtbzC4XtyabzCmxy7WiuazHhlJCrzhePWCxPDmV2DOZDJ08edJ7ffr0aXr77beppaWFFi5cSI8++ij97d/+LS1fvpyWLFlCf/3Xf01dXV103333ERHRqlWr6FOf+hR9+ctfpieffJLK5TJt27aNvvCFL1w1QwyAeuO2m1ZQJBKhm26Trtj8Gna5RuIcUyGtDpEytBAVzcXYEpErOEr7yOuffteVR6zoGaCaDS4WZejJshsWenLIz7Yln5WdgJSp2ThD2juldYpwFcuOIUNFXC19tpTncTiudCObth6uw1c5flmGspw9zXFmd378Nk/OlWUlhvCEa9dQk9NpB/YOTDV1O7EDc4c333yTfleLb9y+fTsRET3wwAP09NNP05//+Z9TNpulhx56iFKpFH384x+nl156iYJBjgP6t3/7N9q2bRt98pOfJNM0aevWrfSd73xn2q8FAAB+HbB3YKrBxA7MOHfddRcp9eGJGIZh0GOPPUaPPfbYh+q0tLTQM888MxXDAwCASQP2Dkw1iLEDAAAAAGgQ6nbFzq045Fom5Ysy9sOvlRqxbY4lsUwZw3ZDJ5cMCYZ4/rp4kSzMeMvHeUl83sqbPfntA08JvYXdfLzOm9byeNpl2QE7HPfkXIHj9PJpGbcxeIHjO0YHz4l9jlaSIBTl5fe2Ntkpov/CW57cMW++J1dyNSVi8lxx3MiO8nmUjJfRY11CAT6Xv1OeNx2oxpoUSpMTcwLAXCcYiVAoEqGmoCxJEQlrJtrmkk5uzYKPocfYabJb09XGLbvaPj6IURNvW9Gi+PSqSMqQek0JLsFScfg9jivLT5HLB1EkY3NN/QQOy44t7Y4i7aIrWmkmVx4voJ3b5/B4IwU5JjXI9m/oFGehLli5QOgNmxP21Jy95Z3A3AIrdgAAAAAADQImdgAAAAAADULdumJ9lk0+y6bRmm4LToGX6kNhrfF1zTJ5Uitx0n8x5cnLPvYpobdgrf6a3a3l8azQi0fZxdq+4lZPztqyefbht37hycU8HyNd04x7+Hwfj92RbuRgkP8s85ewi/XmFbJ7RcXiNH+flWDZX1NhvcDdJnJnuZK9W6npKKFN8zMWuy3CrbKcQEdXtbRKvjA55U4AmOs0xZop2tREypLux5xWdkhpTdyLNR10shm2NSWtg0yxKG1BpcLu0rJWxkTvOkNElMux3c1lOYykUlMWJdrCdjEaT3hyItom9IJ+vyc7Nd0ryNC6SGjdhKJaGAoR0eVL/L6C1pHI1Tr1EBEZxOdyHb5nsah0cy9ayG278jm+f8qVJbHi0ar981k17mUA6hSs2AEAAAAANAiY2AEAAAAANAiY2AEAAAAANAh1G2NXKhTJdB0KB+QQjaCWym5yLIRyZFxEqIn1PvNHn/HkO+75pNCLtXGcxeCpo55smfJ4qXFukTN05rgnXxiXcWY/feEFT24KcbxMoShLkHR2cGxKLCpj2E6f41IoJW0cLV2Lhd6Ktev4hcPxIyMpWT4lp8Uljub5eIaS97aQ5/iZjFYKQWUKQm9VYkK/JlQGAHB9/OjFlykYDJLj+5nYPjrKZTgyY8OeXFt5Q4+50xvIOzV1UVrak57c3MZtCAOWtAXZkZQnn3iP7WI6I+1Y95JFnmz52N7Foq1Cb8kSbj22oFu2OVuylOOIWwJsq6JBGW/oai3VSIt3K9fYfkvrNWlpx+tYXBP3F2ObWVZsxy2/UKOWlup5AwE5HgDqlWtasdu5cyetX7+eotEoJZNJuu++++j48eNCp1AoUG9vL7W2tlJTUxNt3bpVGBoAAJgNwN4BAGYj1zSx27dvH/X29tJrr71GL7/8MpXLZbr77rspm+WMoq9+9av0H//xH/Tcc8/Rvn376MKFC/S5z31u0gcOAABTCewdAGA2ck2u2Jdeekm8fvrppymZTNLBgwfpE5/4BI2NjdE//dM/0TPPPEO/93u/R0RETz31FK1atYpee+012rRp0299LleVqlXTa6qKG1q6fkVxur5hSJdDMMDL9reuY5dlwCeX04+8zd0bRi+878nFonQ/jo+OeHL/ySOenFEhoedz+H1NWqX4WFC6W9ub2RV7cXBA7KtoZQhy4+z66D/dR5LDPI4MlyQI2vJeVALsfrlc4fsSCslyAuEoX0vIZjfFeC4tjzdRDqDiotwJaFym09795Gevk237KLFgpdiuHP78v/XqTzx50QLZHaGtlV2f58+xPan9jIZbEp5cMtmWDmrhH0REn9zQ48m33nyTJ+dq7KLp46+Q031nPfnEe+8LvXcPsZ1NxJvEvq2f/0NPvvOmFZ7sV3LdYcE87hpU0lyxhik74OgdNcpalwvTrulQkWD7F9I6b7iWjDG58o1h123gEgCSj5Q8MTZWjTtraanWcjt48CCVy2XavHmzp3PjjTfSwoUL6cCBA1c9RrFYpHQ6Lf4BAEC9AXsHAJgNXPfEznVdevTRR+nOO++kNWvWEBHRwMAA+f1+SiQSQrejo4MGBgaucpRqHEs8Hvf+dXd3X1UPAABmCtg7AMBs4boXl3t7e+nQoUP0yiuvfKQB7Nixg7Zv3+69TqfTE8bOJSKX3IpcFrd93FHC0TonlEhmRnXEuRr5f/5wtye3dBwWekl9eT/Hma8+n6xS3hRhF6ZtshsgUuPa7UyySyQ/PurJIUse7/IQZ7iVS9JFEA2yS7SkZaG999abQu/isROeXKxwQ2vyyQrpjj7eBZpLOCLvrRlgN0tQq77eTNLdvOqmJURElMuXiehXBECjM9X27r7P/58UCoUpkFwu9HPjPEF8713+rM3rlBNCU3MlhoJsq0puXuitWMPHb57HIRq5Ntm94ffv4VVIPUQjW+OKdTUvaEWxa7dQkXqXLnEoy9nTF8S+cJjHO3DusiefOfye0DO1DjqnBi558oa7bxd6ixZ3ebKeMWsGa9JdfWx3Db3bhCHtsd+oXpffV5OKDECdcl0Tu23bttHu3btp//79tECL9ejs7KRSqUSpVEr8ih0cHKTOzs6rHIkoEAhQIBC46j4AAJhpYO8AALOJa3LFKqVo27Zt9Pzzz9PevXtpyZIlYv+6devI5/PRnj17vG3Hjx+nvr4+6unpqT0cAADULbB3AIDZyDWt2PX29tIzzzxDP/jBDygajXpxJPF4nEKhEMXjcXrwwQdp+/bt1NLSQrFYjB555BHq6em5pgwxAACYaWDvAACzkWua2D3xxBNERHTXXXeJ7U899RR96UtfIiKib3/722SaJm3dupWKxSJt2bKFvve9713zwFzXINc1yG/LeLGgzXEcpKW5K0uWE3FLXDJkeJjjVDJDMqg5VOasNJf4XC3NsnJ6oqvdkytO0ZPPX5DHU8RxGKbJt7dUqamObnBsXiQYFvu0ii5k6S9qSro4JY4JNLVgl3RuVOiVAhxnE+3isWdDKaE37nLMXSHLi7mtsaVCr20ijjCbResJ0LhMp70L+EwK+E06ceyQ2J4eY/ui9DIeJfnZy2S4tp5hsC0I1nRLKOe4LNLYEB9vsE+WO/nxf/7Yk0fHtfdkxoReNMbxcfHmFk+OxKS7+dw5jqtLts0X+4IxjvX72Y/4vCPvvSP0HM2mnxzgItDnsuNCb/kqjiOMx9i2xrUSU0REoTCXO4lH+D75gvI7JxyuXktJt8UA1DHXNLHTDcuHEQwGadeuXbRr167rHhQAAMw0sHcAgNnIR6pjBwAAAAAA6oe6raVtGgEyDZuCAVlqQ2llTSIhXmaPRGWD51yZU+Nbo5zmbteURSmN8ZK+a7JezieX3Ts6OHDa1dwgK2+WFeBf/QkHUpdUzpN9hqyOns/wvlg0Jvb5tRLnlsHjyBRkCYHTF9nlmkrxdRWNrNBrX8Hz9/kJrZSKkun/o8M8Jn9BcxXPl27pfK5aDiCfR+cJACaD8ZFBquRDtPcHPxLb+wfOebJZ5pCKd96pKWys2ZeKHvZhSDv28u69nuzXSjrdetvHhF7JH/XkdJHtwqm+S0Lv8uWj/J4Cn+vCwBmhd/oM691+2zqx7//u5fIvb7zGhZ0rY5eFXrrIYSR5LeTl1JvSjfyzgxc9OWKz+9bnly5WS8tOjmqu2AWLFgu9z279AhER5XIodwJmB1ixAwAAAABoEDCxAwAAAABoEOrWFeuzDfLbJuW05XciIivI2a+u1s0hV5YV1i2tSnjAz+5Hn09mz/rDnCkVj/G+gaFBoZebzy7XZPcNnnz+0rDQu2n9nZ6cGeJMsFMnZMeLbCblybYlxx6Ps2vWIHZvXDwvK7b3ndWyYgM89liHzLJtb9GOp7lzjRF5L5pH+XGYn+QMtwUJ6W4+eaSaqZcvlAlcHZ9t0v+15SaKhqouHtP0kTnRAcQ0bQpMZEIbBlEoGCBzIsP73/e9Q6+8c2pmBg1mjM5kB4XDEVq+WNbKU9rn3zZZtmpCO0yLf6Mrl22fPyg/4+TjTNCuLs5OvWvLFqEWDWvZpEHuSnHkkOw0c+Lk+3wN8xd7ckHJNQNLC5s5dOKY2HfkBHfQCS9e5ckXLshuGM0Jfp30cxhJuEmG64wMnPXky+dPevLQsLTpBUfLMtaqClxMya/FOz5Z3ZfPy3sOGNM0adVNt3sdmyyb7R0ZBpFVtYMGGZRoTpA1se/wO69T3+ljVz0muH7qdmIHwGzGNAxa1hmj5mj1i9Sy/GSavgnZplA46pWliDaFvJZQ+95+/+oHBACAOsUwDIrFWig40Q7Tsv1k2RNxi4ZZndhN2LtksoPsiTjyM+8fmZHxNjpwxQIAAAAANAhYsQNginCVQe6E96y6Ilf9HaUUkVOZcGMbRE7FT8qsuoV0NxoAAMwGTNOkG5Yvp6amaja1ZbErVpFB6krxf4MoHAmTYVRtYSgUvOrxwEejbid2yVaTwkGTypdlynve4TiTrFbVQ5my9IatlQyJxbhch98nK7Hns1w2IOTTbkdJ3po3X33Vk5eu1Kqen5OdJ0ytG0ZYq/puWbISeyjEsS/ZjIyxy+f5daXCpVWaQvIYd9y2wpODWsmUiiVLujhlLleQ7+cYO3NcfqiSYS5xcNuKm3h7okPoHbx4moiICiV5HiBx3WoHFSIiUiYZE8ZNuYpKxYJXoaLiD5BlXdmH6vZzkdHhUSqEirRp4x1i+x2/8zueHAhwuQ7bks6WK658IiJXabF4JEt8lEtsJ/MltguXz50WeiNa/OzI8IgnnzopQwUuXGL715Ts4h0BaVsMP8fYlSoybvrlfa948qJlaz25u6WmQ4XWySeslWopFmTniVNpjmdu0uyio6S9GhjNeHJb22JPzpXlZ3DvvjeIiKhcRqedD8O0LFq/YSO1tFbLjhlkkDnxQ9Z1FTkTXTuUUlQsF7wOTbFo08wMuMGBKxYAAAAAoEGo2xU7AGY7SilyJ1bgXOWQ61ZXDAyj6o5VisigakFZTw8rdgCA2YYiqlTKXg9jkwwyVNUloZS2YkeKlHKqbyAigxB6MhXU7cRuwQI/NYV8FDfkkv7JfnYfDGpNrEuOdFM2NfGlZXNcFsRxM0LP0hYtR4bY7Tuekcv2hTIfw1IsR5tkSv7gALstzmXZ7ekqmSrf0c7uYcOVZUNGU9xRIhDh60rEo0LPr7ljipqLhWzpbs4WWa+U0TpKuHLB9obuTk/u6uTx9Z+TZQIuD1X/BsUyOk98GEopKhQylLeqhq7iZMma+HsZRJ6bgohobDRFauL5GB8f/8CxQOMTDgcoHArQ5bTsLvPWOwc9OZlkW9ORlJ12ymW2IaOjKd5R063G1mzN/CXsOu1ulrbl/Anu3pDNsOs02dEp9MKtCU+2guz2zOXleefNW+jJAxfOiX3Dl9mezuvi+BqjpldvpqjZSZvtYtmVdiighbkEtLIwpctDQo9MtoUdWqmWUlG6XK8M47doHTxnUcql/jOnKHUldKpSIXKqfxdDKbLUlR+1BrV3tZFlV0MEDKdw1eOBj0bdTuwAmO3o3wPVhvLK267IJYMMUqoagyL1AABgduG6iu1XNcC4KiuXXOWSQUTKICKlCBUBpxbE2AEAAAAANAh1u2IXS/ioKeyj/FBObG9OalleEc60Gh6UmVaFEi+n2352EZRqEptczZ1YdvgYY/lRoRfRMlILOV4+zhdk54mSdjxHk5WS2WmZNF9XLCYrp8di3A0jn2e94ctyTE1N7HIwtKw4oyJXffw2H19PVvPXNMVefMNiPq/W8Hr/fllE8p0T1UbgFQfxYL8O13HIueKOMEz+NauIyL0Sf0JUcSre8p7jwr09FwnYLgV8LhULKbH91Vf3eLIqs92JhaXNKJc5dKSgZdXbNb/dFy3u9uQ1m1Z78rKFXUIv1c/u0oFRtnH+msz8Za3smh0a4jCXtSvXCL2b1q705Gf/338V+2ziLhJlLXylVJJuOlXRPhtBvl4rIMe0eMlST77Uf5x3mNLehbQwl1WruMJAISfDdbrnJYmIqFiE2/DDcCoOvfnqfgpMfMEYStFEBSfy+33UnKh+B1uWRd3L51N44rvbF/Bd9Xjgo1G3EzsAZjtKsWvCdZXXaUK5RGqinZFSihzdFUuYLAMAZhdKKRoeGiTfRHy3SQaZEw7XQDBAllWNr7Msi0zb9CZ0epkeMHngroIZZefOnbR+/XqKRqOUTCbpvvvuo+PHjwudQqFAvb291NraSk1NTbR161YaHJQJHX19fXTvvfdSOBymZDJJX/va16hSQZ09AED9AHsHpgOs2IEZZd++fdTb20vr16+nSqVCf/VXf0V33303HTlyhCKRqqv5q1/9Kv3oRz+i5557juLxOG3bto0+97nP0c9//nMiInIch+69917q7OykV199lS5evEj3338/+Xw++ru/+7sZuS7XVfSrsykKT7i7Tcsk02D3q1ZDthp0POGLvTQGdw8AjUqj2jtFisbGx7xC6yYZXoKEnfdRya2GOZmmSa/8/FUKBqsu23Pnz8/EcBseQ9VZGl46naZ4PE5v/c//StGwjy6feVnsH9NKdKTzPC9NXZYurPSoNmd12j0xEpTp+o7eyaKY8uTxnOx4Edbi1JrCHANXVPK8Oa2LRLnI+wwlF0cjAc4LamqSJV1srVxJ2eEU/4uDNV0utPiEeILjCG2/X+ppVdqHszy+8bQsrfGpzbfzPq2tx3//Hz8UeoMTYX+uq+jsaIHGxsYoFovRZDA0NETJZJL27dtHn/jEJ2hsbIza29vpmWeeoc9//vNERHTs2DFatWoVHThwgDZt2kQ//vGP6fd///fpwoUL1NFR7ZLx5JNP0l/8xV/Q0NAQ+Wvux9W48tyB2cNkPnczxZXn7v/5zhMUDoVoMCUn9heGOb7NLfFn0irL1RlXs2PK4lgyy5bPflCLS563ZJ4nR0jGdo5oJY4OnePY3ldfe0XoXR7iEiJLl3Ac3fo7ZAeNiGbjfvwfPxD7VJm/gjq1siOmJdcdXIev2a91CbL9Mk5r5UqOsTtz7G0+jyM7/Lxx8C1PvvljGz05r7c0IqKuZPX7o1Qu0bP//izsHZgRruW5gysW1BVjY9WaVi0tLUREdPDgQSqXy7R582ZP58Ybb6SFCxfSgQMHiIjowIEDtHbtWs/IERFt2bKF0uk0HT58mK5GsVikdDot/gEAwHQCewemAkzsQN3gui49+uijdOedd9KaNdWsuoGBAfL7/ZRIJIRuR0cHDQwMeDq6kbuy/8q+q7Fz506Kx+Pev+7u7qvqAQDAVAB7B6aKuo2xy2ZsMlwfkSWbBDdF2FXhC/ESfqSm6XQ8zq6JTDqvyTIINZPTyp0UWI76W4Ve0MfL/ZUil0WxbTk39msvfVrTbsOQemGtM4ZZ81eo6C6HEO+MJcJCb2SEXanjmks41iLHnqtwjZf3zrCL+di7/UKvo4WXeTsWaOcypbu5baIDhuO6dHZ08mLCent76dChQ/TKK6/8ZuWPyI4dO2j79u3e63Q6DWMHZoxIxEfhsJ/iNYEx0XYuw1HU7E6w5je532D3mwpp5Y3C0i3nFriUx/g4r9pYYeniSS5LePKyMLuD3zv9vhygwTbOF+aQj/MX+4Raa1vzVWUiolKeXZ/FInehyGalbSlqZUjKRS4DZQelXezo4tCbsxfZ3g/2ybEXMnyu9w+/zeNrbRd6qrm6mqbKk5uxDnsHpoq6ndiBucW2bdto9+7dtH//flqwYIG3vbOzk0qlEqVSKfErdnBwkDo7Oz2dN954QxzvShbZFZ1aAoEABWrqXwEAwHQAewemErhiwYyilKJt27bR888/T3v37qUlS5aI/evWrSOfz0d79nCh1uPHj1NfXx/19PQQEVFPTw+9++67dOnSJU/n5ZdfplgsRqtXryYAAKgHYO/AdFC3K3YX+onCQaJiSrpYo+3spgyGOGM0Lj221NLCl5bJ8rJ9KiU7WYxe9msyb7dcWaXc1ZKHr3QTqO6Q2WT6TNkwOfPVsuWtzjusqWrKD/m0Rt2V3AifNy/H7mjZs6kM7yvVNC8Y0VzRZ07yRaYuy+yvUpbf2BnnX36rFs0XelcOV3Zc+uWZEfoo9Pb20jPPPEM/+MEPKBqNejEi8XicQqEQxeNxevDBB2n79u3U0tJCsViMHnnkEerp6aFNmzYREdHdd99Nq1evpj/+4z+mb33rWzQwMEBf//rXqbe3F79SwawglzlJ5ASJXPlb22ewYRscZNfhe0fOCL2glrXvjyc8uS0p3Z5dbZwJaWvFYVvjMnxDbypT0LrwJJPSZTu/q8WTL2rxXSdOHBV6i0s8gdFdykRE4+N8Xbkcu07TYzLAX3fFOiW2aVYgIvQOH2rz5FKRw1CSSRmXNv9m7o6RbOd9be1y1Ss4cfzCJHSegL0D00HdTuzA3OCJJ54gIqK77rpLbH/qqafoS1/6EhERffvb3ybTNGnr1q1ULBZpy5Yt9L3vfc/TtSyLdu/eTQ8//DD19PRQJBKhBx54gB577LHpugwAAPiNwN6B6QATOzCj/DZlFIPBIO3atYt27dr1oTqLFi2iF198cTKHBgAAkwrsHZgOEGMHAAAAANAg1O2KneNrJccXoLL/drG96HJ8hlnhNPxg3BB6iXaOzWs2OYitJSdT1lMjHJuSGua4unxW3hqnopUN0LpIuBV5vEKe4zD0CuCWLWP2xgv8vnxGxm74FMeFRM0on8uUMSflMo8xEOFfgkGfjLNI+Pl4SynhyWtvkbEpK2++xZMX33CDJ2/YJGP7zl2oxroUSxWiX54hAMBHQ5WK5FpEZs1vbbvMdiPmY5tx8LV9Qm9gkG2hoX3+N2xYJ/Q+3sP29EpxXCKid375utDLFtgmnejjskinzpwRevkc2wal2AYHY7JkSFrrcjM+Oiz2ZdMcw6dbcduSNj0e5bImXVrSQXPrPKGX7OIYua7b1npyS0zaO7/eoUOT9RIuROTZe70jEAD1DFbsAAAAAAAahLpbsbsSg5ArVFeZ8oWS2G/4OGPUdXklzszJX3d2lvXI5GzPbF6usGXzrJfTV9EKMhbCFZmrv2bFrsjHc7RfsJYjU1XzRT5+oVQW+5Ti17a22lgoyfTZov7S4ONZSv7iLGp9JUsVHoevpt9kTrvXGa04aL4ox1ecGMeV49ZZu+HrohGuYa7RCH+zK9eQL1Q9EeWa39oV7bNcKLC3wnGl3dGz9g2tWHm5Ij/jBS0jtahljBZL0s6WNJtU0Y7h1pxXaa/1FTu3plqAq/WiVbXH+JC/Y+1m/dx6ZYJKzTWWy9p1addbKNZUOjCvbcXuSlZsIz13YPZwLX8zQ9XZX/jcuXOoiD3L6O/vF0U2ZyOnTp2iZcuWzfQwwDXQCM8d7N3sA88dmAmu5bmru4md67p04cIFUkrRwoULqb+/n2Kx2G9+YwNzpf1Lvd0LpRSNj49TV1cXmebs9uqnUilqbm6mvr4+isfjv/kNYFK5lme8kZ472LsPAns39biuS8ePH6fVq1fX3X2eC0y1vas7V6xpmrRgwQJKp6uJArFYDA/dBPV4LxplEnTlAxOPx+vuHs8lfttnvJGeO9i7q1OP96KRnrv586uF5+vxPs8Vpsreze6fHQAAAAAAwAMTOwAAAACABqFuJ3aBQIC++c1vovcd4V5MB7jHM8tcv/9z/fp1cC+mB9znmWOq733dJU8AAAAAAIDro25X7AAAAAAAwLWBiR0AAAAAQIOAiR0AAAAAQIOAiR0AAAAAQIOAiR0AAAAAQINQlxO7Xbt20eLFiykYDNLGjRvpjTfemOkhTTk7d+6k9evXUzQapWQySffddx8dP35c6BQKBert7aXW1lZqamqirVu30uDg4AyNuLGYi8/cVDNZz3RfXx/de++9FA6HKZlM0te+9rUPNH6fzczFZw/2bmaZi8/cVFNX9k7VGc8++6zy+/3qn//5n9Xhw4fVl7/8ZZVIJNTg4OBMD21K2bJli3rqqafUoUOH1Ntvv60+/elPq4ULF6pMJuPpfOUrX1Hd3d1qz5496s0331SbNm1Sd9xxxwyOujGYq8/cVDMZz3SlUlFr1qxRmzdvVm+99ZZ68cUXVVtbm9qxY8dMXNKkM1efPdi7mWOuPnNTTT3Zu7qb2G3YsEH19vZ6rx3HUV1dXWrnzp0zOKrp59KlS4qI1L59+5RSSqVSKeXz+dRzzz3n6Rw9elQRkTpw4MBMDbMhwDM3PVzPM/3iiy8q0zTVwMCAp/PEE0+oWCymisXi9F7AFIBnrwrs3fSBZ256mEl7V1eu2FKpRAcPHqTNmzd720zTpM2bN9OBAwdmcGTTz9jYGBERtbS0EBHRwYMHqVwui3tz44030sKFC+fcvZlM8MxNH9fzTB84cIDWrl1LHR0dns6WLVsonU7T4cOHp3H0kw+ePQb2bnrAMzd9zKS9q6uJ3fDwMDmOIy6KiKijo4MGBgZmaFTTj+u69Oijj9Kdd95Ja9asISKigYEB8vv9lEgkhO5cuzeTDZ656eF6n+mBgYGr/m2u7JvN4NmrAns3feCZmx5m2t7ZH2HsYIro7e2lQ4cO0SuvvDLTQwFgUsAzDT4MPBug0ZjpZ7quVuza2trIsqwPZIkMDg5SZ2fnDI1qetm2bRvt3r2bfvKTn9CCBQu87Z2dnVQqlSiVSgn9uXRvpgI8c1PPR3mmOzs7r/q3ubJvNoNnD/ZuusEzN/XUg72rq4md3++ndevW0Z49e7xtruvSnj17qKenZwZHNvUopWjbtm30/PPP0969e2nJkiVi/7p168jn84l7c/z4cerr62v4ezOVzOVnbqqZjGe6p6eH3n33Xbp06ZKn8/LLL1MsFqPVq1dPz4VMEXP52YO9mxnm8jM31dSVvZuE5I9J5dlnn1WBQEA9/fTT6siRI+qhhx5SiURCZIk0Ig8//LCKx+Pqpz/9qbp48aL3L5fLeTpf+cpX1MKFC9XevXvVm2++qXp6elRPT88MjroxmKvP3FQzGc/0lfT/u+++W7399tvqpZdeUu3t7Q1V7mQuPnuwdzPHXH3mppp6snd1N7FTSqnvfve7auHChcrv96sNGzao1157baaHNOUQ0VX/PfXUU55OPp9Xf/qnf6qam5tVOBxWf/iHf6guXrw4c4NuIObiMzfVTNYzfebMGXXPPfeoUCik2tra1J/92Z+pcrk8zVczdczFZw/2bmaZi8/cVFNP9s6YGBAAAAAAAJjl1FWMHQAAAAAAuH4wsQMAAAAAaBAwsQMAAAAAaBAwsQMAAAAAaBAwsQMAAAAAaBAwsQMAAAAAaBAwsQMAAAAAaBAwsQMAAAAAaBAwsQMAAAAAaBAwsQMAAAAAaBAwsQMAAAAAaBD+N0XSy/Q4P/1ZAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnYAAACbCAYAAADvEdaMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABId0lEQVR4nO29e5BcV3X/+z2vfk4/5j0azehh2ZJl/ALbksfmR5yga2FIwMHUL9yiAqQoKIjELaMUSVyXQIVKlerHzS0oiMB1U4lN6sYF5fyueTjG+TkykXGQbSzsYMl6WLbk0WtGo5np6enXee77R3fvtXdLspE8o2m11qdqanb3WX3OPqf3Wb3PXi9DCCHAMAzDMAzDXPaYS90BhmEYhmEYZmHgiR3DMAzDMEyHwBM7hmEYhmGYDoEndgzDMAzDMB0CT+wYhmEYhmE6BJ7YMQzDMAzDdAg8sWMYhmEYhukQeGLHMAzDMAzTIfDEjmEYhmEYpkPgiR3DMAzDMEyHwBM7pqPYsWMHVq1ahUQigY0bN+KFF15Y6i4xDMMsCqzvmHOxaBM7HnDMpeaHP/whtm3bhq997Wv49a9/jZtuugmbN2/G6dOnl7prTIfD+o651LC+Y86HIYQQC73TH/7wh/jkJz+JBx98EBs3bsS3vvUtPProozh48CAGBgbe8rNRFOHkyZPIZDIwDGOhu8YsIEIIzM/PY3h4GKa59Iu/GzduxG233Ya/+7u/A1AfS6Ojo/jiF7+Iv/zLv3zLz/K4u3xot3HH+u7KoN3GHeu7K4OLGndiEdiwYYPYsmWLfB2GoRgeHhbbt29/288eO3ZMAOC/y+jv2LFjizGMLgjXdYVlWeKxxx7T3v/kJz8pPvzhD58lX6vVxNzcnPx79dVXl/w68t/lN+6EYH13pf21w7hjfXfl/V3IuLOxwHiehz179uCBBx6Q75mmiU2bNmH37t1nybuuC9d15WvRWEC8ZcNG2LaNublZTT5uRrLdHROyPdKd0uT6euh1by4t2zHT0eSseFJ5YcnmbGFOk/MDOlY+l6NzC339fDw6l1qN2olkXJMLEcp2tVrWtmVzGXohSM7z9GNZoK/PUvrele7S5NIpuha2k6D+uZ4mJwzlacCkfXueLhcIQ37+r779z8hkMlhqzpw5gzAMMTg4qL0/ODiIAwcOnCW/fft2/PVf//Wl6h6zCLTDuFsoffc//uERJFIpnDz0siZ/5s2Dsh2GdE8OjKzV5EZWr5Pt/OCIbCeSuoo/vP952R5/Y69sByVdB1nKsTL5rGzbcV3P3nL7HbJ91dXUp1pR19v7X/2NbEeRrk/8oCbbB/a/Ktvzc9OanKpbA1/R1TNVTa5Uof0FIR2rr69bk8t30+9CJEr0mUATQ61a/458P8BT//ZMW4w71ndXHhcy7hZ8YrdQA862bdi2rU1YAMAyadnYtmiyFXN0ubhDp5aI0WQuZukTOzuuvLboM9WYLmeadKyE8hkz1MRggCaeiGhjomV/oeLeGIX616DuH4LkTAhNzgLJqdcpGdePlUzEZNtxqN26An++iZ3VItec2NF+Lr+l/AceeADbtm2Tr4vFIkZHR5ewR8yF0g7jbqH0XSKVQjKVRjyR0N6Pxeh+VSd2rXJJ5eEtpTzYtU7sEkl6kI3H6WHTbH1oVI+lyNkJ/QE1labJUZfyw2NH+v5SKTpuFOm62vPpe4zH6XzdFp0pFN1qgPZh2/qxbFs5Z4N0sOPo1yKm7D9U1jhah1UY6Hq3HcbdhcL67vLnQsbdgk/sLpTzDbgDB/bDME0UzpzR5HsUfWb00ou+UJ/NGknybSlHM7JdCvWbVBikSCo1erqrVF1Nzg9JqZxRZjoJW99fEJCcZZ5bOdaPRU/IQcsTrFHrlW1T0YG+q/cpadP5l5TVt5lQf+RMpUj5GsqKpdEyyYViv6/USFkGfovSt+vn4votj7ZLSF9fHyzLwuTkpPb+5OQkhoaGzpKPx+NnfScMs9icT9/NF2bhuy568z2avOinCaOwaeVs2YqrNLlQmUiZUUW2o4p+j9ZmaRVMVGlla3mf7gu4YvRq2R69eqVsDy8f0eQGBqh/jkP3U5DXV/ZGR+geDAJd39VqtOJWmKWVszNnZjQ5O6Yqf1KM3b36fZxI0/7mlJXDeEL/uYsEXRvHpn0U5wqanOfWdXzA+o65TFhwD9CLGXDZbFb7Y5gLJRaL4ZZbbsHOnTvle1EUYefOnRgbG1vCnjGdDOs7Zilgfce8FQs+seMBxywV27Ztw9///d/j+9//Pvbv348vfOELKJfL+JM/+ZOl7hrTobC+Y5YK1nfM+VgUU+y2bdvwqU99Crfeeis2bNiAb33rWzzgmEXnj/7ojzA1NYWvfvWrmJiYwM0334wnn3zyLP8nhllIWN8xSwHrO+Z8LMrEbiEGXMI2YJoG0OIWsFLxq1s1SNGpA/26b0pS9StTnA6rbk2Tq/lKhJoiF1OcjAEAigOtiOgzuR7dlyTw1YAO2kfYEmRhxejEXE/vkx9QP1KKnJ3W+5RQtgUG+eyZItLkAtD+1ECIrrTe91KZfHP8QPHZafHZnC/WI4Y9v+Wk2oCtW7di69atS90N5gpiQX5gfR+wfXiu7s9aqZA/2qq1y2W7VNajWD2fdEhPH+lF29GNMtdcQ5Grd9x+q2wvH9R953K5fuqaTfd5qiV4QnUxNpRw0mq5pMm5ip9uKqnrne48+fetueo62d6//6AmB4P24bqkq3JZPdpViQ/DXJFM5AK6b18UUednZ+l6Viu6L3Mz02sQto+PXRPWd8y5WLTgCR5wDMNcKbC+YximXVj69NkMwzAMwzDMgrDk6U7OR8IIYRoRMhm9i2uX07J7b5JC3p1IN2eWZmjZPYxo/lptCf83lWX7bJ7yP9kx3eRQmJunbUqXejK6WWG+SEv6npLSpFrTTSxCMY92KbmgAMD3KFzfVPJJOS3h6qGSHNlWbKxuizknptgmzIjO3y3pSUShpIKJK2lWgkg37c6V66YKL9DfZxjm4ghqNQSGASPQ3RviMXK/mFNSP/UO6abTFe+i9CQDo8OyreatrB+IdIOWGPiUngy48sYUyZmkSw++8l+a3G3ryXT6vg23ybZoqVRZLFLC9/E3T2rbYkrS9FiMooT7+pdrcuPHXiO5BOndUkuC92KRrpPtkF7MZnVdXa2SOVe1sgYtek3m1tNPiWHaFl6xYxiGYRiG6RB4YscwDMMwDNMhtK0pNh+3YJkmki3mx5wSGdqfVUrCRLoJQ31l2Ypd0dTnsq6SsV0tRWO3RJaGLplHhUX7OH26oMspkaLzFVrqr4R6RFZXUklM6up9t5TSOaZB6/9WXC8jVC2TKSXlKPUcW8wgNaWiRlXJnh612BYKJdpfoULXpdSavd6vn38QsimWYRYCt1qBISJ0JfV7PNtD0anvuelm2R696hpNbl6JSD34xjHZLio6CABKhYJsTxfI/HpqQnfLyCpRsTApSvTxH/5PTc7576QLf2fsvfS+o7uDDA2ReRhCryZUmCU3l1+/RDVlbUfX/ekM6bhAcRvxSgVNTlHP6FeyJYQtOnh6hvphQqmnbes/i/l8PcrYb6nAwzDtCq/YMQzDMAzDdAg8sWMYhmEYhukQeGLHMAzDMAzTIbStj11fLgHbMpFxLO39RIJemxb5WSRbKkX4StqASEktIoTuZ+EpFSVCj3woItGSnkTxzxA2pRCY9/RQ+zCk/lUUH7RWf7T5Mu3/xIy+D8ck2WyJ+u5P6L4p1Tnyn1nRp6Q7GNBTIRgZSjXgzpJfTamkH3dunnzszsyRT+HRY3OaXGjVh00kOP6fYRaCeNxGPO7AtzLa+9UkpWA6UqR78uVnX9DkZqap0sOJk1RtwbH0sjGqbnED0mmqHy4ALOunn4bTE2/Kdjaup0+ZLxRl+9CRI/T5ZX36cR3a37LRIW3bsPJ6fIL8Aw++ckyTG1hGfn9HxxVd6Ou6NfLodahUzUi0pLCK2+SjXa2RXDab1eRsu/45EfE6CHN5wCOVYRiGYRimQ2jbFTuGWQoss15b2GjdoNQRNrRo4lZJpabwWyxoqvWLz6LxwbdaDxUCEG8lIZrH0fuoHlVdcW3dk9Y7pa+mYcAw6HlQTUTbcij1Y5pca/Jacb7LKc7ZhBACES8WM8w7JhaLwzAM7R4UEC336Pm0gzjvParTer+f+/4/S1e1qEhD29Yi13htaNsa9ebP8ZlWQqWYexSFLTqYzle7TmcpeEUXvpV+Pw9n7c6g40QXmIGibSd2Q30pxGwL2ZieaqMrRaYAQzOX6lfFUNKVuEqGcbPlh7g3QwWz02lKNVCc082eOWV5fl6pIvHmCV2u5JIpNqZ8F8tT+qW2HcXUOV3QtrlCqaihpDvJZXUzzR3XURHv4ikamKKiX4tcH5kc3Ar1o1TSF2zjDsmNDtGxBgb0YuaTxbrJNggjjO89jk7iwxtXw7FNJKwItlm/jrZlIB5rXhuBmAjRHG+2acNopNARQiBSXABqbk3e/GqlEdO0YTUqChiGgUQiJhWBWyvD9135mRCKad8L5M0/X6mgXK1/D6EwZAoaALAjUkUZx0DaNuWxLMU0d2zeh9twRQhhIAQpRwu0j1QqCauRQ2Iw34eR/nrqChEC5ZlI3nrpLiGrsjjxALF0/T6JogizxTl5LeZLNfiNtEB+GMKVKXgMGFZMHjiIDJn4x/UjFBv3XRBGeOHQBDqJZHIAyWQKpwu6vjt8jMyRr+7bK9umo+uTUKk2U50nFwvL1H8Qqi6ZTgvz1J4vlzS5o8f3y3Y6Sbpg3Zp1escVc+5//uI/ZHvl6tWa2Np1a2W7tzenbYsn6FxyWTKXmoHuAlJ21QpClIKlWpjX5MKQXEoSSdJppaIul1XSp8QVFx/P091wKo2UMb6vfzedwJ/9n/8DiUQS85UQnl+/Pyu1Gk7PzDQkBMKwCDTuRMOypK4SQiBSrolbIX1nGkI+AEcihOdXGnsDypUKosbvs+t58BvVUCzbRirTdD0wkIglpW5NmTYSZv07ijk2BvJZ+SDZ29sNp+Gylc3mkMnVx1csFsPIyHKpT7KZNCzLahzLgNXQi7VqDfv27pV9P7zvVyjP18eeLwx4UX0HQSgwMe1KHewVZxE1xr8QPkRUH5OGaSLdlVQmdyFdP9OUfRAC8ANT/jaEQSAftg3DgNU438AP8O///p/n+QbPTdtO7BhmKUglbMRsC0k7hNOY2FmWiWS88QMhBGLKs6NlWvIGjKIIUUiTvJhhy1U19cnPMC3YMUfe+MmkA7OhwKqmB98LG59pTuwMCCFgmvRUFwaWPFb9Yc6o/wnAFuRjkXRMdDn0g2jZRuMJXSBhmzAaDw6BMBDKTwnYoCfmhGPBbuSCTCdiyKfrOb+iQMAqR/IzXUmg+WxgJ3wkuuqfCcMQoU8/sCII4DcmmH5gwFafLyxbXhc/Iv9Y0wzhNZbpOH8iwywM+XwPkskUzFgItzGxMysVlBp+i0JEiEIBqcEskx5kowhR4wFNCAEDtlRQhhGhuVgWRgFMnyaDfhTJCUxkmICcbNlwEjQhisfTUi8mTBuphm93zLaRyWSkXD6flz6cuXwe+e562dFYPIbBoSEpl8t1ST1mWQbsxmSwUqlgcnICQtRXKqeyGdhG/by8CHBFvQ9+ECFZteV5WH4VUeOBOopMWnk0DaRSCWViF9D1MwxYdnMfgBeYaP4u+H4gJ5f1h3BLvn+hsI8dwzAMwzBMh9C2K3bdXUnEHQu2V9DejysmiFScsoW7VX353FeK3efz3bLdahf3Qprb+r5SyaGrS5M7OUVL/6+/SSaCqXl9Nq0WaViZpOX9e//bzZrcyDLa/7/seUPbtvswmZmCiEwdTdNgk/kCFequlKh/mYyjySEk81siQdtiCT3iOGXQtkCpir1CKSoOAJmZuknD80M802Gm2FTMQcyxEEMIs2EusA0DtpLOPmbZcs0uisjfKwIQRI0ldxhItERqq95ooumnZgBRFCBqLPdblgmjUW0lCAXKill9esZF0PAFKVU9lBsVSyJhoBoY8gg9dmP1DsBQdxZrB7sb+zaQy8Ya2wRS4zPSrWCm4mG61DQBi7qdtYFpCFkBJQo81Cr17z8KBTyXnjKTSRNm4ynVCgXQuBYQERzHkk+0tm1KU4wwTMQMS14fobhQpLNp2I3IxZofIFau90+NeO8U8t29SKbSOHzskPb+qaMUaZpy6B6fK+uVIkrF07JtRHQNC/O6ibVQJR1nx+l+7xsc0OSSiovK8lU3yfZoi8448l+7ZdsySFf5of4dTZ2haPwbblivbbv6mqto/0rka9ft79bkfnNgXLbdGrnNuE5LVCzIxBoJ0mMTEyc1uZhS1SjXrZ6/ni2gWq27zXSiKTaZTCOZSmNmfhbVWn1sVMolFM7UXYwMCNhGBWjc/37oySpPURjAq9WvlRBA5Ptyxc4LfOm3FkYRKh6Nu0BEkK4sMUeuYBmWgVii8ZtuALneYVhWfYxmbQvphg4WgY/Z6SkpF48ZcsUuDEN4jQohtm0jUHTFyMgyxBtR3emuFDINs68BA93ZrobeA65aMQy3Unc/KJZrmG5Eo/uBQDlNeswTHqKGGVnAR9TIuGEYgO3QSlyl4iJoyJmmCatpcTAMxOIZKVc3ezdXSgV8v76/ixl3bTuxY5ilIB6zEXcsmL4Bs6F8LAOwlFJ0tkNmVN8LGqaKhtNxQ0YAiMdicqKjPlBEAgik+VQgiiJp9jRMU04iQ0QIwobiEECp7MtJTakWotownYRCwA1pMpeLG7KsUj6Zwqr+XgCAY5sY6EtKU+zJ+QCFal15BFEJc5XmsQRCqBM7SLOKiEL4DSUdhQKBTz/mUWhB2E0zDQDR9NkT0kcPACzbhNUwbwgDsJvmaiEQKlER6WQMiUT9Bzzm+vLaen7nTewYZimIJxKIJxKNiUT9/ndrNVRLTX9EgaTtSf1UrZXhN1J/BYGLWqWxyCEEIMiTuFKrwWtMSPwwRFnxW7Ri5G6RznQh2VysMQxYUrcaSGa64Tj1+7/LNpBt6Ba3UsLsCVoMqWSTiDV9oA1T+uWapoUgjOSxMpk0ko2SfbZtQ67dGAbSqUTjNAQG+roRuvVJvzNThNdIn+MFEVJx0XgAFbCjJEToNK6SDwFH7kMo+jMMIzk5M03yqTMME3baksFoobKYEgRBPYgDkP8vBDbFMgzDMAzDdAi8YscwCo5tw7EtGJEFNEwJpkHL6gAaq0pCtsOG6UtEkE+LBgxEkWguWkFNDRDJfdQlnUZAQ12KVv0iAVRrvnzfDyL4gZDb5POxYQCNpz4D9ZQtVsNs7xgRHKP+JOjAhBn4MGBAQCBh2mh6CySdOBK219i3gKuYMERE4fau52G+ESUoQgHX9+V5JYMYombUbWTBicjRWIsKtkxYEUXqqtkTfCUwwkAE0XCpsAyBhNM8R851wjALQbU2DxghKtUiytVS470SXJfM0cJ10bzJg4hMsQaAVGNFHQaQiicauhIoVV24jVU6P4xgVclMGU8mYDRMAJlcFql0GgBgx+Lo7msmqzaQSHTBbJhiLSOEEM0VrAi1Gpl250tlaYqNDBtmo4CA4ziIxequJwbqeiwMaB9CNa8ISvFSK5XgNt1NPB/JRkSYYwG5rC0/NxdWEHh1/RSEQllxE/IaAUAUhtLEGkUCUUP3G6alWWuE0gchhDRlR+GFr9i17cSuv7sHiZiN6kxNe980lHQdFSXE39Pt0LahpIlQTDetS5RVxZSU7ybfDC/UfzzeOE7+GTNF5UfP1jOxqyanbILkBmw91D4xQ/4y12T1TOynemgfkwXynXErenb4lw6RP44Z0A+in9YzpyOnpCsxldQCuZQmllHMYDVl6Vx4RU1uVX+6IdN5Pie5rgwSMRteKUDoNaLELBuGMnJqLn2vXhDKSZoBAaspZxhwFTMAGUrrPieeMoGJpbpkvqW6a1pDwQQRTp2pX3shBOZKIZofC01D+S4NWGZ9umMYQDIWId44WMbxkDfrPiIWDMSKXl1IAIOxLnQ1jhv5cbherNG/ENPzDSVjAJEfIWqMr+lqEdMzjfEgBIyQpmxhMoVEWO9T2kgCyeY4FIBhofnjkEjEEIvR/Rkp485VIsNswweC+hhLWBYyubp5pBPH3ZEjexBPJHDg9cPa+ydPvS7boZLGJJNLa3Lrrlkl29evv162T01VNbk3p2gf/UOkF1au0dOTZHrJ52xyVvmRP3NEkxt/k/zepgrkR7f+Ok0M/9ta8qsrl/Q+qZYm4ZGO2/fcbk3umnU3y/bg8rxsP/fCM5rcxCTpK9U/qVbV9efsLOnkZBftLxK6z165Uj//oAN9OydOHUI8kcCxk6cwM1c/z3KpjOmZup+3EAJRuSIf3hLJOJxY/R5PpeJYPlwfQ5Zp4Zo118Bu+MvNlXxUqvVr7/oRJgr179wwDPT09cJsRHz29Pch00jjZTkxpLJ52bfpYlVGwFuVWaBaAFB3fzk9Sb+Lc+V5mI3f3b7+QfT314/b1dWFa6/tkccNgkhWnvK9EM2vOQoFhE9uKKeOvIH5mboPX7a7H8ub6Z0MCz2DGflY+eqBEorF+v6q1Qjl5gOvENpYcV1PjsMgDGXbsix0ZfPyWkQiko/1QRig5jbSil3EuGvbiR3DLAWGQatnehLMs33lzrODiz5u4wC/5T6UlS75DgVQqLs4O9EnBXKc6xzV12+ZBPlCOc/1bL2250vuea7vhWGYpeNcOuPt2ufbZiivz60L3qFuvUjImkKv305XLTU8sWOYc1A3EaoTiWb7LT8FPVE2GQ21LeeYdUmrgLJid3YG+HMfqd47qRb14g3K/mDUTbiUVf3c+29aR0Vzb8oEr/UMm/uh4ylb1bYhzv7gBaKaKRiGeedEYVg39am3KvTJVj2Cv2G9ME2ZW86yqG2aZqNiDwUGSMeUVv35lkq0pQpG84U4W1eJxkNwvRIN6QapP1vkTVN/aD9LjzVz60X1gDagkauv2TaksQOGYTSq8Jz7d0E/tnGOlvoh6u+5KvRcjL5r24ldvrcPybiD7i49ZYRpUoh+oUgh/35L5nRTLRECWloXLRnbu7oobN4Htfe/oacdKCs+B4kEhcknYvr+kmkyb3ZbZAbYc3hSkws8+pyb002x/d3UD0MJ3fcD3Sxd8cikoabF8ALdVGUo5mZ1ZDmmPhqFqVS8sKl/gevqcs0o0LDzfmAz+W4k4w5EOgE0fCYMmDDMZkoOgVK5RGk5XFdebwN6qgmhjLvQMqVfiWGYSKVoHEdmQibinSpMo1AsNnbtoap8x7bjwGo47dm2ISN1DdNETBmT3VYIu5GeoFj1sW98utkhiMBuNlFJ9iBsmHOFKdCTrZtig8iGgS5Khhx58nyDKITfOF8hgFqNxkAQhTAbt10YBECgpCBShpoN8j0ULf6LtqWYaAMlYacAOTB2YH7iX/3nz2E7NuxBvbLDmvU3yHZSKW6//rprNLl1a0dkO6zRNRSmbvYsgyrl2A7pGcvKa3J+QOOpPD8j27kWM3ig6IDx06SPE10nNLlctlu2r1qzStsmFDeHaoGqBB14/mVdrkrnf/3mD8j2DTdepclVXyRT7OuHj8p2KqWnsMrle5VX9HtRVH5XAMB1633qRFPsKy+/AMdxECZ7YcXqv13pfA6jyWa6G4G4F8mHtKFlA8hm69cxnUpgZKSvKQZTxOXELpbwUWrk/qr5Pjyz+Z0YcGKUvNeysjDNuinWMGwEYV0HCSFQrcwjaOiauBsg3nCpivxQGXcCU3MlNF/6wkGzCEufH6ArnW24uRgYWjaIeCPFT92c3HiojgzU5mty8nji4BFMn6pXfOnuO4PKTD3y10kkseLWDTJB8+rVA6jW6n2fmp5BdKxZGSfAyROTckJoWjYSDV9BwzRhNn5LDNOAYZJiq9XKsupJEATwPDbFMsyCYFlWPeO3ZStPVAbMxgQoElE9ZF3QJK15o0MxK9angMrqmRLggMZn6KnRQNP7MwiFTGnih+RzUX84pf2bRt2vrr47A7Z8GhUwTQNWQy6MBLzG/kTUnKvWjxvEI4hmyTqD/EMFhJw0CiEAQXn36udf/0gUQZuw6Q+WQn86FSTbfOIlaNXSVN4XhtG6U4ZhFhCvVkMUBjDjkXzwNGHCafrACoGYci8nE0mkUnX/zlQqLttCAJFLD2mWVc9XCQBWBJmPDmg8KDeDvQwThvSHNxt6rjnhUmqzqsWhW1b0gpCWboIwlLk+wzDSVhgdx4bj1CeOzcles+8iEnJi53s+fLfhi+d6CBp+n6ZlwTIB06qvEjqOjVA0Joq2Jf38jMhsBGc0tbsh+2CYpvRDbLUs11fsqOKHYsfBhcLpThiGYRiGYTqE9l2xM23AdGA4znlF4koVhRT0KDFbmbOaSnJZv8WOE5dLzsCZCYqSqpzRl+Ov6lEynSsW0URajyxdt2Y5HVcRDCz9PNTlftua07ZlYnQuvd1rZHvNNSs0uSPjv5LtA4fI9BGzW0yngszUQUBfudkS0evEFPOgYlKMWh4t1KetTuNffv5S3RwYBbJyQv0Jkuqoer4nn8aCMETYNMsKwNBWmKhtKE+IhmHQKh8ANOrBAkClWkGtYfoOowjVmqfsRk0arPuLWDaZsJIGGbdStkEZ21E3O8i9OfNyFTGAgN/06YgEXCU6MVKqUERRpJ1voNSRTBSrchUx5thIxhsmvNYn05YnUNEq0Hxffeo1DGkO78RasVMnpmFZFt5904e09+NxqsTQoxR9WDasR77PFEh3HTtMplMvimtypkHfpWUrrgJC1xlQ9ETokjlXtFz7rlyfbE+XyF3FjOn6ODrPfVHfqOwvQee1anhUE0tY9DkTpNNuuF6P6M3n87L9k+r/ku2JU7pOXz5AFXVCg3S10+KuU2y4RtSjGXUXncudgwcO1f3jYscAi34PmnrCQH3Frcnhgylpzqwn+aXfRaFUcaq5oUzNFIQRyoo7Tz0KtL7/epRt/biGYcCwqSZ3pVqV48YOXZhhM6K1htOnqOKRF0WIGmNqfuoUJhopWJKJJObOnGwYXgyku1Ky/qphGDITQRQGKM00XBQEMH5gL6rl+v2UGJ9B8mD9t9V0bOT3HpV6t1wtywpN1WoNxUaVlyiKUJovyVFuGhZZKAxDpoSBAVg23dSe5yqVJyK58qhmDfhtad+JHcMsAb94+fW3F1oyFtrHZ+7tRRiG6ViOj594e6HLiFaNtn/fywt8hH0LvL/FofOWXBiGYRiGYa5QeGLHMAzDMAzTIbStKbZWCwBhwPCrLVso3L5cprB2z9fnqIFJtv9ShfxPihW9AsTyUboEIqBtK/t0v581w+R/VqnRtuVrb9LkYoJ8NWbnKN1DUgutBzBNtvXRoWXapkKZfFWuupbSGmS7dX++bDdlc5+dor7PzukL0o7i72IK8rnxW4oLK251CJWM7eY5onfU/wzDvDOS6W7Ytg2n5ZYqKJVn4j152a4Euq+bUmEJye4MfSZquXlratUc5W2/ooklkoovrqH4W5ot6aJ6yU8tJsi3z0p2a3JCrTRi6McyQkU/WbR/J637ACe76HXgkr6bPqGnkupNk1/iRz64WbZf/K+jmlxJqURRc6dk263qvzn5TB4AZCoKhml3eMWOYRiGYRimQ+CJHcMwDMMwTIfQtqbY0AgRGiZEqGc6V81/yQRVpejK6GbKk0rx6yPHaZndbrF1xCZPynZtkuSuGdDTk7z/LjKJvn6CTA6Z5f2aXF8vVZE4PUUmgnxeD/83I9p/TKn4UP8cRSrZiYJsTxVOaXInTlHIv+PQ+eezupmmWlXKlNg0lzdabKxRpKfTIDl9/t+BBScYZkkZGl0Jx4mdda/VauRuMlkkdR3L92lyfkBmSjVFVLWkV+TxBe3ftsktI7D0tCipLKUdGegtyLaY0c2UnuKyYUS072SypWKQouIioev0UKkSZDpK1QxLvxalMplf1Qov8ZZrVlT0bjLVI9vvG7tRkzv4+puyvffVCTpOsazJxRoVOnxf7zfDtCu8YscwDMMwDNMh8MSOYRiGYRimQ2hbU2wul0YyEUNg68vfpRKFfwmflvDn5vVI0DfHJ5XPkDkimdDnsqeOkKljMEHmjOXLV2py+WHKbu7MK6bOhG6yHblpA22aIJNqMpjS5ELQeZTLNW3bshSZdz0l07uR1otYj6QpIi2TJxPw/PSEJnd6clq2fYP6W/Nass2bZGNNxymq2Kvq5pxmhYrwPBUDGIa5MIRhQRjWWea+yjyZH+OKeXO+OKPJeTW6lytF+ozTcotm0mRy7e8mM2W2R3cV6c/TsUKbqvNU43r/ZlaSDnJDxVWkJco2DJTI2pZI3dBUdJxiis336JG1UUj7VKP2cznd7BszSI8V5guyLXxdj928nnRmPkPX5fHH/5cmNzVZr0pwMcXYGWYp4BU7hmEYhmGYDoEndsyS88wzz+AP/uAPMDw8DMMw8KMf/UjbLoTAV7/6VSxbtgzJZBKbNm3Ca6+9psnMzMzgE5/4BLLZLPL5PD7zmc9oK7UMwzDtAOs7ZrHhiR2z5JTLZdx0003YsWPHObd/4xvfwLe//W08+OCDeP7555FOp7F582bUlKysn/jEJ7Bv3z489dRTePzxx/HMM8/gc5/73KU6BYZhmN8K1nfMYtO2PnaluRkENQe2p1eKcAxlLqqE0NuWnjKkUiKfu+4M+Y/k0wlNrjpLPnYDw1QdYvmNv6PJ7T1OPiKHDlP7jmU9mlyhQNsG11BVChO6z4mnZDrPCz09SfE0+cQllWzny3pajhWSX4hzI/mjVFvSovznEz+R7ePH6LhWTPcPhOIzp2RIgd8y/zf9ep9qCxT+f8899+Cee+455zYhBL71rW/hK1/5Cj7ykY8AAP7pn/4Jg4OD+NGPfoSPf/zj2L9/P5588kn86le/wq233goA+M53voMPfvCD+Nu//VsMDw+ftV/XdeG65JdULBbPkmGYS0bgAQZgR572dk5RV6M5uj+vvSqvyXUpqZ8sRUeWiwVNrlYhvZhMk25Zd42uW0ZXjsi26ZC/camg7290GVXNWXeEqmRke3Q929NN6VNsW68oESm6RihqPJHWU1gFNaUajvIZpzVFDOi+7u0jv+RSRdfB5QL5Ii/vJ7/me//gbk3uR//67wAWLt0J6ztmseEVO6atOXLkCCYmJrBp0yb5Xi6Xw8aNG7F7924AwO7du5HP56WSA4BNmzbBNE08//zz59zv9u3bkcvl5N/o6OjingjDMMzbwPqOWQh4Yse0NRMT9afqwcFB7f3BwUG5bWJiAgMDA9p227bR09MjZVp54IEHMDc3J/+OHTu2CL1nGIb57WF9xywEbWuKNQ3AMoCwJdWGUMyFJmhpPDR0U+ysUq+5WFQqL7i6qWNZjsy0t/3u78r2yLrbNbn/76F/lO0hJe2I5emZ2E+88TrJXXWdbCd6r9bk0oJMzJWZ09q2ZERmVa9K5oMz87opId9PKVh6h1bJdrWU1eRM5WUYIz+N1soTvk/XxlBC+w2hh/kHQX3Y+JdxCYp4PI54PP72ggxzCbhzw81IJpK46rqbtPdPnqCUScuHyVy69po1mtxQP/3QW4Lu63kl3QcAuEoaEvX+70rr6U66usiUasXIzOu0mIqrZXLteM/1ZLJdtXaVJudHpJBFy3pCEJEeFxb1yXL0nye/RvomUsyipq3vz0goek3Z5vq+Jmdb5IoSegXZ7u/T00q997/dBgCo1lw89pOf43KE9d2VBa/YMW3N0FA919Tk5KT2/uTkpNw2NDSE06f1yXEQBJiZmZEyDMMw7Q7rO2Yh4Ikd09asXr0aQ0ND2Llzp3yvWCzi+eefx9jYGABgbGwMhUIBe/bskTJPP/00oijCxo0bL3mfGYZhLgbWd8xC0LamWEPU/8KW5XO1SLa6Ai+qLXJKoGlPL0VXDaX0yKb33LpWttffQebX2dO6CTgeUDTZVSMUMRYZekTr0ABFV6lRXJWCbsLwAtrmV/WvIQSZAl4/cVy2X9n7oiZ3x+20z94hiugtzutPc44SXNa3ikwuUUs0WeiRyTVQTNZzUwVNzp2v79D1FyYTe6lUwuHDh+XrI0eO4OWXX0ZPTw9WrFiB+++/H3/zN3+Da665BqtXr8Zf/dVfYXh4GPfeey8AYP369fjABz6Az372s3jwwQfh+z62bt2Kj3/84+eMEGOYduPd71qLdDqNd71bN8VWryeTazpHPhW61gGEobioKCbGnrS+giOUW169+6NI32OgRoAqOth1ddeTNVevkO1kjHRLtaxXAhKmouMMXd8JpVJEJKgdGrqrSKSEz3pV6kcY6WZk01bddegs56d1V5Y3j5Cf2Z3vfbdsV3w9E0OqYdo1xMJU2mF9xyw2bTuxY64cXnzxRfyu4t+4bds2AMCnPvUpPPzww/jzP/9zlMtlfO5zn0OhUMB73/tePPnkk0gkyA/on//5n7F161a8//3vh2mauO+++/Dtb3/7kp8LwzDMW8H6jllseGLHLDl33XUXhDh/IIZhGPj617+Or3/96+eV6enpwSOPPLIY3WMYhlkwWN8xiw372DEMwzAMw3QIbbtiFwUhIstE1dV9P2JKqhHbJl8Sy9R92K4eopQhiSTNX1et1BMz3vReWhJftu5G2X5590Oa3IpR2t/Qu26g/vTraQfsVE62KzXy06sWdb+NyZPk3zE7eVzbFiopCZIZWn7v69MrRRw7+ZJsDy5bLttBpSVFTJUyjhvlWTqO0P1lVF+XZJyOFRvSj1uM131Nat7C+JwwzJVOIp1GMp1GV0JPSZFOKSrappROUcuCj6H62CntqKWqTeRHyjbaidHibxsoXnxqViRh6HJdeUrBEoT0mTDS008hop0I6L65pnqAkNqhresdAeWkAyU1U6TvL64c2wmpv+ma3icxSfpv6g2KQh1ZN6LJnTEb+tS8fNM7MVcWvGLHMAzDMAzTIfDEjmEYhmEYpkNoW1OsY9lwLBuzLdUWwhot1SdTSuHrlmXyASXFybFTBdle854PaHIjN6ivydzqz5c1uVyGTKz9a2+W7bKtF8/e99KvZNut0j6KLcW4z5wYp76Huhk5kaCvZflqMrHeuFavXhFYFObvWHlqx1oyrNeo2kTlTcpkHwUtFSWUaX7JIrNFqldPJzA4XE+tUq0tTLoThrnS6cp2I9PVBWHp5seKknZIKEXc3ZYKOuUS6RpPqSDjurouCAIyl/pKGhO16gwAVCqkdytlciMJWtKiZHpIL2ZyednOZ/o0uUQsJtthS/UKGEoVCaWaUEZxQwGA6dP0uZpSkShSKvUAgAE6VhTSNctmdDP3yhVUtqtaoesnIj0lVi5T13+O1WJeZpg2hVfsGIZhGIZhOgSe2DEMwzAMw3QIPLFjGIZhGIbpENrWx86ruTCjEKm43kUjoYSym+QLIULdLyLZRXIf/qMPy/Yd97xfk8v2kZ/F5Bv7Zdsy9f0V5qlEztTRg7J9cl73M/uPH/1ItruS5C9Tc/UUJEOD5JuSzeg+bEeOUyoUT+lHz/AqTW7tDbfQi5D8R2YKevqUiuKXOFul/RlCv7a1KvnPlJRUCKJU0+TW5xvyLa4yDMNcHP/6xFNIJBIInV9o78/OUhqO0twZ2W7NvKH63KkF5MOWvCg9/QOy3d1HZQjjlq4LyjMF2T70GunFYknXY6OrV8q25ZC+y2Z6NbnVq6n02MioXuZs9VXkR9wTJ12VSej+hpFSUg2Kv5vfovstpdakpexvcFWL31+WdKYvSI9bMU0MPT3148bjen8Ypl25oBW77du347bbbkMmk8HAwADuvfdeHDx4UJOp1WrYsmULent70dXVhfvuu09TNAzDMJcDrO8YhrkcuaCJ3a5du7BlyxY899xzeOqpp+D7Pu6++26UyxRR9KUvfQk//elP8eijj2LXrl04efIkPvrRjy54xxmGYRYT1ncMw1yOXJAp9sknn9ReP/zwwxgYGMCePXvwvve9D3Nzc/iHf/gHPPLII/i93/s9AMBDDz2E9evX47nnnsPtt9/+Wx8rEl49a3pLVnFDCdcPBIXrG4ZuckjEadn+5lvIZBl39OX0V1+m6g2zJ1+XbdfVzY/zszOyfezwq7JdEklNzgnpc11KpvhsQje39neTKfbU5IS2LVDSEFTmyfRx7Mg4dPZRP0qUkiBh69ciiJP5ZTqg65JM6ukEUhk6l6RNZor5SlHfXyMdQBBxuhOmc7mU+u7nv3getu0gP7JOe1+EdP+/9Mufy/bKEb06Ql8vmT5PHCd90nqPpnrysu2ZpEsnFfcPAHj/hjHZvvnGd8l2pUUvmg79hBwZf1O2D732uib3yl7Ss/lcl7btvo/9oWzf+a61sh0T+rrDyDKqGuQppljD1CvgqBU1fKXKhWm3VKjIk/5LKpU3Ikv3MWn+Ytht67jEMDrvKHhibq7ud9bTU8/ltmfPHvi+j02bNkmZa6+9FitWrMDu3bvPuQ/XdVEsFrU/hmGYdoP1HcMwlwMXPbGLogj3338/7rzzTlx//fUAgImJCcRiMeTzeU12cHAQExMT59hL3Y8ll8vJv9HR0XPKMQzDLBWs7xiGuVy46MXlLVu2YO/evXj22WffUQceeOABbNu2Tb4uFosNZRcBiBAF+rK47VBFiVCpnOBBj4wazFE28n/7yeOy3TO4T5MbUJf3KxT56jh6lvKuNJkwbZPMAOkW0+7QAJlEqvOzsp209P1NT1GEm+/pJoJMgkyinhKF9tpLL2pypw4ckm03oILWcPQM6aHa3xHFJJzWr60ZJzNLQsm+3g3d3Lz+XasBAJWqD+C/wDCdzmLru3s/9r8jmUwhPnCNJl+Zpwnia6/QvbZsSJ8QmoopMZkgXeVFVU1u7fW0/+5l5KJR6dOrN/z+PbQKqbpolFtMsZFiBQ0EmXZrgS53+jS5srx55KS2LZWi/k4cn5bto/te0+RMpYLOGxOnZXvD3bdqcitXDcu2GjFrJlrCXR3Su4ZabcLQ9XHMqJ9XzGkJRWaYNuWiJnZbt27F448/jmeeeQYjiq/H0NAQPM9DoVDQnmInJycxNDR0jj0B8Xgc8Xj8nNsYhmGWGtZ3DMNcTlyQKVYIga1bt+Kxxx7D008/jdWrV2vbb7nlFjiOg507d8r3Dh48iPHxcYyNjbXujmEYpm1hfccwzOXIBa3YbdmyBY888gh+/OMfI5PJSD+SXC6HZDKJXC6Hz3zmM9i2bRt6enqQzWbxxS9+EWNjYxcUIcYwDLPUsL5jGOZy5IImdt/73vcAAHfddZf2/kMPPYRPf/rTAIBvfvObME0T9913H1zXxebNm/Hd7373gjsWRQaiyEDM1v3FEjb5cUAJcxeWnk4k8ihlyJkz5KdSmtKdmpM+RaVFoGP1dOuZ0/PD/bIdhK5snzip70+A/DBMky6vF7RkRzfINy+dSGnblIwusNQXLSldQo98Ak3F2aVYmdXkvDj52WSGqe/lZEGTm4/I565WpsXc3uxVmlxfw4+wXObSE0zncin1XdwxEY+ZOHRgr/Z+cY70i1DTeHj6vVcqUW49wyBdkGipluBXKC3S3BTtb3JcT3fys3/7mWzPziufKc1pcpks+cfluntkO53Vzc3Hj5Nf3UDfcm1bIku+fr/4VzruzGu/0eRCRacfnqAk0MfL85rcNevJjzCXJd2aU1JMAUAyRelOcmm6Tk5C/81Jpern4qm6mGHamAua2KmK5XwkEgns2LEDO3bsuOhOMQzDLDWs7xiGuRx5R3nsGIZhGIZhmPahbXNpm0YcpmEjEddTbQglrUk6Scvs6Yxe4LniU2h8b4bC3O2WtCjeHC3pRybJVRx92X1wkBynI8UMsu5GPQP8L39OjtSeqMi2Y+jZ0asl2pbNZLVtMSXFuWVQP0o1PYXAkVNkci0U6Lxco6zJ9a+l+fvyvJJKRejh/7NnqE+xmmIqXq6bpauVejqAapUrTzDMQjA/M4mgmsTTP/5X7f1jE8dl2/TJpeI3v2lJbKzol0B1+zB0PfbU40/LdkxJ6XTzu9+jyXmxjGwXXdILb4yf1uSmp/fTZ2p0rJMTRzW5I0dJ7tZ336Jt+z+2UPqXF56jxM7B3LQmV3TJjaSquLy88aJuRv7FnlOynbbJfOvEdBOrpUQnZxRT7MjKVZrcR+77OACgUuF0J8zlAa/YMQzDMAzDdAg8sWMYhmEYhukQ2tYU69gGYraJirL8DgBWgqJfI6WaQ8XXM6xbSpbweIzMj46jR8/GUhQplcvStompSU2uspxMrgOjV8v2idNnNLl33XanbJemKBLsjUN6xYtyqSDbtqX3PZcj06wBMm+cOqFnbB9/U4mKjVPfs4N6lG1/j7I/xZxrzOjXonuWhsPyAYpwG8nr5ubDr9Yj9ao1HwzDvHOGBgaRSqVxzSo9V55Q7n/bpLbV4tphWvSMLiLSfbGEfo/DoUjQ4WGKTr1r82ZNLJNSokkTVJXi1b16pZlDh1+nc1i+SrZrQl8zsBS3mb2HDmjbXj1EFXRSq9bL9smTejWM7jy9HoiRG0mqS3fXmZl4U7anTxyW7akzuk6vhUqUsZJV4FRB/1m84/31bdWqfs0Zpl3hFTuGYRiGYZgOgSd2DMMwDMMwHQJP7BiGYRiGYTqEtvWxG+g1kUqY8Kf1kPdqSH4mZSWrhzD11Bu2kjIkm6V0HTFHz8ReLVPagKSjXA5PvzQv/vKXsn3VOiXr+XG98oSpVMNIKVnfLUvPxJ5Mku9LuaT72FWr9DoIKLVKV1Lfxx3vXivbCSVlSmDpKV1Cn9IVVI+Rj505n9DkBlKU4uDda99F7+cHNbk9p44AAGqefhyGYS6O2TOzqCVd3L7xDu39O37nd2Q7Hqd0HbalP5ObJr2OhOKLBz3Fh++Rnqx6pBemjx/R5GYU/9mZMzOy/YbiUwcAJ0+T/usaGKYNcV23GDHysfMC3W/6qV3PyvbKNTfI9mhPS4UKpZJPSknV4tb0yhNvFMmfuUvRi6HQ9dXEbEm2+/pWyXbF11PEPL3rBQCA73OlHebygFfsGIZhGIZhOgSe2DEMwzAMw3QIbWuKHRmJoSvpIGfoS/qHj5H5YFIpYu2Fupmyq4tOrVyhtCBhVNLkLGVuOzNFZt/5kr5sX/NpH5agdqZLD8mfnCCzxfEymT0joYfKD/aTediI9LQhswWqKBFP03nlcxlNLqaYY1zFxAJbNzeXXZLzSkpFiUif1189OiTbw0PUv2PH9TQB01P178D1ufIEwywEqVQcqWQc00W9usxLv9kj2wMDpGsGB/RKO75POmR2tkAbWqrV2IquWb6aTKej3bpuOXGIqjeUS2Q6HRgc0uRSvXnZthJk9qxU9eMuW7ZCtidOHte2nZkmfbpsmPxrjJZavSVX0ZM26UU/0vVQXHFziStpYbzpKU0OJunCQSVVi+fqJtdmN36L0sEM0xbwih3DMAzDMEyHwBM7hmEYhmGYDqFtTbHZvIOulIPqVEV7v3tAifJKU6TVmUk90qrm0XK6HSMTgdcS2BQp5kQ/pH3MVWc1ubQSkVqrkJmhWtMrT3jK/kKlLYQenVYq0nlls3rm9GyWqmFUqyR3ZlrvU1cXmRwMJSrOCHSbQcym/avBarGWotirrl5Fx1UKXj/zzKua3G8O1QuBB6EePcYwzMURtyPEnQhuraC9/8tf7pRt4ZPeyaZ0neH75DpSU6Lq7ZZn95WrRmX7+tuvk+01K4Y1ucIxMpdOzJKOi7VE5q/pJdPs1BS5udyw7npN7l03rJPtH/y//6Rts0FVJHzFfcXzdHOuCBSTa4LO14rrfVq1+irZPn3sIG0wdX2XVNxc1q+nDAO1iu6uM7psAADgunp/GKZd4RU7hmEYhmGYDoEndsySsn37dtx2223IZDIYGBjAvffei4MHD2oytVoNW7ZsQW9vL7q6unDfffdhclIP6BgfH8eHPvQhpFIpDAwM4Mtf/jKCgPPsMQzTPrC+Yy4FPLFjlpRdu3Zhy5YteO655/DUU0/B933cfffdKCvZp7/0pS/hpz/9KR599FHs2rULJ0+exEc/+lG5PQxDfOhDH4LnefjlL3+J73//+3j44Yfx1a9+dSlOiWEY5pywvmMuBYYQ7RXEXSwWkcvl8NK//HdkUg6mjz6lbZ9TUnQUq+QiWJjW/b2Ks4r7YNgvm+mEHq4fqpUs3IJsz1f0ihcpxU+tK0U+cK7Qj1tRqkj4Lm0zhD6HTscpDL+rS0/pYivpSvyQQvxPTbZUuVAqW+Ty5Edox2K6nJKl/UyZ+jdf1DO2f2DTrbRNUTT/1//9E01usuH2F0UCb87WMDc3h2w2i4VgamoKAwMD2LVrF973vvdhbm4O/f39eOSRR/Cxj30MAHDgwAGsX78eu3fvxu23346f/exn+P3f/32cPHkSg4P1KhkPPvgg/uIv/gJTU1OItVyPc9Ecd8zlw0KOu6WiOe7+n29/D6lkEpMF3Y/r5Bnyb4s8uictX1+diRQ9JizyJbNsfewnFL/kZauXyXYaesqQGSXF0d7j5Nv7y+ee1eSmpyiFyFWryY/utjv0ChppRcf97Kc/1rYJn36ChpS0I6alu4BHIZ1zTKkSZMf09E7r1pGP3dEDL9NxQr3Czwt7XpLtG9+zUbarakkjAMMD9d8Pz/fwg//5A9Z3zJJwIeOOV+yYtmJurp7TqqenBwCwZ88e+L6PTZs2SZlrr70WK1aswO7duwEAu3fvxg033CCVHABs3rwZxWIR+/btw7lwXRfFYlH7YxiGuZSwvmMWA57YMW1DFEW4//77ceedd+L66+tRdRMTE4jFYsjn85rs4OAgJiYmpIyq5Jrbm9vOxfbt25HL5eTf6OjoOeUYhmEWA9Z3zGLRtulOyiUbRuQAVpf2fleaTBVOkpbw0y1Fp3M5Mk2UilWlrTuhlipKupMatTOxXk0u4dByf+BSWhTb1ufGMeWloxTtNgxdLqVUxjBbvoVANTkkaWM2n9LkZmbIlDqvmISzPXrfKwHleHntKJmYD7xyTJMb7KFl3sER5Vimbm7ua1TACKMIb84uXAqALVu2YO/evXj22WffXvgd8sADD2Dbtm3ydbFYZGXHLBnptINUKoZci2NMpp/ScLiK3km0PJPHDDK/iaSS3iilm+WiGqXymJ+nVRsrpZt4BtbkZXtNiszBrx15Xe+gQTrOSZHLx4lT45pYb1/3OdsA4FXJ9Om6VIWiXNZ1i6ukIfFdSgNlJ3S9ODhMrjdvniJ9Pzmu971WomO9vu9l6l9vvyYnuuuracJf2PROrO+YxaJtJ3bMlcXWrVvx+OOP45lnnsHIyIh8f2hoCJ7noVAoaE+xk5OTGBoakjIvvPCCtr9mFFlTppV4PI54S/4rhmGYSwHrO2YxYVMss6QIIbB161Y89thjePrpp7F69Wpt+y233ALHcbBzJyVqPXjwIMbHxzE2NgYAGBsbwyuvvILTp09LmaeeegrZbBbXXXcdGIZh2gHWd8yloG1X7E4eA1IJwC3oJtZMP5kpE0mKGM3pFlv09NCplcq0bF8o6JUsZqdjSpvetyI9S3mkBA+HoRJB1lKAWp0pGyZFvlq2fqmrIUmKlvRDjlKoO6jM0HGret9DJXq2UKJtnt4lzCim6KOH6SQL03r0l1emDw7l6Mlv/crlmlxzd34Y4ddHZ/BO2LJlCx555BH8+Mc/RiaTkT4iuVwOyWQSuVwOn/nMZ7Bt2zb09PQgm83ii1/8IsbGxnD77bcDAO6++25cd911+OM//mN84xvfwMTEBL7yla9gy5Yt/JTKXBZUSoeBMAFE+rO2Y5Bim5wk0+Frrx7V5BJK1H4sl5ftvgHd7DncR5GQtlKtpjenu2+oRWVqShWegQHdZLt8uEe2Tyn+XYcO7dfkVnk0gVFNygAwP0/nVamQ6bQ4pzv4q6bY0COdZsXTmty+vX2y7bnkhjIwoPulLb+RqmMM9NO2vn591SvR2H9tASpPsL5jLgVtO7Fjrgy+973vAQDuuusu7f2HHnoIn/70pwEA3/zmN2GaJu677z64rovNmzfju9/9rpS1LAuPP/44vvCFL2BsbAzpdBqf+tSn8PWvf/1SnQbDMMzbwvqOuRTwxI5ZUn6bNIqJRAI7duzAjh07ziuzcuVKPPHEEwvZNYZhmAWF9R1zKWAfO4ZhGIZhmA6hbVfsQqcXoROHH7tVe9+NyD/DDCgMP5EzNLl8P/nmdZvkxNZT0UPWCzPkm1I4Q3511bJ+acJASRugVJGIAn1/tSr5YagZwC1b99mbr9HnqiXdd8MR5BeSMTN0LFP3OfF96mM8TU+CCUf3s8jHaH9XIS/bN9yk+6asu/Em2V519dWyveF23bfv+Mm6r4vrBcCvj4JhmHeG8FxEFmC2PGvbPumNrEM6Y89zuzS5iUnShYZy/2/YcIsm994x0qfN5LgA8JtfP6/JlWukkw6NU1qkN44e1eSqFdINQpAOTmT1lCFFpcrN/OwZbVu5SD58qha3LV2n5zKU1mRYCTro7l2myQ0Mk4/c8LtvkO2erK7vYmqFDqWtpnABIPW9WhGIYdoZXrFjGIZhGIbpENpuxa7pg1Cp1VeZqjVP2244FDEaRbQSZ1b0pzu7THIwKdqzXNVX2MpVkquoq2g13Rci0iJX32LFzqX9hcoTrBXqoapVl/Zf83xtmxD02lZWG2ueHj7rqi8N2p8l9CdOV6kr6QXUD6el3mRFudYlJTlo1dX75zb60dxvm5Ubvig64RyuNDrhO2ueQ7VWt0T4Lc/agXIv12pkrQgjXe+oUfuGkqzcD/R7vKZEpLpKxKjr6XrWU3RSoOwjajmuUF6rK3ZRS7aASKlFK1r3cZ7vsfVt9dhqZoKg5Rx9Xzkv5XxrbkumA/PCVuyaUbGdNO6Yy4cL+c4M0Wbf8PHjxzkj9mXGsWPHtCSblyNvvPEG1qxZs9TdYC6AThh3rO8uP3jcMUvBhYy7tpvYRVGEkydPQgiBFStW4NixY8hms2//wQ6mWf6l3a6FEALz8/MYHh6GaV7eVv1CoYDu7m6Mj48jl8u9/QeYBeVCxngnjTvWd2fD+m7xiaIIBw8exHXXXdd21/lKYLH1XduZYk3TxMjICIrFeqBANpvlQdegHa9Fp0yCmjdMLpdru2t8JfHbjvFOGnes785NO16LThp3y5fXE8+343W+UlgsfXd5P3YwDMMwDMMwEp7YMQzDMAzDdAhtO7GLx+P42te+xrXvwNfiUsDXeGm50q//lX7+KnwtLg18nZeOxb72bRc8wTAMwzAMw1wcbbtixzAMwzAMw1wYPLFjGIZhGIbpEHhixzAMwzAM0yHwxI5hGIZhGKZD4IkdwzAMwzBMh9CWE7sdO3Zg1apVSCQS2LhxI1544YWl7tKis337dtx2223IZDIYGBjAvffei4MHD2oytVoNW7ZsQW9vL7q6unDfffdhcnJyiXrcWVyJY26xWagxPT4+jg996ENIpVIYGBjAl7/85bMKv1/OXIljj/Xd0nIljrnFpq30nWgzfvCDH4hYLCb+8R//Uezbt0989rOfFfl8XkxOTi511xaVzZs3i4ceekjs3btXvPzyy+KDH/ygWLFihSiVSlLm85//vBgdHRU7d+4UL774orj99tvFHXfcsYS97gyu1DG32CzEmA6CQFx//fVi06ZN4qWXXhJPPPGE6OvrEw888MBSnNKCc6WOPdZ3S8eVOuYWm3bSd203sduwYYPYsmWLfB2GoRgeHhbbt29fwl5dek6fPi0AiF27dgkhhCgUCsJxHPHoo49Kmf379wsAYvfu3UvVzY6Ax9yl4WLG9BNPPCFM0xQTExNS5nvf+57IZrPCdd1LewKLAI+9OqzvLh085i4NS6nv2soU63ke9uzZg02bNsn3TNPEpk2bsHv37iXs2aVnbm4OANDT0wMA2LNnD3zf167NtddeixUrVlxx12Yh4TF36biYMb17927ccMMNGBwclDKbN29GsVjEvn37LmHvFx4eewTru0sDj7lLx1Lqu7aa2J05cwZhGGonBQCDg4OYmJhYol5deqIowv33348777wT119/PQBgYmICsVgM+Xxek73Srs1Cw2Pu0nCxY3piYuKc301z2+UMj706rO8uHTzmLg1Lre/sd9B3ZpHYsmUL9u7di2effXapu8IwCwKPaeZ88NhgOo2lHtNttWLX19cHy7LOihKZnJzE0NDQEvXq0rJ161Y8/vjj+PnPf46RkRH5/tDQEDzPQ6FQ0OSvpGuzGPCYW3zeyZgeGho653fT3HY5w2OP9d2lhsfc4tMO+q6tJnaxWAy33HILdu7cKd+Logg7d+7E2NjYEvZs8RFCYOvWrXjsscfw9NNPY/Xq1dr2W265BY7jaNfm4MGDGB8f7/hrs5hcyWNusVmIMT02NoZXXnkFp0+fljJPPfUUstksrrvuuktzIovElTz2WN8tDVfymFts2krfLUDwx4Lygx/8QMTjcfHwww+LV199VXzuc58T+XxeixLpRL7whS+IXC4n/uM//kOcOnVK/lUqFSnz+c9/XqxYsUI8/fTT4sUXXxRjY2NibGxsCXvdGVypY26xWYgx3Qz/v/vuu8XLL78snnzySdHf399R6U6uxLHH+m7puFLH3GLTTvqu7SZ2Qgjxne98R6xYsULEYjGxYcMG8dxzzy11lxYdAOf8e+ihh6RMtVoVf/qnfyq6u7tFKpUSf/iHfyhOnTq1dJ3uIK7EMbfYLNSYPnr0qLjnnntEMpkUfX194s/+7M+E7/uX+GwWjytx7LG+W1quxDG32LSTvjMaHWIYhmEYhmEuc9rKx45hGIZhGIa5eHhixzAMwzAM0yHwxI5hGIZhGKZD4IkdwzAMwzBMh8ATO4ZhGIZhmA6BJ3YMwzAMwzAdAk/sGIZhGIZhOgSe2DEMwzAMw3QIPLFjGIZhGIbpEHhixzAMwzAM0yHwxI5hGIZhGKZD+P8BVwmQrFY6sosAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# We can also support different types of ablations. For example, we can use block or column ablations.\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = get_cifar_data()\n",
    "for ablation_type in ['block', 'row']:\n",
    "    art_model = PyTorchDeRandomizedSmoothing(model='vit_small_patch16_224', # Name of the model acitecture to load\n",
    "                                        loss=torch.nn.CrossEntropyLoss(), # loss function to use\n",
    "                                        optimizer=torch.optim.SGD, # the optimizer to use: note! this is not initialised here we just supply the class!\n",
    "                                        optimizer_params={\"lr\": 0.01}, # the parameters to use\n",
    "                                        input_shape=(3, 32, 32), # the input shape of the data: Note! that if this is a different shape to what the ViT expects it will be re-scaled\n",
    "                                        nb_classes=10,\n",
    "                                        verbose=False,\n",
    "                                        ablation_type=ablation_type,\n",
    "                                        ablation_size=4, # Size of the retained column\n",
    "                                        replace_last_layer=True, # Replace the last layer with a new set of weights to fine tune on new data\n",
    "                                        load_pretrained=True) # if to load pre-trained weights for the ViT\n",
    "    \n",
    "    # We can see behind the scenes how PyTorchDeRandomizedSmoothing processes input by passing in the first few CIFAR\n",
    "    # images into art_model.ablator.forward along with a start position to retain pixels from the original image.\n",
    "    original_image = np.moveaxis(x_train, [1], [3])\n",
    "\n",
    "    ablated = art_model.ablator.forward(torch.from_numpy(x_train[0:10]).to(device), column_pos=6)\n",
    "    ablated = ablated.cpu().detach().numpy()\n",
    "\n",
    "    # Note the shape:\n",
    "    # - The ablator adds an extra channel to signify the ablated regions of the input.\n",
    "    # - The input is reshaped to be 224 x 224 to match the image shape that the ViT is expecting\n",
    "    print(f\"The shape of the ablated image is {ablated.shape}\")\n",
    "\n",
    "    ablated_image = ablated[:, 0:3, :, :]\n",
    "    \n",
    "    # shift the axis to disply\n",
    "    ablated_image = np.moveaxis(ablated_image, [1], [3])\n",
    "\n",
    "    # plot the figure: Note the axis scale!\n",
    "    f, axarr = plt.subplots(1,4)\n",
    "    axarr[0].imshow(original_image[0])\n",
    "    axarr[1].imshow(ablated_image[0])\n",
    "    axarr[2].imshow(original_image[1])\n",
    "    axarr[3].imshow(ablated_image[1])\n",
    "    plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6ddf5329",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Running algorithm: levine2020\n",
      "INFO:art.estimators.classification.pytorch:Inferred 6 hidden layers on PyTorch classifier.\n",
      "INFO:art.estimators.certification.derandomized_smoothing.pytorch:MNISTModel(\n",
      "  (conv_1): Conv2d(2, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))\n",
      "  (conv_2): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))\n",
      "  (fc1): Linear(in_features=6272, out_features=500, bias=True)\n",
      "  (fc2): Linear(in_features=500, out_features=100, bias=True)\n",
      "  (fc3): Linear(in_features=100, out_features=10, bias=True)\n",
      "  (relu): ReLU()\n",
      ")\n",
      "Normal Acc 0.965 Cert Acc 0.494: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 79/79 [00:02<00:00, 33.61it/s]\n"
     ]
    }
   ],
   "source": [
    "# The algorithm is general such that we do not have to supply only ViTs. \n",
    "# We can use regular CNNs as well, howevever we will loose the advantages \n",
    "# that were discussed at the start of the notebook. Here we will demonstrate it for a simple MNIST case \n",
    "# and also illustrate the use of the algorithm in https://arxiv.org/pdf/2002.10733.pdf\n",
    "\n",
    "class MNISTModel(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(MNISTModel, self).__init__()\n",
    "\n",
    "        self.device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "        self.conv_1 = torch.nn.Conv2d(in_channels=2, # input channels are doubled as per https://arxiv.org/pdf/2002.10733.pdf\n",
    "                                      out_channels=64,\n",
    "                                      kernel_size=4,\n",
    "                                      stride=2,\n",
    "                                      padding=1)\n",
    "\n",
    "        self.conv_2 = torch.nn.Conv2d(in_channels=64,\n",
    "                                      out_channels=128,\n",
    "                                      kernel_size=4,\n",
    "                                      stride=2, padding=1)\n",
    "\n",
    "        self.fc1 = torch.nn.Linear(in_features=128*7*7, out_features=500)\n",
    "        self.fc2 = torch.nn.Linear(in_features=500, out_features=100)\n",
    "        self.fc3 = torch.nn.Linear(in_features=100, out_features=10)\n",
    "\n",
    "        self.relu = torch.nn.ReLU()\n",
    "\n",
    "    def forward(self, x: \"torch.Tensor\") -> \"torch.Tensor\":\n",
    "        \"\"\"\n",
    "        Computes the forward pass though the neural network\n",
    "        :param x: input data of shape (batch size, N features)\n",
    "        :return: model prediction\n",
    "        \"\"\"\n",
    "        x = self.relu(self.conv_1(x))\n",
    "        x = self.relu(self.conv_2(x))\n",
    "        x = torch.flatten(x, 1)\n",
    "        x = self.relu(self.fc1(x))\n",
    "        x = self.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "\n",
    "def get_mnist_data():\n",
    "    \"\"\"\n",
    "    Get the MNIST data.\n",
    "    \"\"\"\n",
    "    train_set = datasets.MNIST('./data', train=True, download=True)\n",
    "    test_set = datasets.MNIST('./data', train=False, download=True)\n",
    "\n",
    "    x_train = train_set.data.numpy().astype(np.float32)\n",
    "    y_train = train_set.targets.numpy()\n",
    "\n",
    "    x_test = test_set.data.numpy().astype(np.float32)\n",
    "    y_test = test_set.targets.numpy()\n",
    "\n",
    "    x_train = np.expand_dims(x_train, axis=1)\n",
    "    x_test = np.expand_dims(x_test, axis=1)\n",
    "\n",
    "    x_train = x_train / 255.0\n",
    "    x_test = x_test / 255.0\n",
    "\n",
    "    return (x_train, y_train), (x_test, y_test)\n",
    "\n",
    "\n",
    "model = MNISTModel()\n",
    "(x_train, y_train), (x_test, y_test) = get_mnist_data()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=0.0005)\n",
    "\n",
    "art_model = PyTorchDeRandomizedSmoothing(model=model,\n",
    "                                         loss=torch.nn.CrossEntropyLoss(),\n",
    "                                         optimizer=optimizer,\n",
    "                                         input_shape=(1, 28, 28),\n",
    "                                         nb_classes=10,\n",
    "                                         ablation_type='column',\n",
    "                                         algorithm='levine2020', # Algorithm selection\n",
    "                                         threshold=0.3, # Requires a threshold\n",
    "                                         ablation_size=2,\n",
    "                                         logits=True)\n",
    "\n",
    "scheduler = torch.optim.lr_scheduler.MultiStepLR(art_model.optimizer, milestones=[200], gamma=0.1)\n",
    "\n",
    "# Uncomment to train.\n",
    "'''\n",
    "art_model.fit(x_train, y_train,\n",
    "              nb_epochs=400,\n",
    "              scheduler=scheduler)\n",
    "torch.save(art_model.model.state_dict(), 'trained_mnist.pt')\n",
    "\n",
    "'''\n",
    "art_model.model.load_state_dict(torch.load('trained_mnist.pt'))\n",
    "acc, cert_acc = art_model.eval_and_certify(x_test, y_test, size_to_certify=5)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}