notebooks/art-for-tensorflow-v2-keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ART for TensorFlow v2 - Keras API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrate applying ART with the new TensorFlow v2 using the Keras API. The code follows and extends the examples on www.tensorflow.org."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "import tensorflow as tf\n",
    "tf.compat.v1.disable_eager_execution()\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "from art.estimators.classification import KerasClassifier\n",
    "from art.attacks.evasion import FastGradientMethod, CarliniLInfMethod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "if tf.__version__[0] != '2':\n",
    "    raise ImportError('This notebook requires TensorFlow v2.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "\n",
    "x_test = x_test[0:100]\n",
    "y_test = y_test[0:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow with Keras API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a model using Keras API. Here we use the Keras Sequential model and add a sequence of layers. Afterwards the model is compiles with optimizer, loss function and metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.InputLayer(input_shape=(28, 28)),\n",
    "    tf.keras.layers.Flatten(),\n",
    "    tf.keras.layers.Dense(128, activation='relu'),\n",
    "    tf.keras.layers.Dropout(0.2),\n",
    "    tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the model on training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples\n",
      "Epoch 1/3\n",
      "60000/60000 [==============================] - 3s 45us/sample - loss: 0.3007 - accuracy: 0.9140\n",
      "Epoch 2/3\n",
      "60000/60000 [==============================] - 3s 44us/sample - loss: 0.1438 - accuracy: 0.9572\n",
      "Epoch 3/3\n",
      "60000/60000 [==============================] - 3s 43us/sample - loss: 0.1074 - accuracy: 0.9667\n"
     ]
    }
   ],
   "source": [
    "model.fit(x_train, y_train, epochs=3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate model accuracy on test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on test data: 100.00%\n"
     ]
    }
   ],
   "source": [
    "loss_test, accuracy_test = model.evaluate(x_test, y_test)\n",
    "print('Accuracy on test data: {:4.2f}%'.format(accuracy_test * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a ART Keras classifier for the TensorFlow Keras model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = KerasClassifier(model=model, clip_values=(0, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fast Gradient Sign Method attack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a ART Fast Gradient Sign Method attack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "attack_fgsm = FastGradientMethod(estimator=classifier, eps=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate adversarial test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test_adv = attack_fgsm.generate(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate accuracy on adversarial test data and calculate average perturbation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on adversarial test data: 0.00%\n",
      "Average perturbation: 0.18\n"
     ]
    }
   ],
   "source": [
    "loss_test, accuracy_test = model.evaluate(x_test_adv, y_test)\n",
    "perturbation = np.mean(np.abs((x_test_adv - x_test)))\n",
    "print('Accuracy on adversarial test data: {:4.2f}%'.format(accuracy_test * 100))\n",
    "print('Average perturbation: {:4.2f}'.format(perturbation))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualise the first adversarial test sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAECCAYAAAD+eGJTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAQNklEQVR4nO3df6zV9X3H8dfb6y3oBVp+bBQtzF8sQ7sM3Q24ahsWY2UljZpsZsQ41i7DbZpp1nQzbIn+s8wsKm2W1RQHLS7qYqZWjHTKSFPaVEnRMkVuFcZgiNeLQBk/Eilc3vvjfnV39p7P99z7Pd8f976fj8Rw7nmfc75vvufy8vs9n8/5fM3dBSCuc+puAEC9CAEgOEIACI4QAIIjBIDgCAEguFpCwMyWmtmbZrbbzO6po4cUM9trZq+b2XYz29aAftaZ2UEz2zHsvhlmtsnMdmV/Tm9Yf/eZ2YFsH243sy/U2N9cM/ueme00szfM7K7s/kbsw0R/lexDq3qegJl1SXpL0vWS3pb0Y0nL3X1npY0kmNleSb3ufqjuXiTJzD4n6YSkR93909l9fy/piLvfnwXpdHf/qwb1d5+kE+7+QB09DWdmcyTNcfdXzWyqpFck3STpD9WAfZjo7xZVsA/rOBJYJGm3u+9x959L+hdJN9bQx7jh7lskHfnI3TdKWp/dXq+hX5patOivMdy9391fzW4fl9Qn6UI1ZB8m+qtEHSFwoaT9w35+WxX+hdvkkl40s1fMbGXdzbQw2937s9vvSppdZzMt3Glmr2WnC7WdrgxnZhdJulLSVjVwH36kP6mCfcgHgyO71t2vkvQ7ku7IDncby4fO6Zo2//thSZdKWiipX9KD9bYjmdkUSU9Jutvdjw2vNWEfjtBfJfuwjhA4IGnusJ8/ld3XGO5+IPvzoKRnNHQK0zQD2bnkB+eUB2vu5/9x9wF3H3T3s5IeUc370My6NfQP7DF3fzq7uzH7cKT+qtqHdYTAjyXNN7OLzexjkn5f0oYa+hiRmfVkH87IzHokfV7SjvSzarFB0ors9gpJz9bYyy/44B9X5mbVuA/NzCStldTn7g8NKzViH7bqr6p9WPnogCRlQx1fk9QlaZ27/23lTbRgZpdo6P/+knSupMfr7s/MnpC0RNIsSQOS7pX0HUlPSponaZ+kW9y9lg/nWvS3REOHsS5pr6Tbh51/V93ftZJ+IOl1SWezu1dp6Ly79n2Y6G+5KtiHtYQAgObgg0EgOEIACI4QAIIjBIDgCAEguFpDoMFTciXRX1FN7q/JvUnV9lf3kUCj3wjRX1FN7q/JvUkV9ld3CACoWaHJQma2VNLXNTTz75/c/f7U4z9mk3yyej78+bROqVuTPvx5cGbPSE/7UNfhk8l63vNH68z7J3Xu5PZfM6+/Thvt/svT6f6rfn9H8/zRvrftvH6e0bw/Y+kv5dSJIzrz/kkbqXbuWF80WxzkHzVscRAz25BaHGSyerTYrmv5mkeX/VZym5/455eS9bznly2vv7IV/fuX3X/Z72/Zvx9F90+dv587n1/dslbkdIDFQYAJoEgIjIfFQQDkGPPpQLuyoY6VkjRZ55e9OQCjVORIoK3FQdx9jbv3unvv8A+JADRDkRBo9OIgANoz5tMBdz9jZndKekH/tzjIGx3rbARHbyv26XDR1y/7+XWPLuQp+vcrW937v+n7p5VCnwm4+0ZJGzvUC4AaMGMQCI4QAIIjBIDgCAEgOEIACI4QAIIrfdpwlYrOI8j9FlrN48Blb7/o/qt7nkVRZc9DyVPX7x9HAkBwhAAQHCEABEcIAMERAkBwhAAQHCEABFfpPIHBmT2FVlwt+/vedX+fv+5x8jxN7y9P3e9vnjLXQ+jy1sulcyQABEcIAMERAkBwhAAQHCEABEcIAMERAkBwhS5NPlrTbIYnr0pc8zj+eB8Hj26izwMoYufzq3Xy0P4RL03OkQAQHCEABEcIAMERAkBwhAAQHCEABEcIAME1ap5AnrLXxa97nJl5Cs02nueppOYJFFpUxMz2SjouaVDSGXfvLfJ6AKrXiZWFftvdD3XgdQDUgM8EgOCKhoBLetHMXjGzlZ1oCEC1ip4OXOvuB8zslyVtMrOfuvuW4Q/IwmGlJE3W+QU3B6DTCh0JuPuB7M+Dkp6RtGiEx6xx91537+3WpCKbA1CCMYeAmfWY2dQPbkv6vKQdnWoMQDWKnA7MlvSMmX3wOo+7+78VaabscfyJPg+gruvbTxTjeR5AEWMOAXffI+k3OtgLgBowRAgERwgAwRECQHCEABAcIQAERwgAwXXiW4RtG5zZo6PLWo+V1j1Om7f9c85PT3ve9xcLk/XH/2h1sv4nf3NXsv7xx15O1vOM93kEdc/zmKg4EgCCIwSA4AgBIDhCAAiOEACCIwSA4AgBILhK5wnUreg485vfWJCsXzp3f7L+1T2/m6xP/fKBZP3sl+cm64dO9CTrJ/qmJ+u7/uDhZD3P/Ef/tNDr5z3/yP3F5jG8dOsDyfqtc69J1pt+3Yqx4kgACI4QAIIjBIDgCAEgOEIACI4QAIIjBIDgzN0r29g0m+GL7brKtlc1u/KKZN1/8kah57+3aFqy/j+fez9Zv/iT6YtHb1rwXLJ+fd8XS33+qTPpaSv9hz+erF80+3CynufUP8xJ16d2FXr9sqXmKWz1zTrmR2ykGkcCQHCEABAcIQAERwgAwRECQHCEABAcIQAEN6HWEyj7+96F1+X/dLH+Zv0k/fKzvpmuv7W2N1m/4br0dRN2r5td6PnnKL3ewnnJqnRJTv3tp9LzLC74xLFkfUrfkWT9vDd3J+tNv25DK7lHAma2zswOmtmOYffNMLNNZrYr+zO9WgWAxmrndODbkpZ+5L57JG129/mSNmc/AxiHckPA3bdI+uhx0o2S1me310u6qcN9AajIWD8YnO3u/dntdyWlTxYBNFbh0QEf+gZSy28hmdlKM9tmZttO61TRzQHosLGGwICZzZGk7M+DrR7o7mvcvdfde7s1aYybA1CWsYbABkkrstsrJD3bmXYAVC13PQEze0LSEkmzJA1IulfSdyQ9KWmepH2SbnH39CCr6l9PoOxx3Lxx/qavW1/3/slzzuTJyfp397xc6PVvuCA9z6Hp8wDGup5A7mQhd1/eojRxVwcBAmHaMBAcIQAERwgAwRECQHCEABAcIQAEN6HWE8hTdBy/7O2Xrenj3Hn+69H5OY9IzxO44qVbk/Upt6Wv6zBRcSQABEcIAMERAkBwhAAQHCEABEcIAMERAkBwlc4TGJzZo6PLWo9V1z2Onqfp/TV9HkBef1Pe+XmyPndmywWs2nJ6V8x5AHk4EgCCIwSA4AgBIDhCAAiOEACCIwSA4AgBILhQ6wnkGe/XDWi6g58ZTNa7L/5Zsn5ezuvnXTfgYo3v96eseSAcCQDBEQJAcIQAEBwhAARHCADBEQJAcIQAEFyl8wS6Dp8c12Ppdffe9PUC8vzZZzcn6y8OXJ6sv/39ucn6PO0fdU9Vauo8k9wjATNbZ2YHzWzHsPvuM7MDZrY9++8L5bYJoCztnA58W9LSEe5f7e4Ls/82drYtAFXJDQF33yLpSAW9AKhBkQ8G7zSz17LThekd6whApcYaAg9LulTSQkn9kh5s9UAzW2lm28xs22mdGuPmAJRlTCHg7gPuPujuZyU9ImlR4rFr3L3X3Xu7NWmsfQIoyZhCwMzmDPvxZkk7Wj0WQLOZu6cfYPaEpCWSZkkakHRv9vNCSS5pr6Tb3b0/b2PTbIYvtuta1sseB697nD/PeJ8HkLd/d3/t6mT9k7+Wvq7AlKV7Rt1TlZr8/u18frVOHtpvI9VyJwu5+/IR7l5buCsAjcC0YSA4QgAIjhAAgiMEgOAIASA4QgAIrlHXHSi67n/dmt5fUXnvz+CSq5L1S379QLL+zr+n1wuYovQ8gaLf16/7/Ss6j2Ws/XMkAARHCADBEQJAcIQAEBwhAARHCADBEQJAcI2aJ1D2OG/d48BNV3Sc+syq9Hq0Pzs+JVn/1N/9qND280z09z/1/nX5yZY1jgSA4AgBIDhCAAiOEACCIwSA4AgBIDhCAAiu0nkCgzN7dHRZ67Hapl8XILq3vvWbyfplXenrBjx31SPJ+u/d9tVR9zSRFJ3HMNZ/PxwJAMERAkBwhAAQHCEABEcIAMERAkBwhAAQXKXzBLoOnyw0F2Cifx+8bHn7/j8fvDpZv2xe+roBmxY8l6wv+OZfJuvT5Mn6RFfXdRFyjwTMbK6Zfc/MdprZG2Z2V3b/DDPbZGa7sj+nl9IhgFK1czpwRtJX3P1ySVdLusPMLpd0j6TN7j5f0ubsZwDjTG4IuHu/u7+a3T4uqU/ShZJulLQ+e9h6STeV1SSA8ozqg0Ezu0jSlZK2Sprt7v1Z6V1JszvaGYBKtB0CZjZF0lOS7nb3Y8Nr7u7SyJ/qmNlKM9tmZttO61ShZgF0XlshYGbdGgqAx9z96ezuATObk9XnSBrxK2Tuvsbde929t1uTOtEzgA5qZ3TAJK2V1OfuDw0rbZC0Iru9QtKznW8PQNls6Eg+8QCzayX9QNLrks5md6/S0OcCT0qaJ2mfpFvcPbnw/DSb4YvtupZ15gEUU3Q9hhfe2V7o+b/6/RXJ+vSN5xd6/fGuzvUytvpmHfMjNlItd7KQu/9Q0ohPltT6XzSAcYFpw0BwhAAQHCEABEcIAMERAkBwhAAQXKXrCSCt6Djy/r/+TLK+845vFHr9L/33Z5P16d+NPQ8gT948mLrmEXAkAARHCADBEQJAcIQAEBwhAARHCADBEQJAcJXOExic2aOjy1qPlda17vpE8aXlL5T6+lteviJZnxX7sgGFFZ1HkHr+4PMvt6xxJAAERwgAwRECQHCEABAcIQAERwgAwRECQHCVzhPoOnyy1rXX8xTtreg8hrznH7xmMFl/YcaaQttHvVhPAEAtCAEgOEIACI4QAIIjBIDgCAEgOEIACC53noCZzZX0qKTZklzSGnf/upndJ+mPJb2XPXSVu28sq9F21D3On6dof4vv6k7Wr+/7YrK+acFzhbZ/2d2tv5PejujrQRRdL6OseQTtTBY6I+kr7v6qmU2V9IqZbcpqq939gVI6A1CJ3BBw935J/dnt42bWJ+nCshsDUI1RfSZgZhdJulLS1uyuO83sNTNbZ2bTO9wbgAq0HQJmNkXSU5Ludvdjkh6WdKmkhRo6UniwxfNWmtk2M9t2Wqc60DKATmorBMysW0MB8Ji7Py1J7j7g7oPuflbSI5IWjfRcd1/j7r3u3tutSZ3qG0CH5IaAmZmktZL63P2hYffPGfawmyXt6Hx7AMrWzujANZJuk/S6mW3P7lslabmZLdTQsOFeSbeX0iGAUrUzOvBDSTZCadRzAvKuO5Cn6dclKPv74D/988uT9U3/ur7Q699wwcJCz8/T5LUkOqHu37+xYsYgEBwhAARHCADBEQJAcIQAEBwhAARHCADBVXrdgTxljyOP93Fq+9F/JOtlj/Mjre7fr9T2u/xkyxpHAkBwhAAQHCEABEcIAMERAkBwhAAQHCEABGfuXt3GzN6TtG/YXbMkHaqsgdGjv2Ka3F+Te5M639+vuPsvjVSoNAR+YeNm29y9t7YGctBfMU3ur8m9SdX2x+kAEBwhAARXdwisqXn7eeivmCb31+TepAr7q/UzAQD1q/tIAEDNCAEgOEIACI4QAIIjBIDg/hdPERq2aNIHzwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.matshow(x_test_adv[0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Carlini&Wagner Infinity-norm attack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a ART Carlini&Wagner Infinity-norm attack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "attack_cw = CarliniLInfMethod(classifier=classifier,\n",
    "                              max_iter=100,\n",
    "                              learning_rate=0.01,\n",
    "                              initial_const=1e0,\n",
    "                              largest_const=2e0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate adversarial test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "55caf40ad922412ead9416b007891841",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "C&W L_inf:   0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_test_adv = attack_cw.generate(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate accuracy on adversarial test data and calculate average perturbation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on adversarial test data: 35.00%\n",
      "Average perturbation: 0.02\n"
     ]
    }
   ],
   "source": [
    "loss_test, accuracy_test = model.evaluate(x_test_adv, y_test)\n",
    "perturbation = np.mean(np.abs((x_test_adv - x_test)))\n",
    "print('Accuracy on adversarial test data: {:4.2f}%'.format(accuracy_test * 100))\n",
    "print('Average perturbation: {:4.2f}'.format(perturbation))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualise the first adversarial test sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAECCAYAAAD+eGJTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAOGElEQVR4nO3df6xf9V3H8ddr7e1lvS2uHaPWUqhjbJHNUcwdbAFNF2SyLaSQbbgmNjWZK1FIwCwqIVlook4k/BCdkhSp6xZgwxWEbHWuaaZIxI7SFFpaFMSirZdeoNOWAf359o97ild27+d7e7/f7znf2/fzkTTf7/e8z/ecd09vX/dzzvdzz3VECEBe72i6AQDNIgSA5AgBIDlCAEiOEACSIwSA5BoJAduX2f4X28/bvqGJHkps77K9zfZW25t7oJ81todtbx+1bK7tDbafqx7n9Fh/q2zvqY7hVtufarC/hbZ/YHuH7WdsX1ct74ljWOivlmPouucJ2J4m6V8lXSppt6QnJC2LiB21NlJge5ekwYh4peleJMn2L0l6TdLXI+JD1bJbJO2LiJurIJ0TEb/XQ/2tkvRaRNzaRE+j2Z4vaX5EbLE9W9KTkq6Q9OvqgWNY6O8q1XAMmxgJXCDp+Yh4ISIOSfqmpKUN9DFlRMSjkva9bfFSSWur52s18kXTiHH66xkRMRQRW6rnByTtlLRAPXIMC/3VookQWCDpP0e93q0a/8ITFJK+b/tJ2yubbmYc8yJiqHr+kqR5TTYzjmttP12dLjR2ujKa7UWSzpe0ST14DN/Wn1TDMeTC4NgujohfkPRJSddUw92eFSPndL02//suSWdLWixpSNJtzbYj2Z4laZ2k6yNi/+haLxzDMfqr5Rg2EQJ7JC0c9fqMalnPiIg91eOwpIc0cgrTa/ZW55LHzymHG+7n/4mIvRFxNCKOSbpbDR9D230a+Q92b0Q8WC3umWM4Vn91HcMmQuAJSefY/lnbMyR9XtIjDfQxJtsD1cUZ2R6Q9AlJ28vvasQjklZUz1dIerjBXn7C8f9clSvV4DG0bUn3SNoZEbePKvXEMRyvv7qOYe2fDkhS9VHHn0iaJmlNRPxh7U2Mw/Z7NfLdX5KmS7qv6f5s3y9piaTTJO2VdJOkv5H0gKQzJb0o6aqIaOTi3Dj9LdHIMDYk7ZJ09ajz77r7u1jSP0raJulYtfhGjZx3N34MC/0tUw3HsJEQANA7uDAIJEcIAMkRAkByhACQHCEAJNdoCPTwlFxJ9NeuXu6vl3uT6u2v6ZFAT/9DiP7a1cv99XJvUo39NR0CABrW1mQh25dJulMjM//+MiJuLq0/w/1xigbeen1YB9Wn/knvv9vorz293F8v9yZ1vr839WMdioMeqzbpEJjMzUFO9dy40JdMan8AJm9TbNT+2DdmCLRzOsDNQYCTQDshMBVuDgKghend3kH1UcdKSTpFM7u9OwAnqJ2RwIRuDhIRqyNiMCIGe/lCDJBVOyHQ0zcHATAxkz4diIgjtq+V9Hf6v5uDPNOxzgDUoq1rAhGxXtL6DvUCoAHMGASSIwSA5AgBIDlCAEiOEACSIwSA5AgBIDlCAEiOEACSIwSA5AgBIDlCAEiOEACSIwSA5AgBIDlCAEiOEACSIwSA5AgBIDlCAEiOEACSIwSA5AgBIDlCAEiOEACSIwSA5AgBIDlCAEiOEACSIwSA5AgBILnp7bzZ9i5JByQdlXQkIgY70RSA+rQVApWPR8QrHdgOgAZwOgAk124IhKTv237S9spONASgXu2eDlwcEXtsny5pg+1nI+LR0StU4bBSkk7RzDZ3B6DT2hoJRMSe6nFY0kOSLhhjndURMRgRg33qb2d3ALpg0iFge8D27OPPJX1C0vZONQagHu2cDsyT9JDt49u5LyK+15GuANRm0iEQES9IOq+DvQBoAB8RAskRAkByhACQHCEAJEcIAMkRAkBynfgpwjRe/eLHivUzlz9frD87PK9YP3Swr1hfcH+5PnP3a8X6sa07inXkxEgASI4QAJIjBIDkCAEgOUIASI4QAJIjBIDkmCdwAn73d+4r1j8z8KPyBs5us4El5fKuI68X63e+/PE2G5jafjh8VrE+cNtPFevTNz7ZyXZ6BiMBIDlCAEiOEACSIwSA5AgBIDlCAEiOEACSc0TUtrNTPTcu9CW17a/TfvzZC4v1Vz5cztQ5O8vH+kc/52J9xof/u1i/5UMPFuuXvvONYv27r88q1j89s3y/gna9EYeK9U0HB4r1Jaccbmv/7/vu1cX6+1c+0db2m7QpNmp/7BvzC4yRAJAcIQAkRwgAyRECQHKEAJAcIQAkRwgAyXE/gRMw8O1NLertbf/U9t6uP/vpJcX6H1y0qLz/fyj/3oRblrzvBDs6MdPfOFasDzw9VKy/+9F1xfrPz2jxext2lesnq5YjAdtrbA/b3j5q2VzbG2w/Vz3O6W6bALplIqcDX5N02duW3SBpY0ScI2lj9RrAFNQyBCLiUUn73rZ4qaS11fO1kq7ocF8AajLZC4PzIuL4CdpLksq/ZA9Az2r704EY+QmkcX8yxvZK25ttbz6sg+3uDkCHTTYE9tqeL0nV4/B4K0bE6ogYjIjBPvVPcncAumWyIfCIpBXV8xWSHu5MOwDq1nKegO37NXLH+9Ns75Z0k6SbJT1g+wuSXpR0VTebxMQceWlvsT6wrlw/2mL7A99+9QQ76qy9v/GxYv2DM8pfzrfu+0CxvuivXijWjxSrU1fLEIiIZeOUpu7dQQC8hWnDQHKEAJAcIQAkRwgAyRECQHKEAJAc9xNAz5h+1sJi/as3frVY7/O0Yv2v7/zlYv3dQ48X6ycrRgJAcoQAkBwhACRHCADJEQJAcoQAkBwhACTHPAH0jGd/e0Gx/pF+F+vPHHqjWJ+74/UT7ikDRgJAcoQAkBwhACRHCADJEQJAcoQAkBwhACTHPAHU5uCnP1Ksb/nsHS22UP4NVr953XXF+jv/6Ycttp8TIwEgOUIASI4QAJIjBIDkCAEgOUIASI4QAJJjngBq8x+fLH/PmeXyPIBl/35psT7ze08V61Gs5tVyJGB7je1h29tHLVtle4/trdWfT3W3TQDdMpHTga9JumyM5XdExOLqz/rOtgWgLi1DICIelbSvhl4ANKCdC4PX2n66Ol2Y07GOANRqsiFwl6SzJS2WNCTptvFWtL3S9mbbmw/r4CR3B6BbJhUCEbE3Io5GxDFJd0u6oLDu6ogYjIjBvhY/BQagfpMKAdvzR728UtL28dYF0NtazhOwfb+kJZJOs71b0k2SltherJGPXndJurqLPWKKeMfs2cX68l98rFjff+zNYn34K+8t1vsPPlGsY2wtQyAilo2x+J4u9AKgAUwbBpIjBIDkCAEgOUIASI4QAJIjBIDkuJ8AOua5VR8s1r9z2l8U60uf+0yx3r+eeQDdwEgASI4QAJIjBIDkCAEgOUIASI4QAJIjBIDkmCeACfufX/tosf70r/5psf5vRw4X66/98RnFer+GinVMDiMBIDlCAEiOEACSIwSA5AgBIDlCAEiOEACSY54A3jJ9wc8U69d/+VvFer/LX06ff2p5sf6ev+V+AU1gJAAkRwgAyRECQHKEAJAcIQAkRwgAyRECQHLME0jE08v/3Od9Z3ex/rlZrxbr9x44vVif9+Xy95xjxSq6peVIwPZC2z+wvcP2M7avq5bPtb3B9nPV45zutwug0yZyOnBE0pci4lxJH5V0je1zJd0gaWNEnCNpY/UawBTTMgQiYigitlTPD0jaKWmBpKWS1larrZV0RbeaBNA9J3Rh0PYiSedL2iRpXkQcv+nbS5LmdbQzALWYcAjYniVpnaTrI2L/6FpEhKQY530rbW+2vfmwDrbVLIDOm1AI2O7TSADcGxEPVov32p5f1edLGh7rvRGxOiIGI2KwT/2d6BlAB03k0wFLukfSzoi4fVTpEUkrqucrJD3c+fYAdNtE5glcJGm5pG22t1bLbpR0s6QHbH9B0ouSrupOi+iY8z5QLP/+6d9oa/N//pXPFevveurxtraP7mgZAhHxmCSPU76ks+0AqBvThoHkCAEgOUIASI4QAJIjBIDkCAEgOe4ncBKZdu77i/WV32xvPte5a64p1hd945/b2j6awUgASI4QAJIjBIDkCAEgOUIASI4QAJIjBIDkmCdwEnn2t8p3fb985v5ivZUz/v5QeYUY8w5z6HGMBIDkCAEgOUIASI4QAJIjBIDkCAEgOUIASI55AlPIm5dfUKxvvPy2FluY2blmcNJgJAAkRwgAyRECQHKEAJAcIQAkRwgAyRECQHIt5wnYXijp65LmSQpJqyPiTturJH1R0svVqjdGxPpuNQrpvy6aVqyfOb29eQD3Hji9WO/bX76fAHcTmJomMlnoiKQvRcQW27MlPWl7Q1W7IyJu7V57ALqtZQhExJCkoer5Ads7JS3odmMA6nFC1wRsL5J0vqRN1aJrbT9te43t8r2tAPSkCYeA7VmS1km6PiL2S7pL0tmSFmtkpDDmxHXbK21vtr35sA52oGUAnTShELDdp5EAuDciHpSkiNgbEUcj4pikuyWN+dMtEbE6IgYjYrBP/Z3qG0CHtAwB25Z0j6SdEXH7qOXzR612paTtnW8PQLdN5NOBiyQtl7TN9tZq2Y2SltlerJFPhnZJurorHQLoqol8OvCYJI9RYk7AFPNHr55brD/+K4uK9Rja1sFu0CuYMQgkRwgAyRECQHKEAJAcIQAkRwgAyRECQHKOGn+n/KmeGxf6ktr2B2DEptio/bFvrPk+jASA7AgBIDlCAEiOEACSIwSA5AgBIDlCAEiu1nkCtl+W9OKoRadJeqW2Bk4c/bWnl/vr5d6kzvd3VkS8Z6xCrSHwEzu3N0fEYGMNtEB/7enl/nq5N6ne/jgdAJIjBIDkmg6B1Q3vvxX6a08v99fLvUk19tfoNQEAzWt6JACgYYQAkBwhACRHCADJEQJAcv8LId/VeNhqNOUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.matshow(x_test_adv[0, :, :])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}