character-level-language-model/gen_sentence.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Character level language model \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading text data...\n",
      "Creating training set...\n",
      "number of training examples: 31412\n",
      "Vectorizing training set...\n",
      "Loading model...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/engine/saving.py:304: UserWarning: Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.\n",
      "  warnings.warn('Error in loading the saved optimizer '\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "from keras.callbacks import LambdaCallback\n",
    "from keras.models import Model, load_model, Sequential\n",
    "from keras.layers import Dense, Activation, Dropout, Input, Masking\n",
    "from keras.layers import LSTM\n",
    "from keras.utils.data_utils import get_file\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from out_utils import *\n",
    "import sys\n",
    "import io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To save you some time, we have already trained a model for ~1000 epochs on a collection of Shakespearian poems called [*\"The Sonnets\"*](shakespeare.txt). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train the model for one more epoch. When it finishes training for an epoch---this will also take a few minutes---you can run `generate_output`, which will prompt asking you for an input (`<`40 characters). The poem will start with your sentence, and our RNN-Shakespeare will complete the rest of the poem for you! For example, try \"Forsooth this maketh no sense \" (don't enter the quotation marks). Depending on whether you include the space at the end, your results might also differ--try it both ways, and try other inputs as well. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "31412/31412 [==============================] - 60s 2ms/step - loss: 2.7262\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f66b1857320>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print_callback = LambdaCallback(on_epoch_end=on_epoch_end)\n",
    "\n",
    "model.fit(x, y, batch_size=128, epochs=1, callbacks=[print_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Write the beginning of your poem, the Shakespeare machine will complete it. Your input is:  test\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Here is your poem: \n",
      "\n",
      "test titepe thy been me with me,\n",
      "loage prise may my enture, to spantand his browly was so dides you,\n",
      "in weres to fo cufrefite a sivel dated ent,\n",
      "that delese is spite is somd or men whemell kite:\n",
      "on the woolfy ufride his, in than themerfy sris worsf,\n",
      "that nowling happer to dis shall i leprith grawe\n",
      "thy sallen cinnell of this lack, wend in is every\n",
      "my hif furts essed that u had with my crint,\n",
      "exchound n"
     ]
    }
   ],
   "source": [
    "# Run this cell to try with different inputs without having to re-train the model \n",
    "generate_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The RNN-Shakespeare model is very similar to the one you have built for dinosaur names. The only major differences are:\n",
    "- LSTMs instead of the basic RNN to capture longer-range dependencies\n",
    "- The model is a deeper, stacked LSTM model (2 layer)\n",
    "- Using Keras instead of python to simplify the code \n",
    "\n",
    "If you want to learn more, you can also check out the Keras Team's text generation implementation on GitHub: https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py.\n",
    "\n",
    "Congratulations on finishing this notebook! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**References**:\n",
    "- This exercise took inspiration from Andrej Karpathy's implementation: https://gist.github.com/karpathy/d4dee566867f8291f086. To learn more about text generation, also check out Karpathy's [blog post](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).\n",
    "- For the Shakespearian poem generator, our implementation was based on the implementation of an LSTM text generator by the Keras team: https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "nlp-sequence-models",
   "graded_item_id": "1dYg0",
   "launcher_item_id": "MLhxP"
  },
  "kernelspec": {
   "display_name": "conda_tensorflow_p36",
   "language": "python",
   "name": "conda_tensorflow_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}