exercise_web_scraping.ipynb

{
 "metadata": {
  "name": "exercise_web_scraping"
 }, 
 "nbformat": 2, 
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown", 
     "source": [
      "## Prerequisites", 
      "", 
      "This exercise assumes a basic level of understanding of the web, HTML/HTTP/CSS.", 
      "", 
      "Also, you will need to install a few third party packages.  From the command line run the following:", 
      "", 
      "    easy_install requests", 
      "    easy_install beautifulsoup4", 
      "", 
      "After you install these packages you should be able to do the following imports without any errors:"
     ]
    }, 
    {
     "cell_type": "code", 
     "collapsed": true, 
     "input": [
      "import requests", 
      "import bs4"
     ], 
     "language": "python", 
     "outputs": [], 
     "prompt_number": 1
    }, 
    {
     "cell_type": "markdown", 
     "source": [
      "## Introduction to Requests", 
      "", 
      "Requests allows you to make HTTP requests, here is an example:"
     ]
    }, 
    {
     "cell_type": "code", 
     "collapsed": false, 
     "input": [
      "response = requests.get('http://www.google.com/')", 
      "response.text"
     ], 
     "language": "python", 
     "outputs": [], 
     "prompt_number": 2
    }, 
    {
     "cell_type": "markdown", 
     "source": [
      "this gets the page located at 'http://www.google.com/' and displays the HTML it contains."
     ]
    }, 
    {
     "cell_type": "code", 
     "collapsed": false, 
     "input": [
      "response.headers"
     ], 
     "language": "python", 
     "outputs": [], 
     "prompt_number": 3
    }, 
    {
     "cell_type": "markdown", 
     "source": [
      "this displays the headers of the response "
     ]
    }, 
    {
     "cell_type": "code", 
     "collapsed": false, 
     "input": [
      "response.status_code"
     ], 
     "language": "python", 
     "outputs": [], 
     "prompt_number": 4
    }, 
    {
     "cell_type": "markdown", 
     "source": [
      "and this shows the status code, pretty simple.", 
      "", 
      "For more information on the Requests library visit: <a href=\"http://docs.python-requests.org/\" target=\"_blank\">http://docs.python-requests.org/</a>"
     ]
    }, 
    {
     "cell_type": "markdown", 
     "source": [
      "## Introduction to Beautiful Soup", 
      "", 
      "Beautiful Soup allows you to parse HTML into something that you can query.  Using the previous response let's use Beautiful Soup to extract all the images.", 
      "", 
      "First create a BeautifulSoup object:"
     ]
    }, 
    {
     "cell_type": "code", 
     "collapsed": true, 
     "input": [
      "soup = bs4.BeautifulSoup(response.text)"
     ], 
     "language": "python", 
     "outputs": [], 
     "prompt_number": 5
    }, 
    {
     "cell_type": "markdown", 
     "source": [
      "With this object we can select anything in the document with CSS selectors:"
     ]
    }, 
    {
     "cell_type": "code", 
     "collapsed": false, 
     "input": [
      "image_tags = soup.select('img')", 
      "image_tags"
     ], 
     "language": "python", 
     "outputs": [], 
     "prompt_number": 6
    }, 
    {
     "cell_type": "markdown", 
     "source": [
      "This gives us all the image tags but we just want the links. Loop over them to extract just the links:"
     ]
    }, 
    {
     "cell_type": "code", 
     "collapsed": false, 
     "input": [
      "for tag in image_tags:", 
      "    print tag['src']"
     ], 
     "language": "python", 
     "outputs": [], 
     "prompt_number": 7
    }, 
    {
     "cell_type": "markdown", 
     "source": [
      " Let's do something a little more complicated:"
     ]
    }, 
    {
     "cell_type": "code", 
     "collapsed": false, 
     "input": [
      "# replace John Doe with your name", 
      "your_name = 'John Doe'", 
      "# the user_agent string is used to make the HTML of the search results consistent", 
      "user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.5 Safari/534.55.3'", 
      "# go get it!", 
      "response = requests.get('http://www.google.com/search', params={'q': your_name}, headers={'User-Agent': user_agent})", 
      "# parse the results", 
      "soup = bs4.BeautifulSoup(response.text)", 
      "# select the links", 
      "tags = soup.select('.r a')", 
      "for tag in tags:", 
      "    # test to make sure it is a valid link", 
      "    if tag['href'].startswith('http'):", 
      "        print tag['href']"
     ], 
     "language": "python", 
     "outputs": [], 
     "prompt_number": 9
    }, 
    {
     "cell_type": "markdown", 
     "source": [
      "The above code queries google for your name and uses beautiful soup to extract the links.", 
      "", 
      "For more information on the Beautiful Soup library visit: <a href=\"http://www.crummy.com/software/BeautifulSoup/bs4/doc/\" target=\"_blank\">http://www.crummy.com/software/BeautifulSoup/bs4/doc/</a>"
     ]
    }, 
    {
     "cell_type": "markdown", 
     "source": [
      "## Exercise", 
      "", 
      "Use Google Finance to retrieve the stock price for Facebook.", 
      "", 
      "Hint: this is the URL you should use: http://finance.google.com/finance?q=FB", 
      "", 
      "Extra Credit:", 
      "", 
      "Generalize this process into a function and using a loop, retrieve the stock", 
      "prices for the folloing stock symbols:", 
      "", 
      "    IBM", 
      "    MSFT", 
      "    AAPL", 
      "    GOOG", 
      "    ORCL"
     ]
    }
   ]
  }
 ]
}