Web_Scraping_Review_Lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ca9eb2c0-8dee-418e-a7b0-936b5dbe7d68",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a730c212-0b51-4ac5-a6d1-dec90177be70",
   "metadata": {},
   "source": [
    "# **Web Scraping Lab**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd3c1387-2ca7-4341-8ab0-c65d55f9ad90",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d309f83-03a7-45d6-be1f-67056d2b06e8",
   "metadata": {},
   "source": [
    "After completing this lab you will be able to:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94559e58-df3a-493b-baa9-6859e14e6c95",
   "metadata": {},
   "source": [
    "* Download a webpage using requests module\n",
    "* Scrape all links from a web page\n",
    "* Scrape all image urls from a web page\n",
    "* Scrape data from html tables\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d48c7844-2e37-4e66-b2e6-3b9b4a8d1d64",
   "metadata": {},
   "source": [
    "## Scrape www.ibm.com"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55224938-bc17-4bff-8995-71379891a5f1",
   "metadata": {},
   "source": [
    "Import the required modules and functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "95ef6e48-c948-493e-88b7-f8c4d3a0abef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n!pip install pandas==1.3.3\\n!pip install requests==2.26.0\\n!mamba install bs4==4.10.0 -y\\n!mamba install html5lib==1.1 -y\\n!pip install lxml==4.6.4\\n'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "!pip install pandas==1.3.3\n",
    "!pip install requests==2.26.0\n",
    "!mamba install bs4==4.10.0 -y\n",
    "!mamba install html5lib==1.1 -y\n",
    "!pip install lxml==4.6.4\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dee985d6-fcc9-439d-afd2-6bb002d04b6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup # this module helps in web scrapping.\n",
    "import requests  # this module helps us to download a web page"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53400d4f-9ec9-4a1f-b0df-4e899ff08861",
   "metadata": {},
   "source": [
    "Download the contents of the web page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bdf89b78-9574-4f46-9be3-d0c3b6207034",
   "metadata": {},
   "outputs": [],
   "source": [
    "url=\"http://www.ibm.com\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "797010b1-2ebe-4f46-bb6a-5060548d9287",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the contents of the webpage in text format and store in a variable called data\n",
    "data=requests.get(url).text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3df00aeb-0dbc-476e-9d0d-58a9ea491e46",
   "metadata": {},
   "source": [
    "Create a soup object using the class BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1cc6f04a-3ab1-4069-b124-b18949eaf250",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup=BeautifulSoup(data, \"html5lib\")  # create a soup object using the variable 'data'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41d5065e-d07d-4b46-9143-7d08b768b6e9",
   "metadata": {},
   "source": [
    "Scrape all links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "920eaf31-4200-426b-a2a4-c79274227855",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.ibm.com/thought-leadership/best-of/2022/\n",
      "None\n",
      "#ibm-hp--tech-section\n",
      "https://www.ibm.com/consulting/?lnk=ushpv18intro2\n",
      "https://www.ibm.com/about\n",
      "None\n",
      "https://www.ibm.com/consulting/?lnk=flathl\n",
      "https://www.ibm.com/consulting/strategy/?lnk=flathl\n",
      "https://www.ibm.com/consulting/ibmix?lnk=flathl\n",
      "https://www.ibm.com/consulting/technology/\n",
      "https://www.ibm.com/consulting/operations/?lnk=flathl\n",
      "https://www.ibm.com/strategic-partnerships\n",
      "None\n",
      "None\n",
      "https://www.ibm.com/employment/?lnk=flatitem\n",
      "https://www.ibm.com/impact\n",
      "https://research.ibm.com/\n",
      "https://www.ibm.com/\n"
     ]
    }
   ],
   "source": [
    "for link in soup.find_all('a'): # in html anchor/link is represented by the tag <a>\n",
    "    print(link.get('href'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c49101a1-8b72-4b64-888d-1407c87f935d",
   "metadata": {},
   "source": [
    "Scrape all images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "28052f9d-526c-4125-b962-e029b0694514",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://1.dam.s81c.com/p/0a23e414312bcb6f/08196d0e04260ae5_cropped.jpg.global.sr_16x9.jpg\n",
      "https://1.dam.s81c.com/p/06655c075aa3aa29/CaitOppermann_2019_12_06_IBMGarage_DSC3304.jpg.global.m_16x9.jpg\n",
      "https://1.dam.s81c.com/p/08f951353c2707b8/052022_CaitOppermann_InsideIBM_London_2945_03.jpg.global.sr_16x9.jpg\n",
      "https://1.dam.s81c.com/p/064e0139f5a3aa5e/0500002_Lowell_LI_100119.jpg.global.sr_16x9.jpg\n",
      "https://1.dam.s81c.com/p/0795cae91a25156f/conveyorrobottopview.jpg.global.sr_16x9.jpg\n",
      "https://1.dam.s81c.com/p/06dfa9ccdba4ec79/1f417900-9042-44d1-9c219a854bbb62ea.jpg.global.sr_16x9.jpg\n"
     ]
    }
   ],
   "source": [
    "for link in soup.find_all('img'):  # in html image is represented by the tag <img>\n",
    "    print(link.get('src'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "186113a5-3bbe-46cd-8f40-d1e69a03d534",
   "metadata": {},
   "source": [
    "## Scrape data from html tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f17ed2d1-03a4-4d7c-9362-616c7d5620a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "#The below url contains a html table with data about colors and color codes.\n",
    "url=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/HTMLColorCodes.html\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1652f0d2-dd99-4cae-a05d-97f2fb9b7a08",
   "metadata": {},
   "source": [
    "Before proceeding to scrape a web site, you need to examine the contents, and the way data is organized on the website. Open the above url in your browser and check how many rows and columns are there in the color table.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e4d8f720-551e-473f-a547-2e18d43f2fd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the contents of the webpage in text format and store in a variable called data\n",
    "data=requests.get(url).text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "852a2db0-adff-4a7b-99fe-f955023918c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup=BeautifulSoup(data, \"html5lib\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "52ae70e3-1240-43b9-a7ce-2bbf81ff2494",
   "metadata": {},
   "outputs": [],
   "source": [
    "#find a html table in the web page\n",
    "table=soup.find(\"table\") # in html table is represented by the tag <table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7beaae1f-9afb-4072-85a6-730c6d762f00",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number --Color Name------->Hex Code#RRGGBB\n",
      "1--lightsalmon------->#FFA07A\n",
      "2--salmon------->#FA8072\n",
      "3--darksalmon------->#E9967A\n",
      "4--lightcoral------->#F08080\n",
      "5--coral------->#FF7F50\n",
      "6--tomato------->#FF6347\n",
      "7--orangered------->#FF4500\n",
      "8--gold------->#FFD700\n",
      "9--orange------->#FFA500\n",
      "10--darkorange------->#FF8C00\n",
      "11--lightyellow------->#FFFFE0\n",
      "12--lemonchiffon------->#FFFACD\n",
      "13--papayawhip------->#FFEFD5\n",
      "14--moccasin------->#FFE4B5\n",
      "15--peachpuff------->#FFDAB9\n",
      "16--palegoldenrod------->#EEE8AA\n",
      "17--khaki------->#F0E68C\n",
      "18--darkkhaki------->#BDB76B\n",
      "19--yellow------->#FFFF00\n",
      "20--lawngreen------->#7CFC00\n",
      "21--chartreuse------->#7FFF00\n",
      "22--limegreen------->#32CD32\n",
      "23--lime------->#00FF00\n",
      "24--forestgreen------->#228B22\n",
      "25--green------->#008000\n",
      "26--powderblue------->#B0E0E6\n",
      "27--lightblue------->#ADD8E6\n",
      "28--lightskyblue------->#87CEFA\n",
      "29--skyblue------->#87CEEB\n",
      "30--deepskyblue------->#00BFFF\n",
      "31--lightsteelblue------->#B0C4DE\n",
      "32--dodgerblue------->#1E90FF\n"
     ]
    }
   ],
   "source": [
    "# Get all rows from the table\n",
    "for row in table.find_all('tr'): # in html table row is represented by the tag <tr>\n",
    "    # Get all columns in each row.\n",
    "    cols=row.find_all('td') # in html a column is represented by the tag <td>\n",
    "    color_number=cols[0].getText() # store the value in column 1 as number\n",
    "    color_name=cols[2].getText()  # store the value in column 3 as color_name\n",
    "    color_code=cols[3].getText() # store the value in column 4 as color_code\n",
    "    print(\"{}--{}------->{}\".format(color_number,color_name,color_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a1926bb-87ec-4687-bff4-7359d0e81421",
   "metadata": {},
   "source": [
    "## Authors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15bad826-46b2-4de4-a4e3-ecec8198f53e",
   "metadata": {},
   "source": [
    "Ramesh Sannareddy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ee4cf92-6b26-475c-ace4-320d485024e5",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a81e53b-af67-4d86-a0b9-d405e1d832cb",
   "metadata": {},
   "source": [
    "Rav Ahuja"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b94ee8b-cd42-4a84-b4d5-68268758a073",
   "metadata": {},
   "source": [
    "Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a2f7d11-17f8-475a-b942-784ebde10bca",
   "metadata": {},
   "source": [
    "|  Date (YYYY-MM-DD) |  Version | Changed By  |  Change Description |\n",
    "|---|---|---|---|\n",
    "| 2020-10-17  | 0.1  | Ramesh Sannareddy  |  Created initial version of the lab |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9d2a415-9761-4cf7-b1f8-17719684070b",
   "metadata": {},
   "source": [
    " Copyright &copy; 2020 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}