Auto Crawler System

Purpose

• Scrape OS releases data from web-sites, and detect and send a mail if new release version is update.

System Flow Diagram

Jenkins Pipeline Details

Enviroment Setting

• You need to make virtual Environment and set it in the venv

• Python & pip3 Version

  • Python 3.10.3 , pip 22.0.2

  $ sudo apt-get install python3-scrapy python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
    

• Scrapy Version

  • Scrapy 2.5.1

  $ pip3 install scrapy 

• ChromeDriver

  $ sudo apt-get update

  $ sudo apt-get upgrade

  $ wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

  $ sudo apt install ./google-chrome-stable_current_amd64.deb 

  $ google-chrome --version

  $ wget [google-chrome 버전과일치하는 chromedriver 버전.zip 링크] 

  $ unzip chromedriver_linux64.zip 

  $ cd chromedriver [shell_scripts/]
    
    *If you want to change location of chromedriver, move it and modify [CHROME_DRIVER_PATH] in settings.py

• Selenium

  $ pip3 install scrapy