Lesson Recaps

#Lesson Recaps

Below is a log of the assignments covered and a bulleted recap of what issues were covered in the material. Each section corresponds to a set of videos from Udemy and a set of written instructions from automatetheboringstuff.com

##9/6/16 - Week 1 - Introduction to the course, introduction to GitHub, scoping phase

9/13/16 - Week 2 - Udemy Videos: Section 1: Python Basics, Section 2: Flow Control

####Python Basics

• Explain what you are trying to do, not just what you did
• If you get an error message, specify the point at which the error happens
• Copy and paste the entire error message and your code into a pastebin site like pastebin.com or gist.github.com
• Explain what you have already tried to do to solve your problem
• List the version of Python you're using
• IDLE is an editor.
• Interactive Shell (>>>) & File Editor
• Expressions = Values + Operations
• Int, Float, String
• Type programs into the file editor
• The execution starts at the top and moves down

• comments are ignored by Python

• Functions are mini-programs in your program
• print() displays the value passed to it
• input() lets user type in a value
• len() takes a string value and returns an integer value of the string's length
• int(), str(), and float() convert values' data type

####Flow Control

• Boolean Data Type: True, False
• Comparison Operators: ==, !=, <, >, <=, >=
• == is comparison, = is assignment
• Boolean Operators: and, or, not
• An if statement can be used to conditionally execute code, depending on whether or not the if statement's condition is True or False
• An elif (that is, "else if") statement can follow an if statement. Its block executes if its condition is True and all of the previous conditions have been False.
• An else statement comes at the end. It's block is executed if all of the previous conditions have been False.
• The values, 0, 0.0, and the empty string are considered to be Falsey values. When used in conditions they are considered False. You can always see for yourself which values are Truthy or Falsey by passing them to the bool() function.
• When the execution reaches the end of a while statement's block, it jumps back to the start to re-check the condition.
• Press Ctrl-C to interrupt an infinite loop.
• A break statement causes the execution to immediately leave the loop, without re-checking the condition.
• A continue statement causes the execution to immediately jump back to the start of the loop and re-check the condition.
• for loops will loop a specific number of times.
• The range() function called with one, two, or three arguments
• break and continue can be used in for loops.

##9/20/16 - Week 3 - Udemy Videos: Section 3: Functions, Section 4: Handling Errors with Try/Except, Section 5: Writing a Complete Program: Guess the Number

####Functions

• You can import modules and get access to new functions.
• The modules that come with Python are called the standard library, but you can also install third-party modules using the pip tool.
• the sys.exit() function will immediately quit your program
• The pyperclip third-party module has copy() and paste() functions for reading and writing text to the clipboard.
• Functions are like a mini-program inside your program.
• The main point of functions is to get rid of duplicate code.
• The def statement defines a function.
• The input to functions are arguments. The output is the return value.
• The parameters are the variables in between the function's parentheses in the def statement.
• The return value is specified using the return statement.
• Every function has a return value. If your function doesn't have a return statement, the default value is None.
• Keyword arguments to functions are usually for optional arguments. The print() function has keyword arguments and end sep.
• A scope can be thought of as an area of the source code, and as a container of variables.
• The global scope is code outside of all functions. Variables assigned here are global variables.
• Each function’s code is in it’s own local scope. Variables assigned here are local variables.
• Code in the global scope cannot use any local variables.
• Code in a function’s local scope cannot use variables in any another other function’s local scope.
• If there’s an assignment statement for a variable in a function, that is a local variable.

####Handling Errors with Try/Except

• A divide-by-zero error happens when Python divides a number by zero
• Errors cause the program to crash
• An error that happens inside a try block will cause code in the except block to execute. That code can handle the error or display a message to the user so that the program can keep going.

##9/27/16 - Week 4 - Udemy Videos: Section 6: Lists, Section 7: Dictionaries

####Lists

• A list is a value that contains multiple values.
• The values in a list are also called items.
• You can access items in a list with its integer index
• The indexes start at 0, not 1
• You can also use negative indexes. -1 refers to the last item, -2 refers to the second to last item, and so on.
• You can get multiple items from the list using a slice
• The slice has two indexes. The new list’s items start at the first index and go up to, but doesn’t include, the second index.
• The len() function, concatenation, and replication work the same way with lists that they do with strings.
• You can convert a value into a list by passing it to the list() function.
• For loops technically iterate over the values in a list.
• The range() function returns a list-like value, which can be passed to the list() function if you need an actual list value.
• Variables can swap their values using multiple assignment
• Augmented assignment operators like += are used as shortcuts
• Methods are functions that are “called on” values
• The index() list method returns the index of an item in the list
• The append() list method adds a value to the end of a list
• The insert() list method adds a value anywhere inside a list.
• The remove() list method removes an item, specified by the value, from a list
• The sort() methods reverse = True keyword argument can sort in reverse order.
• Sorting happens in “ASCII-betical” order. To sort normally, pass key=str.lower
• These list methods operate on the list “in place”, rather than returning a new list value.
• Strings can do a lot of the same things lists can do, but strings are immutable
• Immutable values like strings and tuples cannot be modified “in place”
• Mutable values like lists can be modified in place.
• Variables don’t contain lists, they contain references to lists.
• When passing a list argument to a function, you are actually passing a list reference
• Changes made to a list in a function will affect the list outside the function
• the \ line continuation character can be used to stretch Python instructions across multiple lines.

####Dictionaries

• Dictionaries contain key-value pairs. Keys are like a list's indexes.
• Dictionaries are mutable. Variables hold references to dictionary values, not the dictionary value itself.
• Dictionaries are unordered. There is no "first" key-value pair in a dictionary.
• The keys(), values(), and items() methods will return list-like values of a dictionary's keys, values, and both keys and values respectively.
• The get() method can return a default value if a key doesn't exist
• The setdefault() method can set a value if a key doesn't exist
• The pprint module's pprint() "pretty print" function can display a dictionary value cleanly. The pformat() function returns a string value of this output

##10/3/16 - Week 5 - Udemy Videos: Section 8: More About Strings, Section 9: Running Programs from the Command Line

####More About Strings

• Table 6-1 of the automate the boring stuff book has escape characters
• Strings can begin and end with double quotes.
• Escape characters let you put quotes and other characters that are hard to type inside strings
• Raw strings will literally print any backslashes in the string and ignore escape characters.
• Multiline strings begin and end with three quotes and can span multiple lines.
• Indexes, slices, and the in and not in operators all work within strings.
• upper() and lower() return an uppercase or lowercase string.
• isupper(), islower(), is alpha(), isalnum(), isdecimal(), isspace(), istitle(), returns True or False if the string is that respective kind of string
• startswith() and endswith() also return bools.
• ‘,’.join([‘cat’, ‘dog’]) returns a string that combines the strings in a list.
• ‘Hello World’.split() returns a list of strings, split from the string its called on.

####Running Programs from the Command Line

• The shebang line tells your computer that you want to run the script using Python 3.
• On Windows, you can bring up the Run dialog by pressing Win+R
• A batch file can save you a lot of typing by running multiple commands
• The batch files you’ll make will look like this:
  • @py C:\Users\Al\MyPythonScripts\hello.py%*
  • @pause
• You’ll need to add the MyPythonScripts folder to the PATH environment variable first
• Command-line arguments can be read in the sys.argv list.

##10/11/16 - Week 6 - Udemy Videos: Section 10: Regular Expressions

####Regular Expressions

• Regular expressions are mini-language for specifying text patterns. Writing code to do pattern matching without regular expressions is a huge pain
• Regex strings often use \ backslashes (like \d) so they are often raw strings: r’\d\
• Import the re module first
• Call the re.compile() function to create a regex object
• Call the regex object’s search() method to create a match object
• Call the match object’s group() method to get the matched string
• \d is the regex for a numeric digit character
• Groups are created in regex strings with parentheses
• The first set of parentheses is group 1, the second is 2, and so on
• Calling group() or group(0) returns the full matching string, group (1) returns group 1’s matching string, and so on
• Use ( and ) to match literal parentheses in the regex string
• The | pipe can match one of many possible groups
• The ? says the group matches zero or one times
• The * says the group matches zero or more times
• The + says the group matches one or more times
• The curly braces with two numbers matches a minimum and maximum number of times
• Leaving out the first or second number in the curly braces says there is no minimum or maximum
• Greedy matching match the longest string possible, nongryeedy matching match the shortest string possible
• Putting a question mark after the curly braces makes it do a nongryeedy match
• search() returns match objects
• findall() returns a list of strings
• Table 7-1
  • \d = Any numeric digit from 0 to 9.
  • \D = Any character that is not a numeric digit from 0 to 9.
  • \w = Any letter, numeric digit, or the underscore character. (Think of this as matching “word” characters.)
  • \W = Any character that is not a letter, numeric digit, or the underscore character.
  • \s = Any space, tab, or newline character. (Think of this as matching “space” characters.)
  • \S = Any character that is not a space, tab, or newline.
• The regex method findall() is passed a string, and returns all matches in it, not just the first match
• If the regex has 0 or 1 group, findall() returns a list of strings
• If the regex has 2 or more groups, findall() returns a list of tuples of strings
• \d is a shorthand character class that matches digits. \w matches word characters, \s matches whitespace characters
• The uppercase shorthand character classes \D, \W, \S match characters that are not digits, word characters, or spaces
• You can make your own character classes with square brackets: [aeiou]
• A ^ caret makes it a negative character class, matching anything not in the brackets
• ^ means the string must start with the pattern, $ means the string must end with the pattern. Both mean the entire string must match the pattern
• The .dot is a wildcard; it matches anything except new lines.
• Pass re.DOTALL as the second argument to re.compile() to pass the .dot match new lines as well
• Pass re.I as the second argument to re.compile() to make the matching case-insensitive
• The sub() regex method will substitute matches with some other text
• Using \1, \2, and so on will substitute 1, 2, etc. in the regex pattern
• Passing re.VERBOSE lets you add whitespace and comments to the regex string passed to re.compile()
• If you want to pass multiple arguments (re.DOTALL, re.IGNORECASE, re.VERBOSE), combine them with the | bitwise operator.

##10/18/16 - Week 7 - Udemy Videos: Section 11:Files, Section 12: Debugging

####Files

• Files have a name and a path
• The root folder is the lowest folder. It’s C:\ on Windows and / on Linux and Mac
• In a file path, the folders and filenames are separated by backslashed on Windows and forward slashes on Linux and Mac
• Use the os.path.join() function to combine folders with the correct slash
• The current working directory is the folder that any relative paths are relative to
• os.getcwd() will return the current working directory
• Absolute paths begin with the root folder, relative paths do not
• The . folder represents “this folder,” the .. folder represents “the parent folder”
• os.path.abspath() returns an absolute path form of the path passed to it
• os.path.isabs() returns True if the path passed to it is absolute
• os.path.relpath() returns the relative path between two paths passed to it
• os.makedirs() can make folders
• os.path.getsize() returns a file’s size
• os.listdir() returns a list of strings of filenames
• os.path.exists() returns True if the filename passed to it exists
• os.path.isfile() and os.path.isdir() return True if they were passed a filename or file path
• open() will return a file object which has reading and writing-related methods
• Pass ‘r’ (or nothing) to open() the file in read mode, ‘w’ for write mode, ‘a’ for append mode
• Opening a nonexistent filename in write or append mode will create that file.
• Call read() or write() to read the contents of a file or write a string to a file.
• Call readlines() to return a list of strings of the file’s content.
• Call close() when you are done with the file.
• The shelve module can store Python values in a binary module.
• shelve.open() returns a dictionary-like shelf value.
• In the shutil (shell utilities) module, shutil.copy() will copy a file
• shutil.copytree() will copy a folder
• shutil.move() will move a folder or rename it
• os.unlink() will delete a file
• os.rmdir() will delete a folder (but the folder must be empty
• shutil.rmtree() will delete a folder and all its contents
• Deleting can be dangerous so do a dry run first.
• send2trash.send2trash() will send a file or folder to the recycling bin
• must install the send2trash module from pip.exe
• For “Walking a Directory Tree” see video

####Debugging

• You can raise your own exceptions: raise Exception(‘This is the error message.’)
• You can also use assertions: assert condition, ‘Error message’
• Assertions are for detecting programmer error that are not meant to be recovered from
• User errors should raise exceptions
• The logging module lets you display logging messages
• Log messages create a breadcrumb trail of what your program is doing
• After calling basicConfig() to set up logging, call logging.debug() to create a log message
• When done, you can disable the log messages with logging.disable(logging.CRITICAL) up near the top of the program
• The five log levels are: DEBUG, INGO, WARNING, ERROR, and CRITICAL
• You can also log a file instead of the screen with the filename keyword argument to the basicConfig() function
• The debugger is a tool that lets you execute Python code one line at a time and shows you the values in variables
• Open the Debug Control window the Debug > Debugger before running the program.
• The Over button will step over the current line of code and pause on the next one
• The step button will step into a function call
• The Out button will step out of the current function you are in
• The Go button will continue the program until the next breakpoint or end of the program
• The Quit button will immediately terminate the program
• Breakpoints can be set by right-clicking the file editor window and selecting “Set Breakpoint"

##10/18/16 - Week 8 - Udemy Videos: Section 13: Web Scraping, Section 14: Excel, Word, and PDF Documents

####Web Scraping

• Python comes with a webbrowser module, which has a function called open, which you can pass a string to open up a web browser
• The requests module is a third-party module for downloading web pages and files
• requests.get() returns a Response object
• The raise_for_status() Response method will raise an exception if the download failed
• You can save a downloaded file to your hard drive with calls to the iter_content() method
• Web pages are plaintext files formatted as HTML
• HTML can be parsed with the BeautifulSoup module
• BeautifulSoup is imported with the name bs4
• Pass the string with the HTML to the bs4.BeautifulSoup() function to get a Soup object
• The Soup object has a select() method that can be passed a string of the CSS selector for an HTML tag
• You can get a CSS selector string from the browser’s developer tools by right-clicking the element and selecting Copy CSS Path
• The select() method will return a list of matching Element objects
• To import selenium, you need to run: from selenium import web driver
• To open the browser, run: browser = web driver.Firefox()
• To send the browser to a URL, run: browser.get(‘http://inventwithpython.com')
• The browser.find_elements_by_css_selector() method will return a list of WebElement objects
• WebElement objects have a text variable that contains the element’s HTML in a string
• The click() method will click on an element in the browser
• The send_keys() method will type into a specific element in the browser
• The submit() method will simulate clicking on the Submit button for a form
• The browser can also be controlled with these methods: back(), forward(), refresh(), quit()

####Editing Excel, PDF, and Word documents

• The OpenPyXL third-party module handles Excel spreadsheets (.xlsx files).
• openpyxl.load_workbook(filename) returns a Workbook object
• get_sheet_names() and get_sheet_by_name() help get Worksheet objects
• The square brackets in sheet[‘A1’] get Cell objects
• Cell objects have a value member variable with the content of that cell
• The cell() method also returns a Cell object from a sheet
• You can view and modify a sheet’s name with its title member variable
• Changing a cell’s value is done using the square brackets, just like changing a value in a list or dictionary
• Changes you make to the workbook object can be saved with the save() method
• The PyPDF2 module can read and write PDFs
• Opening a PDF is done by calling open() and passing the file object to PdfFileReader()
• A Page object can be obtained with the extractText() method, which can be imperfect
• New PDFs can be made from PdfFileWriter()
• New pages can be appended to a writer object with the addPage() method
• Call the write() method to save its changes
• Python-Docx can read and write .docx Word files
• Open to a Word file with docx.Document()
• Access one of the Paragraph objects from the paragraphs member variable, which is list of paragraph objects.
• Paragraph objects have a text member variable containing the text as a string value.
• Paragraphs are composed of “runs”. The runs member variable of a Paragraph object contains a list of Run objects
• Run objects also have a text member variable
• Run objects have a bold, italic, and underline member variables which can be set to True or False
• Paragraph and run objects have a style member variable that can be set to one of Word’s built-in styles
• Word files can be created by calling add_paragraph() and add_run() to spend text content

11/1/16 - Week 9 - Udemy Videos: Section 15: Email, Section 16: GUI Automation

####Email

• import smtplib
• call smtp function, passing it the domain name of your email provider
  • example:
    • variable = smtplib.SMTP(‘smtp.gmail.com’, 587)
    • variable.ehlo()
    • variable.starttls()
    • variable.login(‘senderEmailAddress.gmail.com’, ‘password')
    • variable.sendmail(‘senderEmailAddress.gmail.com’, ‘recipientEmailAddress’, ‘Subject: So long… \n\nDear Recipient,\n What a long strange trip its been.\n\nSender’)
• Full documentation for IMAP module available at https://impaclient.readthedocs.org
• Full documentation for pyzmail module available at http://www.magiksys.net/pyzmail
• Also available at https://automatetheboringstuff.com/chapter16

##GUI Automation

• Controlling the mouse and keyboard is called GUI automation

• The PyAutoGUI Third-party module has many functions to control the mouse and the keyboard

• pyautogui.size() returns the screen resolution, pyautogui.position() returns the mouse position, both as a tuple of two ints

• pyautogui.moveTo() moves the mouse to an x, y coordinate

• The mouse move is instant, unless you pass an int for the duration keyword argument

• pyautogui.moveRel() moves the mouse relative to its current position

• pyautogui’s click(), doubleClick(), rightClick(), and middleClick() click the mouse buttons

• dragTo() and dragRel() will move the mouse while holding down a mouse button

• If your program gets out of control, quickly move the mouse cursor to the top-left

• There’s more documentation at pyautogui.readthedocs.org

• PyAutoGUI’s virtual key presses will go to the window that currently has focus

• typewrite() can be passed a string of characters to type. It also has an interval keyword argument

• Passing a list of strings to typewrite() lets you use hard-to-type keyboard keys, like ‘shift’ or ‘f1’

• These keyboard key strings are in the pyautogui.KEYBOARD_KEYS list

• pyautogui.hotkey() can be used for keyboard shortcuts, like Ctrl+O

• A screenshot is an image of the screen’s content

• The pyautogui.screenshot() will return an Image object of the screen, or you can pass it a filename to save it to a file.

• locateOnScreen() is passed a sample image file, and returns the coordinates of where it is open on the screen.

• locateCenterOnScreen() will return an (x, y) tuple of where the image is on the screen

• Combining the keyboard/mouse/screenshot functions lets you make awesome stuff! :)