tweep

Tweep is an advanced Twitter scraping tool written in Python that allows for scraping Tweets from Twitter profiles without using Twitter's API.

Tweep utilizes Twitter's search operators to let you scrape Tweets from specific users, scrape Tweets relating to certain topics, hashtags & trends, or sort out sensitive information from Tweets like e-mail and phone numbers. I find this very useful, and you can get really creative with it too.

tl;dr Benefits

Some of the benefits of using Tweep vs Twitter API:

• Can fetch almost all Tweets (Twitter API limits to last 3200 Tweets only)
• Fast initial setup
• Can be used anonymously and without sign up
• No rate limitations

Requirements

• Python 3.5/3.6
• pip3 install -r requirements.txt

Usage

• -u The user's Tweets you want to scrape.
• -s Search for Tweets containing this word or phrase.
• -g Retrieve tweets by geolocation. Format of the argument is lat,lon,range(km or mi).
• -o Save output to a file.
• --year Filter Tweets before the specified year.
• --fruit Display Tweets with "low-hanging-fruit".
• --tweets Display Tweets only.
• --verified Display Tweets only from verified users (Use with -s).
• --users Display users only (Use with -s).
• --csv Write as a .csv file.
• --db Write as a database.
• --hashtags Extract hashtags.
• --userid Search from Twitter user's ID.
• --limit Number of Tweets to pull (Increments of 20).
• --count Display number Tweets scraped at the end of session.
• --stats Show number of replies, retweets, and likes.

Low-Hanging Fruit

The --fruit feature will display Tweets that might contain sensitive info such as:

• Profiles from leaked databases (Myspace or LastFM)
• Email addresses
• Phone numbers
• Keybase.io profiles

Basic Examples and Combos.

A few simple examples to help you understand the basics:

• python3 tweep.py -u username - Scrape all the Tweets from user's timeline.
• python3 tweep.py -u username -s pineapple - Scrape all Tweets from the user's timeline containing pineapple.
• python3 tweep.py -s pineapple - Collect every Tweet containing pineapple from everyone's Tweets.
• python3 tweep.py -u username --year 2014 - Collect Tweets that were tweeted before 2014.
• python3 tweep.py -u username --since 2015-12-20 - Collect Tweets that were tweeted since 2015-12-20.
• python3 tweep.py -u username -o file.txt - Scrape Tweets and save to file.txt.
• python3 tweep.py -u username -o file.csv --csv - Scrape Tweets and save as a csv file.
• python3 tweep.py -u username -o file.db --db - Scrape Tweets and save as a database.
• python3 tweep.py -u username --fruit - Show Tweets with low-hanging fruit.
• python3 tweep.py -s "Donald Trump" --verified --users - List verified users that Tweet about Donald Trump.
• python3 tweep.py -g="48.880048,2.385939,1km" -o file.csv --csv - Scrape Tweets from a radius of 1km around a place in Paris and export them to a csv file.

Example String

955511208597184512 2018-01-22 18:43:19 GMT <now> pineapples are the best fruit

Screenshot

Thanks

Thanks to @hpiedcoq for contributing several features!

Contact

Shout me out on Twitter: @now