sidmine

WIP pending updates

A series of mining scripts for scraping metadata from URLs right in the Shell

---

Currently this shell script is tested in the Bourne Shell. It uses gawk, curl and unix regex. Runs on bash 3.2 and above.

Run it as a shell script: ./sidemine.sh

Roadmap

• Better error handling
• Option to deal with multiple matches per page
• Parse through an array of URLs from an input source
• Option to extract attributes
• Reference an external file like a sitemap
• Reference a series of nested sitemaps
• Improve memory use and performance by refactoring without pipes
• Accept content with any character encoding