Cloudfeaster is a web crawling and business process automation infrastructure that leverages modern software engineering tools, services and trends to create:
- (this repo is) a spider authoring and maintenance environment
- a service infrastructure for discovering and running spiders via a RESTful API
This is the story behind Cloudfeaster.
- this describes how to author spiders using Cloudfeaster
- see these instructions describe how to setup your development environment and start contributing to Cloudfeaster
- take a look at cloudfeaster/samples for some spiders that illustrate the core concepts
- and simonsdave/gaming-spiders is an entire repo of spiders so you can get a sense of the end-to-end