mass-scan: Create framework for massive scanning

[pombredanne]

This is all the preparatory work required to successfully run a massive scan. This includes indexing all major registries and Linux distributions, applying AI/ML to spot issues and generate automated fixes and improvements, optimizing the scanning process by eliminating duplicates, and provisioning cloud resources.
This means working on these sub issues:

• mass-scan: Design and create code to spot incorrect licenses scancode.io#1497
• mass-scan: Design and create code to spot incorrect copyrights scancode.io#1498
• mass-scan: Create seed list of projects to scan across the Software Heritage archive purldb#565
• mass-scan: Create PurlDB index of all major package registries and Linux distributions purldb#566
• mass-scan: Design and create code to spot near duplicated projects to avoid scanning duplicates, and integrate in scanning loop purldb#567
• mass-scan: Refactor PurlDB to efficiently process 100+ million project scans using partitioned databases purldb#568
• mass-scan: Create library to collect a whole project version from Software Heritage commoncode#77
• mass-scan: Implement logging and monitoring of PurlDB scans for massive scans purldb#569
• mass-scan: Provision cloud resources: Create estimate for cloud computing requirements #11
• mass-scan: Provision cloud resources: Acquire and/or provision compute and storage to run scans #12