For an overview of Scrapy, visit [url removed, login to view]
There are two parts to this project, crawl templates and the scraper itself.
The scraper runs based on the crawl template, pulling either the default spider code, or the Selenium spider code, and using the database stored variables for the start URL, restrictions, etc.
I would like to store the scraped URLs in a database, so that the crawler can skip pages that have already been scraped. Updates and changes to information for pages in the database would be handled separately.