I need a simple crawler that will crawl about 30 torrent sites and store in an sql database:
- Title (of the downloadable file)
- date crawled
- Short description of file (if possible)
This database needs to be refreshed every 3-5 days. And links crawled longer than 3-5 days ago should be removed.
The crawler should be implemented as a webscript so it can run on a webserver with cron. I need just the backend script, I will write the front end myself.
I need a web based crawler that can run efficiently, without overloading my server for instance with sql queries (simple greengeeks account)!
This is part of a larger project, so I have a quite limited budget for this.