Website Spider / Data Collector
I want a program that will spider a selection of YouTube type sites and save the thumbnail, and video page URL, and collect the title and run-time of every video on every sites we choose to spider. This data is to be saved to MySQL.
I want to scan the sites often to keep up with updates, so i only want to collect new URL's. I want an update feature that will scan old URL's and make sure the same video is still there and working.
This is to be a command line script to run on crontab.. System is FreeBSD 6.1