WEBSITE SCRAPER/DATA EXTRACTION CRAWLER MYSQL DATA APPEND
I need a crawler automatically finding required websites via various SE's or pre provided DB's and reading those websites to extract particular data and saving these data to a mysql database, according to tables and rules I specify.
The crawler must run on server, whose configuration I will specify (any further clarification must be asked.
The crawler must run at all times, auto-restarting and keeping the process up, (a second triggering server is available, if needed). Data must be read continuously, at quite a fast pace compared with saved data, and if new data are present they must be appended to database under a format we will agree.
It needs to append address, phone, zip, fax, revenue size, employee size etc which is specified by me.
Pls try to figure out all info you need before starting, and to ask all possible questions