Website contains approx 5,000 records total. The spider is relatively simple. It would:
1) Reads all listings on this page: [url removed, login to view], then
a. Puts the listings into a SQL table if new
b. Updates the listing information if existing
c. Writes the changed values to a log table
d. Stores and updates the associated manufacturer information
2) The script would run on our site every 24 hours and needs to also contain:
a. An email when complete or if there were errors
b. Included in the email the output of everything that was checked and updated.
12 freelancers are bidding on average $134 for this job
I have worked on scraping different sites like google images. I know the issues that arises. The spider should be built intelligently otherwise it will be the wastage of time and money. I am looking for your response.