You have chosen to sponsor your bid up to a maximum amount of .
I need an expert web scraper to scrape a website for information, put this info into a database and create an Intranet site to parse this information.
I need a feasibility study first before commencement, the right programmer will be needed again in the future for additional work.
This will be a difficult project so you must have proven samples of your work, your code will be examined and tested.
Scrape icecat.biz using a scraper with the following capability:
Phase 1 -
Read an excel sheet of products
Go to the ice cat webpage
Search a product by name, EAN, Manufacturer Number or product code (we will need to be flexible)
Retrieve product page
Grab the following
Put the data into a database
Repeat for each product
Phase 2 - Export
Need ability to map data field
Example we might want Internal memory to map to a different word such as Memory
Allow data to map to additional fields
Example we might want Internal memory to map to Memory and also map to RAM (1+n number of fields)
Need ability to export all product data from database
Binarys to go in a folder based on product name
Text data to go to Excel File (master file)
The scraper must be able to
Overcome all obstacles to retrieve its data, redirection scripts, cookies, leech protection etc
Cross navigate to get pictures from manufacturer websites, pictures must come back full size, uncompressed.
Rotate its queries through proxies specified in a proxy list to avoid any limits set by ip for HTTP requests, with an ability to set how many requests should be sent before changing proxy
The scraper must be bug free and well coded