we are an on-line business that sells approx 30,000 products via our web site. We require the creation of a pricing crawler / spider which will gather pricing data from [url removed, login to view], [url removed, login to view], [url removed, login to view] & various competitors websites. The program would need to have the ability to automatically increase / decrease our pricing based on pre-defined minimum margins to ensure maximum margins are obtained where possible.
The program would need to be capable of searching one product across many (predetermined) web sites and also have the ability to search as many products as required across each competitors site. We currently offer in excess of approx 30,000 product lines and the range of product will continue to expand dramatically.
We need a crawler that will extract / scrape all required data from pre-defined target sites in their entirety. Generating product data which will be filtered into the database where it will interact with the predefined minimum / maximum margin rules to price us where possible just a few pence cheap than our competitors. The program must be robust and stable.