What I'm looking for is a way to scrape Google Shopping using the Google API or another way for this.
I need to scrape at least 60,000 SKU's per week, about 8500 a day.
Speed is important as well. Using the Google Shopping API I believe it's must faster than other methods I've seen.
The results I'm looking for would be:
SKU
Store Name
url
upc
Price
It also needs to go under "Compare Prices" buttons and grab those prices as often the best prices are under those buttons.
There will be an input file of about 60,000 upcs so this will be provided.
I'm open to not using the Google API as long as the result is fast and accurate. I'm also open to selecting specific competitors to grab from Google Shopping if that would make things faster and accurate. There are perhaps 5 competitors I care about when comparing prices.
I would prefer if the project was written in perl but Im ok with another language.