I want the framework for a multi site scraper written so I can get the price, description, link and availability (in-stock/out of stock) of a single product that is sold on 200+ sites. Preferably, it would use CSS/jquery selectors to find the text to scrape.
I need to be able to put the links in to the pages to be scraped in myself and also the selectors. I would like the scraper to run on a schedule and output the data in SQL. I need the format in columns, with as many rows as there are pages to be scraped. The columns would be "id, description, price and status". The main thing is that I can select what selectors text from each site goes into what column and that it can be a different selector on every row. I understand manually going through 200+ sites to find on where their price information for 1 product is kept will be a lot of work, but it will be me doing it. I already started it using pjscrape, but I can't get all the data from multiple sites organized.