Back-end scraper for a price comparison website

Closed

Description

1) I have already developed a desktop based application using Scrapy/ Python that is hard coded to crawl to three separate sites (using three "spiders") that can pull out product details such as Product ID, Title, Price, Vendor and Stock Position. At present, these details are used to generate .sql files that need to be uploaded to the web server to update the Product Table in the database.

2) The current requirement is to develop a Server version of the scraper. The expected features are as under:-

a) The Products Table in the server database to be automatically populated by the scraper. The required fields are Product ID, Title, Price, Vendor, Stock Position, Payment Options, Delivery Time

b) Easy extensibility (with some python coding) to add more sites in future.

c) To meet the above, the scraper to be implemented as two modules. The "Scraper Module" and the "Parameter Module".

d) The "Scraper Module" would do the actual scraping of multiple sites (based on parameters read from the Parameters Module), and also automatically populate the Products Table in the database server. For sites with content rendered in JavaScript, Scrapy to be used with Selenium for effective scraping.

e) The "Parameters Module" would include a Form through which scrape parameters such as the primary URL, scraping rules for each field to be scraped, format of data to be extracted, and whether to use simple crawl (for sites without JavaScript) or complex crawl (for sites with content rendered in JavaScript). These parameters would be stored in a table, and accessed by the "Scraper Module" at run time.

f) The scraped URLs (referred by the primary URL) to be saved in a Database Table with "processed flag", so that these can be skipped if scraping needs to be resumed after interruption.

g) Primary URLs also to be saved with the date of last successful scraping, to enable scheduling of periodic repeat scrapings.

h) While executing scraping, only those fields that have changed since last scrape are to be extracted and the original table entry for the product to be "updated", as required. In case of new products, the details to be "inserted" as a new row in the Products Table.

i) Scrapy to be used with Selenium for effective scraping of sites with heavy JavaScript content.

j) Performance must be adequate to enable scraping of the sites in order to generate the Products database

Expected Skills: Web Scraping, Scrapy, Selenium, Python, Data Mining, Javascript, MySQL

Budget: USD 200 to USD 300

Skills: Data Mining, MySQL, Python, Web Scraping

See more: comparison scraper, web scraping price comparison, price scraper, sql coding, website scraper price comparison, web scraper comparison price, price comparison web scraping, price scraping, website price scraper, price scraper sql database, price comparison scraper, end scraping, rendered javascript selenium scraping python, website scraping price comparison, web scraping price, web scraping application, web develop on python, vendor comparison format, sql price, scraping a server, scheduling javascript, requirement to develop website, primary modules, sql server data mining, sql data mining

Project ID: #4223516

Awarded to:

zhoumo

Hi, I have written many previous python scrapers, and I know how to do this job.

$250 USD in 5 days
(6 Reviews)
4.3

7 freelancers are bidding on average $336 for this job

mantislin

Hi sir, please check PM, thx Kimi.

$250 USD in 5 days
(92 Reviews)
6.3
akhila27

Scraping Experts Here. Check the message and contact us. Scraping samples are also attached.

$300 USD in 14 days
(5 Reviews)
5.9
proauthor

Hi, Ready to start your work. Eagerly awaiting for your positive reply. Please check your inbox for further details. Thanks, Shaik.

$250 USD in 5 days
(26 Reviews)
5.1
alexmilano

Hi, you can connect scrapy with django, the other things are really easy, i can do it!

$600 USD in 7 days
(1 Review)
3.0
vmrvision

I'd already done such type of work before for US parks & recreation. Please let me know If I can get the oppertunity further. Thanks Vikas Choudhary

$500 USD in 2 days
(1 Review)
2.2
zyxerdima

work will be successful

$250 USD in 5 days
(2 Reviews)
0.4
nitinrajpal

Best work guaranteed. Please check PM.

$250 USD in 10 days
(0 Reviews)
0.0