I need an experienced developer that is able to scrape data from a list of URLs that I will provide and on a webpage that has some scripts preventing web scraping.
Here are more details about the project:
1.) The list currently has 22.135 URLs.
2.) Each Url is a single web page with 27 data types to be scrapped.
3.) The URL belongs to a domain that has some scripts to prevent bots from scraping the site, so I imagine that we will need some proxies and other techniques to gather all the data.
4.) I will need that data in a CSV/Excel format that I will provide a sample of.
5.) The full amount will be set as a milestone, payable on delivery of the full CSV file.
In your bid, please confirm that you will be able to provide the scraping and the proxy etc. infrastructure needed to accomplish the project as described above.
Please send me your questions if you have any.