I'd like to deploy a server where an app can send a request to one or more web scrapers and actually choose which scraper returns data based on parameters sent in a JSON string.
This is a learning project for me, so I'm looking for someone who is an expert and can help me answer the following design questions:
1) What are the components of this setup that need to be on place on the server?
2) What is the best way to structure communication between the app and the scrapers?
3) What is the best way to store web scraping results?
4) What are the best web technologies to use for the scraper server to ensure continued modularity as the scale of data requested grows?
Attached is a simple diagram of how I'd like the system to work.
In terms of work style, I prefer to use GitHub for collaboration.
Please send a github or bitbucket link of a REST API or web scraper/crawler you have built.
Please not the budget is negotiable.
28 freelancers are bidding on average $535 for this job
hi, I am a professional web scraper. I have scraped over 200 sites in my 3.5 year experience. please ping me to discuss,. I know much about apis, I have good experience in Python. I will help you within your budget.
Hello. I have good skills in Git, Linux, Python, RESTful, Web Scraping, etc. Contact me please. Thanks.........................................................................