Python scraping and API expert

I'd like to deploy a server where an app can send a request to one or more web scrapers and actually choose which scraper returns data based on parameters sent in a JSON string.

This is a learning project for me, so I'm looking for someone who is an expert and can help me answer the following design questions:

1) What are the components of this setup that need to be on place on the server?

2) What is the best way to structure communication between the app and the scrapers?

3) What is the best way to store web scraping results?

4) What are the best web technologies to use for the scraper server to ensure continued modularity as the scale of data requested grows?

Attached is a simple diagram of how I'd like the system to work.

In terms of work style, I prefer to use GitHub for collaboration.

Please send a github or bitbucket link of a REST API or web scraper/crawler you have built.

Please not the budget is negotiable.


Skills: Git, Linux, Python, RESTful, Web Scraping

See more: youtube api expert, simple icon communication, google base api expert, python irc api, need google api expert, python provisioning api sample, python provsioning api google example, asp api expert developer, python services api drupal, google python provisioning api change user password, google map api expert, google apps python provisioning api authentication, Google API Expert, python provisioning api examples, api expert india, google maps api expert

About the Employer:
( 0 reviews ) United States

Project ID: #17144824

26 freelancers are bidding on average $543 for this job


Hey! I saw your project, and I think that flask-based solution will work best. 1) On a server, you will have running flask app that will accept calls from outside 2) Communication will be done with http request More

$677 USD in 10 days
(57 Reviews)

Hi. I am a prof. scraper and did a lot of technologies related to that. Few questions I have: 1. So, you will have different scrappers that will perform different type of operations? 2. When you call the scrap More

$700 USD in 10 days
(103 Reviews)
$555 USD in 10 days
(55 Reviews)

Hi, I have experience of working on similar projects, I developed 1000+ scrapers in past in python or php. I can also help to guide you in your learning process. Please let me know about time when you would be avail More

$500 USD in 10 days
(60 Reviews)

Hi,sir I have read your post and I am very interested in your project. Because I am a expert in web scraping. I have expert and smart technic with web scraping (1000 over projects for scraping), automation and bot(5 More

$555 USD in 10 days
(82 Reviews)

Hi there,I am Miljan,Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immediately and finish it within More

$333 USD in 2 days
(75 Reviews)

Hello. I'm web scraping expert with 5+ years experience. For your project we can use scrapy framework. For create connect between application and spiders we can use scrapyd service. Also you can check products from More

$500 USD in 5 days
(109 Reviews)

Hello, I can help with you in your project Python scraping and API expert. I have more than 5 years of experience in Git, Linux, Python, RESTful, Web Scraping. We have worked on several similar projects before! We More

$500 USD in 10 days
(33 Reviews)

Hi charlie348, I can do anything regarding to Git, Linux, Python, RESTful, Web Scraping , in time and accurate , I hope I can help you to proceed your project "Python scraping and API expert" I can do this withi More

$525 USD in 12 days
(8 Reviews)

Hello, Greetings! With a proven track record of successful achievements, I am pleased to present my application for your consideration as a Freelancer. Please have a look at my profile and portfolio to get an idea o More

$555 USD in 10 days
(51 Reviews)

Hello, We can use FLASK and SQLAlchemy on the server-side to manage. Redis is a stand-alone message passing and is a suitable candidate for communication. Scrapy is a powerful framework for scraping. Anyhow some More

$250 USD in 10 days
(8 Reviews)

I am a python developer. I have great experience in web scraping and I am an expert in it. I have all necessary skills to scrape almost any website. I have even scraped sites like google, whatsapp web, instagram which More

$375 USD in 2 days
(28 Reviews)
$444 USD in 10 days
(11 Reviews)

Hey there, Hope you are doing well. We have gone through all the details of your project and after having necessary information to check our caliber, we have come to the result that we can make your project done succe More

$400 USD in 5 days
(14 Reviews)

1) I can build scraping tool with node js or python. So we need to place pip python or node and it's required dependency. 2)Api is best way to communicate with scrapper 3) Data can be store to Mongo db or SQL mongo be More

$666 USD in 10 days
(25 Reviews)

hi, I am a professional web scraper. I have scraped over 200 sites in my 3.5 year experience. please ping me to discuss,. I know much about apis, I have good experience in Python. I will help you within your budget.

$466 USD in 5 days
(15 Reviews)

Hello. I have good skills in Git, Linux, Python, RESTful, Web Scraping, etc. Contact me please. Thanks.........................................................................

$555 USD in 10 days
(14 Reviews)

Hi employer I am an expert in Web development, server configurations, [login to view URL] the project requirements listed i am willing to work on the tasks. Reverts to the queries: #1 An http server that listens to H More

$722 USD in 10 days
(17 Reviews)

Hi, I am an experienced python developer with web scraping experience. I have done this kind of project which you can check at below links: [login to view URL] [login to view URL] More

$555 USD in 10 days
(7 Reviews)
$250 USD in 1 day
(2 Reviews)