Closed

Build me a crawling tool

- We need you to build us a crawling tool from scratch which follow these requirements:

--- Requirements ---

- The tool should be used Scapy Python and MongoDB.

- It should dockerized into docker-compose separated into two containers: web and db.

- It should be easier to extend if we gonna crawl the new site.

Such as we will have folderA will crawl site A, folderB will crawl site B...etc

- It should be easier for configure the database and the site we gonna crawling.

- When the structure of codebase is done, we will review it and you need to send us a documentation to show:

+ How can we setup crawler for new site ?

+ Which file/folder need to be created

+ Which command need to be executed to crawl all the sites which were configured in the codebase.

- We will send you an example site, so you will do an example for us when the structure of codebase is done.

- I will send you invitation to github repository then you will work on it.

---- Note ---

- Please read it carefully before you ask us or bid with the right price.

- Only release payment if the codebase is good enough and easy for us to crawling the site.

- Please mark prefix 'DDPythonMongo' when you bid. Otherwise, I'll skip the bid due to automatically bid or bot.

Skills: Python, Web Scraping, MongoDB, Elasticsearch

See more: read mail php file, read data excel file, build msn chat bot msn messenger, read values txt file html, company will read book audio file, read delete file folder php code, read lines text file, php read complex xml file, read check text file, tool program read hotmail, read skype dbb file, read header flv file, read line text file net, read simple xml file, java read hebrew utf file, read certain part file, bot crawling flash website scraping data xml file, discord bot read text file

About the Employer:
( 2 reviews ) Ho Chi Minh, Vietnam

Project ID: #21358106

27 freelancers are bidding on average $172 for this job

bestit4u

Hello Respect sir I saw your description, I can't stand the excitement of seeing the project you post on... I bet one's shirt on I have the ability to complete your project perfetly because I have experience with your More

$140 USD in 7 days
(89 Reviews)
6.9
susanna2018

DDPythonMongo Hi, Client!! @$@$@$PLEASE CONTACT ME. I CAN DO IT WONDERFULLY.$@$@$@ ##% Your idea is attracted me to your world.%## I am Python expert. i have various skills. @example@ python selenium and beautiful More

$140 USD in 7 days
(47 Reviews)
6.0
schoudhary1553

Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl More

$220 USD in 4 days
(35 Reviews)
6.2
osmanaydin

'DDPythonMongo' Hello sir, Please take a look at my reviews and portfolio. I have completed web/data scraping jobs many times. I am interested in your project as well. I would like to discuss details via pm. I loo More

$400 USD in 5 days
(45 Reviews)
5.5
smsaurabhv

Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON More

$222 USD in 3 days
(62 Reviews)
5.3
songhwa2019

Hi there. Thank you for your posting. My name is YK Sun and I am a cloud computing engineer. While working with several companies, I have been familiar those things such as elasticsearch, kibana, fluentd, etc. I used e More

$300 USD in 2 days
(11 Reviews)
4.8
phantom1222

Hello, How are you? I'm very interested in your job post.I believe my skills would be ideal for your project. I am Python Expert and I have rich experience with Web Scraping and I can help you perfectly. And then I am More

$140 USD in 5 days
(6 Reviews)
4.3
ankurs13

DDPythonMongo. I can create the crawler with python, scrapy and mongo; dockerized with docker-compose, that can be managed from the database (depending on the type and content of sites being crawled and scaled). Please More

$200 USD in 7 days
(13 Reviews)
4.5
freelancerdanie9

DDPythonMongo. Hello I have read your description carefully to get a clear understanding of what you want and what I should achieve. Now I want to say that I am really interested in your task and I can do it successful More

$140 USD in 7 days
(3 Reviews)
3.8
StephanChebyshev

Hi.Dear.. Credit is my motto. I am a freelancer working in 24/7 service. I can scrape anything.I know all the angles of website scraping. I have much experience for 10+ years such as avoiding recaptcha, using multi thr More

$100 USD in 3 days
(5 Reviews)
3.5
nikhil929

Hello "DDPythonMongo'" We are familiar with creating crawlers with python scrapy and selenium as well meanwhile we have experienced with IP rotation,User agent rotat More

$140 USD in 7 days
(4 Reviews)
3.3
akweb2015

Hello, I have majored in web scraping by using scrapy, selenium, python, etc. Also, I am familiar with MongoDB and Docker. I can help you to build your crawling tool perfectly. Ready to start. Thanks.

$200 USD in 3 days
(6 Reviews)
3.5
Mickey102644

DDPythonMongo Dear, employer. OK, NO PROBLEM! I am a Web Scraping expert. After reading over your description this looks like a perfect fit for my skill sets. I estimate to complete your project within the time-line a More

$150 USD in 7 days
(5 Reviews)
3.5
superdev1888

DDPythonMongo Hi I have checked your requirement and understand it well. I am a full stack developer with enough experience and skills in Web. I have best experiences in Web Developing such as PHP, Wordpres, Woocomerce More

$140 USD in 5 days
(2 Reviews)
2.1
Zied130

Hi , I am a mathematician. I use scrapy to crawl websites. I use python scripts to shell websites and xpath tofind textual data. I test the python commands in shell using python jupyter before transposing it to code in More

$100 USD in 7 days
(1 Review)
1.2
codestar0714

Hello Sir! I'm skilled in web scraping with Scrapy framework, and scraping automation with Selenium and beautifulsoup. If you need to get information off the web in an automatic fashion, I can write you a special scrip More

$140 USD in 7 days
(2 Reviews)
0.6
cuixin1997

DDPythonMongo Hello, I have been developing reptiles for 2 years. I am qualified for this job and look forward to working with you.

$140 USD in 7 days
(0 Reviews)
0.0
paymentdeveloper

Hi, Thanks for posting !!! I have 12 years of experience in Python development. I have checked The requirement and understand that as well. I have created Website SIMILAR to Build me a crawling tool. Do you w More

$140 USD in 7 days
(0 Reviews)
0.0
unix515

Hi, nice to meet you. I am senior scraping expert. I am interesting in your project. If you hire me, I will give you satisfaction. Please contact me and discuss more details. Thank you for reading my proposal. Regards.

$200 USD in 7 days
(0 Reviews)
0.0
amitbanerjee2812

Dear Hiring Manager, I read your job description and I am confident that I can exceed your expectations. I am a professional programmer on PHP+MySQL+WordPress. Please have a look at my recent work for WordPress : 1. More

$140 USD in 7 days
(0 Reviews)
0.0