- We need you to build us a crawling tool from scratch which follow these requirements:
--- Requirements ---
- The tool should be used Scapy Python and MongoDB.
- It should dockerized into docker-compose separated into two containers: web and db.
- It should be easier to extend if we gonna crawl the new site.
Such as we will have folderA will crawl site A, folderB will crawl site B...etc
- It should be easier for configure the database and the site we gonna crawling.
- When the structure of codebase is done, we will review it and you need to send us a documentation to show:
+ How can we setup crawler for new site ?
+ Which file/folder need to be created
+ Which command need to be executed to crawl all the sites which were configured in the codebase.
- We will send you an example site, so you will do an example for us when the structure of codebase is done.
- I will send you invitation to github repository then you will work on it.
---- Note ---
- Please read it carefully before you ask us or bid with the right price.
- Only release payment if the codebase is good enough and easy for us to crawling the site.
- Please mark prefix 'DDPythonMongo' when you bid. Otherwise, I'll skip the bid due to automatically bid or bot.
27 freelancers are bidding on average $172 for this job
Hello, I have majored in web scraping by using scrapy, selenium, python, etc. Also, I am familiar with MongoDB and Docker. I can help you to build your crawling tool perfectly. Ready to start. Thanks.
Hi, nice to meet you. I am senior scraping expert. I am interesting in your project. If you hire me, I will give you satisfaction. Please contact me and discuss more details. Thank you for reading my proposal. Regards.