Python Web Crawler for large websites
Budget $100-250 USD
I am looking for a detailed web crawl of any website.
I am aiming to crawl each page of a website and pick only certain information to finally store in a database (suitable, to be suggested by you).
So, input will be the domain and you need to find a way to compile all the URLs and then collect info as in the excel sheet.
- Tab “Crawled URLs” will list out all the URLs of the sites
- Tab “Internal Links Raw Data” will list out all the specifics of the internal links
Now, for each crawl, you may need to record them under a unique crawl ID. This is the 1st phase of the project. We will expand the scope once we get the data correctly and reliably for large websites.
I can explain the details of the required information in the attached sheet.
To qualify for serious consideration of your proposal, you must provide the following in your bid:
- What Python library/package you will use and why
- What are the challenges you foresee and how you will overcome them? It is extremely important to get details here. This is the chance to show how good a fit you are for this project.
- What is your suggestion for data storage and why?
- What similar project did you do earlier and whether I can check that in action?
Please note without the points above in your bid, it is likely that we will not consider the bid seriously.
12 freelancers are bidding on average $176 for this job
Hello, sir! How are you? I am a web scraping specialist. I have rich experience about web scraping. I've been using bs4, selenium or scrapy... I've ever scrapped dozens of sites at once also. At that time, there were a More
Hello sir, I am a python developer with more than 2 years of experience. I have done many projects in past. I can work on : 1. Web Scraping / Data Science / ML 2. Django 3. APP development 4. C/C++ 5. Wordpress Lets More
Hi, The attachment show some of your requirements. I would like to work on this project, but would like to ask some questions to make things clear. Hereunder my answers to your questions: 1- Python Selenium, Beautiful More
Hi there, I am web scrapping and automation expert with more than 3+ years of experience. I have seen you requirements and according to them i would use beautifulsoup and selenium as library. There might be one probl More
Hey Dear We are 45 Persons team and my deliver some services . 1. React Native Experts and Developer 2. Digital Marketing (social media & management) 3. Designing (photoshop and illustrator) 4. Android Development (ja More
Hello, I am willing and able to help with your web_scraping project. I am a seasoned/experienced python programmer who is specialised in data extraction(selenium, BeautifulSoup, request library etc.), modeling, process More
Hi My name is Mohamed Khaled I'm a Data Analyst I can do this job for you as I/O console application as I've made a similar project in python it's goal is to scrap amazon search results on whatever your input such as " More
Hello there, Hope you are doing well! [login to view URL] is available 24/7 for the zoom call . We have our representative all over the World. The attachment show some of your requirements. I would like to work on this proj More
