Python Web Crawler for large websites

I am looking for a detailed web crawl of any website.

I am aiming to crawl each page of a website and pick only certain information to finally store in a database (suitable, to be suggested by you).

So, input will be the domain and you need to find a way to compile all the URLs and then collect info as in the excel sheet.

- Tab “Crawled URLs” will list out all the URLs of the sites

- Tab “Internal Links Raw Data” will list out all the specifics of the internal links

Now, for each crawl, you may need to record them under a unique crawl ID. This is the 1st phase of the project. We will expand the scope once we get the data correctly and reliably for large websites.

I can explain the details of the required information in the attached sheet.

To qualify for serious consideration of your proposal, you must provide the following in your bid:

- What Python library/package you will use and why

- What are the challenges you foresee and how you will overcome them? It is extremely important to get details here. This is the chance to show how good a fit you are for this project.

- What is your suggestion for data storage and why?

- What similar project did you do earlier and whether I can check that in action?

Please note without the points above in your bid, it is likely that we will not consider the bid seriously.

Skills: Python, Web Crawling

About the Client:
( 2 reviews ) Kolkata, India

Project ID: #34030972

12 freelancers are bidding on average $176 for this job


Hello, sir! How are you? I am a web scraping specialist. I have rich experience about web scraping. I've been using bs4, selenium or scrapy... I've ever scrapped dozens of sites at once also. At that time, there were a More

$250 USD in 1 day
(28 Reviews)
(14 Reviews)

Hello sir, I am a python developer with more than 2 years of experience. I have done many projects in past. I can work on : 1. Web Scraping / Data Science / ML 2. Django 3. APP development 4. C/C++ 5. Wordpress Lets More

$100 USD in 4 days
(41 Reviews)

Hi, The attachment show some of your requirements. I would like to work on this project, but would like to ask some questions to make things clear. Hereunder my answers to your questions: 1- Python Selenium, Beautiful More

$175 USD in 3 days
(8 Reviews)

Hello, I am interested to work on this project. I plan to use libraries like requests, bs4 and selenium. Requests for making http requests to the page, bs4 for scraping and filtering the site html, selenium for dynam More

$200 USD in 7 days
(19 Reviews)

Hello: After reading in detail the requirements of your project and concluding that they match my areas of knowledge and skills, I would like to introduce myself. My name is Anthony Muñoz and I am the lead engineer More

$208 USD in 7 days
(1 Review)

Hi. I’m experienced Data engineer, I use Python and MySQL/Oracle/Hive databases in my professional life. I’m experienced in Data mining so crawling is not a bug deal for me. I’m doing PhD research which includes webs More

$175 USD in 7 days
(2 Reviews)

Hi there, I am web scrapping and automation expert with more than 3+ years of experience. I have seen you requirements and according to them i would use beautifulsoup and selenium as library. There might be one probl More

$100 USD in 7 days
(1 Review)

Hey Dear We are 45 Persons team and my deliver some services . 1. React Native Experts and Developer 2. Digital Marketing (social media & management) 3. Designing (photoshop and illustrator) 4. Android Development (ja More

$200 USD in 2 days
(0 Reviews)

Hello, I am willing and able to help with your web_scraping project. I am a seasoned/experienced python programmer who is specialised in data extraction(selenium, BeautifulSoup, request library etc.), modeling, process More

$175 USD in 4 days
(0 Reviews)

Hi My name is Mohamed Khaled I'm a Data Analyst I can do this job for you as I/O console application as I've made a similar project in python it's goal is to scrap amazon search results on whatever your input such as " More

$100 USD in 1 day
(0 Reviews)

Hello there, Hope you are doing well! [login to view URL] is available 24/7 for the zoom call . We have our representative all over the World. The attachment show some of your requirements. I would like to work on this proj More

$175 USD in 7 days
(0 Reviews)