Cancelled

Python Web Crawler

I need you to develop some software for me. I would like this software to be developed for Windows using Python.

I need to have a custom web crawler that can capture all the same fields as Screaming Frog SEO Spider (Title, Description, HTTP Status, etc.) but, gives me the flexibility to choose which fields to capture and when.

I also need the bot to export all data to Excel and CSV.

I need to the bot to be able to capture HTML elements on a page using XPATH.

I need the bot to be able to crawl JS heavy pages.

I need the bot to. E able to throttle up/down depending on my preferences.

i need the bot to be able to crawl at least 100M pages without any issues.

Skills: Python, Software Architecture, Windows Desktop

See more: export data excel database java, export data excel internet website form, export data excel asp vbscript, python web crawler source code, python web crawler for keywords, python web crawler under 50 lines, python web crawler in less than 50 lines, python web crawler beautifulsoup, python web crawler from scratch, python web crawler github, python scrapy example, export data excel aspnet open xml, export data excel aspnet, dotnet export data excel, export data excel web form, export data excel perl, horde email export data excel, export data excel flash, export data excel file using jdbc servlet, export data excel dot net

About the Employer:
( 2 reviews ) Simi Valley, United States

Project ID: #18016595

20 freelancers are bidding on average $206 for this job

zekovicm

Hi there,I am Miljan,Python Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immediately and finish it More

$222 USD in 3 days
(17 Reviews)
5.9
etuannv

Hi there, I have 4+ years experience at Web scraping. I think I can do your project. I would approach your project by using Python with Scrapy If you'd like to view my previous work, take a look at my profile ht More

$500 USD in 3 days
(45 Reviews)
5.9
schoudhary1553

I am expert who understands the value of time. I pride myself in my attention to detail. I am very hard working and aim to deliver in less time than quoted. I want to make you, my employer happy without changing my bid More

$250 USD in 4 days
(35 Reviews)
5.7
leemilun

Hello, Dear. Nice to meet you. I've read your post carefully. I'm Python/C++ expert. Please discuss more details on chatting room. Regards. Gao M.

$222 USD in 3 days
(27 Reviews)
5.6
ZhenExpert

Hi, I have done several projects similar with this. Please view my portfolios and contact me to discuss your project. Let me know the details via interview. Thank you. Regards,

$155 USD in 3 days
(8 Reviews)
5.5
kipdev13

Hi, Dear. My name is Fang G. Nice to meet you. I've read your post carefully. I'm Python/Web expert. We can do discuss more details on chatting room. Regards.

$222 USD in 3 days
(44 Reviews)
5.7
shinelancer

how are you? I have read your description carefully. I am a scrap and software expert and have rich experience for such project. I can complete ur project quickly and qualitatively using script. If you want, we can More

$130 USD in 1 day
(23 Reviews)
5.1
stead121

Hi Client, I am so happy to see your serious project. Thanks for taking precious time to looking for my bid. My name is Li G. As you can see in my profile page(https://www.freelancer.com/u/stead121) , I have 100 More

$133 USD in 3 days
(16 Reviews)
5.3
DevStar925

How are you? I read your description. I am C++/C#/Java/Python programming Expert. I can finish your project quickly with best quality. I want to discuss more on chat. Thank you for your time.

$155 USD in 3 days
(16 Reviews)
5.1
sodiqa32

i have gone through your project details and i will be glad to complete this for you as i have the required skills to make this a [login to view URL] expert

$30 USD in 1 day
(14 Reviews)
5.0
a4logic

Hello, I am interested in your project. I am representing the team of professional developers. I believe our team can add value to your project for the following reasons: list the programs and skills of the t More

$155 USD in 3 days
(10 Reviews)
4.6
kroim

Honerable Seniors. I read your whole proposal, and thanks for your post on my good experience. This is very good job for me. So I think I can do this in high quality. My releavant skills are: Python, Software Archit More

$140 USD in 3 days
(3 Reviews)
3.5
crocodile305

Hi.. How are you? I saw your description carefully and i 'm very interesting your project. Owing to my rich experience in website scrapping & python , i can say i can do this perfectly. I have many top skills like More

$155 USD in 3 days
(2 Reviews)
3.6
smsaurabhv

Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON More

$55 USD in 3 days
(4 Reviews)
2.7
ilushawebdev

I have done many similar projects related to web scraping information from different websites. Very interested to work on this project. I am absolutely confident I can finish this work on time and on budget to highest More

$100 USD in 5 days
(3 Reviews)
2.5
james880606

Hello? I have read your job description very carefully and i understood what you want. I have python selenium experienced for 4 years. I want to discuss with you via chat. Thanks you. James.

$155 USD in 3 days
(2 Reviews)
1.8
acamso

Hi, I've done similar work in the past. This is worth more than your budget, hence why I bid more. If you want the project done right, PM me to discuss and we'll get started.

$555 USD in 3 days
(0 Reviews)
0.0
goyalsunny1990

Hello Hiring Manager, Greetings of the day!! I have read your job description, you need an expert Python developer I am a good fit for your task. Before we go ahead I had some queries: ---------------------- More

$388 USD in 3 days
(1 Review)
0.0
Speedy2010

i can delivery a powerful tool for you with 0 issues because i`m sdet engineer in big company like ibm ......................................

$244 USD in 5 days
(0 Reviews)
0.0
akhsolution786

Need an animated video? whiteboard animation? Stunning explainer video? I'll get it done with a proven record of *DELIVERING* stunning unique videos to all of my clients - no single exceptions, just read all of my cl More

$155 USD in 3 days
(0 Reviews)
0.0