Scrapy - Python

In Progress


Im looking for someone to build a simple spider using scrapy and python.

All I'm looking for is the spider to crawl multiple sites using the crawl spider option. The spider will look for all the links / email addresses on the site and then store them into an array associated to that site.

Example output will be as follows using the export to CSV option

domain, links, email address

[url removed, login to view], [link1, link2, link3, link4], [email{at}[url removed, login to view],email2{at}[url removed, login to view],email3{at}[url removed, login to view]]

[url removed, login to view], [link1, link2, link3, link4], [email{at}[url removed, login to view],email2{at}[url removed, login to view],email3{at}[url removed, login to view]]

The spider should get the start urls from a external text file and also use these domain names as the only allowed domains to crawl.

The arrays should only store unique variables i.e if email{at}[url removed, login to view] is captured twice it will only store one copy of it in the array.

The spider should allow us to ignore urls containing certain keywords that we can specify somewhere within the script. i.e if we specify it to ignore "blog" it will not crawl [url removed, login to view] or [url removed, login to view]

Finally the script should allow us to set a maximum amount for pages to call for that site. So for example if we set it to 30 it would call a maximum of 30 urls for that site.

Skills: Python, Web Scraping

See more: scrapy python, scraping email addresses from the web, python look for file, web scrapy, scraping python, Python Scraping, scraping scrapy, scrapy script simple, web script python, text file scraping, python copy text file, spider store, copy text python, python domain, store multiple domains, web python script, python option, using file python, python csv text, python script scraping, simple crawl script, python web script, scraping python script, simple python script, python script copy text file

Project ID: #5050953

Awarded to:


Hi. I have done many scraping tasks using scrapy framework. I would be glad to help. Can I have the websites' address to see how they are organized? Thanks

$150 USD in 5 days
(13 Reviews)

16 freelancers are bidding on average $123 for this job


Dear Client, I can help in your project. We have already experience of working on similar projects. Please see below to get idea of our experience: Amazon/Ebay Bots: More

$103 USD in 3 days
(34 Reviews)

Hi.. Expert web scraper/Data Minor here. Interested in your project. I assure you 100% accurate and good quality work. I have too too scraping experience. let me know if you want sample before awarding. lets start. Reg More

$157 USD in 3 days
(56 Reviews)

Hi Sir, I am ready to work for you.I have 9 years of experience in C/C++ , java python, MySQL. please see some of my works also check my reviews you will get better idea about my skill.I deliver quality work within ti More

$103 USD in 3 days
(21 Reviews)

Dear "mk2021" Hope you are doing well. I have reviewed the project details and would like to offer our services. We have completed many Research/Data collection/Product add/Data mining assignments on More

$116 USD in 3 days
(17 Reviews)

Hello, my name is Max. I'm experienced python developer. My specialization is data mining, scraping and parsing. I have all needed skills for that task. Also I've developed a lot of crawlers, bots and parsers f More

$111 USD in 3 days
(4 Reviews)

i am expert in scraping with python. i can make this simple scraper with simply pthon or with scrapy

$100 USD in 3 days
(3 Reviews)

I am Data Entry ,MS Word and MS Excel Expert. i am very much professional in this work i am pretty sure that you cant find a best person for this job like me so i am ready to work on your project with low rate and high More

$24 USD in 3 days
(1 Review)

A proposal has not yet been provided

$25 USD in 3 days
(0 Reviews)

I am a programmer from India with 4 years of experience in IT/Software and skill set of Python,sql,Pl/sql and oracle database.I am a quick learner and very dedicated to whatever work i have at my hand.

$25 USD in 3 days
(0 Reviews)

I AM CERTAIN THAT I CAN DO THIS!!! I'll start by saying that, since I am sure that is what you really want to know. I have a Master's degree in Computer Science, as well as over four years of professional programming e More

$111 USD in 7 days
(0 Reviews)

A proposal has not yet been provided

$166 USD in 7 days
(0 Reviews)

I have read what you require and understand what you need completely! I am very good at python and web scraping and I can do what you want easily and fast. If you hire me, you will be satisfied. Best regards!

$100 USD in 3 days
(0 Reviews)

For the work you require, it's not possible for your budget range, if you can increase, I can help you out, I'm an expert with Python and the Scrapy framework.

$500 USD in 10 days
(0 Reviews)

I already did a project similar to this.I need to make some changes only.I can provide the script at any time

$90 USD in 3 days
(0 Reviews)

Hello, I have done lots of web scraping projects in Python in the past and would gladly help you with yours. Let me know if you are interested.

$166 USD in 1 day
(0 Reviews)

Dear Sir, MindTech Solution is specialized in internet data mining, list collection, targeted list building, list rental services. We can dig internet and generate targeted leads which is will be based on your require More

$33 USD in 3 days
(0 Reviews)