Website Crawler to identify phrases in html source code
Project ID:
293178Project Type:
FixedBudget:
$30-$250 USD
Project Description:
I need a crawler to identify phrases in the html of websites, for example "google analytics". There will be about 10 phrases in total, i want this to be an input that i can control. I want to be able to control the depth of the crawl in terms of how many levels "deep" the crawler goes into the website (e.g., home page --> about us --> management would be 3 layers deep). Also, i want to be able to control the total number of pages crawled per site, e.g., cut-off search after 100 pages crawled.Finally, the crawler needs to be able to crawl 20,000 sites in about a week. Therefore, the winner bidder needs to be able to build a "fast" crawler--e.g., utilizing multi-threading etc. Also, i will need to be able to upload the urls of the websites I want to crawl.
Finally, this crawler needs to be completed in a couple days.
This crawler should be straightforward for a skilled programmer.
Skills required:
.NET, PHP, Python
Public Clarification Board
Awarded Bids
All Bids ()
hussainweb
|
hussainweb
From India
Offline
|
General Freelancer Orientation (90%, 98th percentile)
Foundation EUFreelance.com Member
$215
in 2 days
over 3 years ago
Hi,
Please check PMB.
please check pm
Hi,
Highly skilled & Experienced with crawling application. Please see PMB.
Regards,
Shyam
Hi Patrick,
I can develop such crawler as python/django web-service (more time is required) or as standalone python script (lessen time).
Optimistic timeframe is just about two days. Pessimistic estimation (with ... more
Hi Patrick,
I can develop such crawler as python/django web-service (more time is required) or as standalone python script (lessen time).
Optimistic timeframe is just about two days. Pessimistic estimation (with huge amount of risk time) is the whole current week.
I have experience with sites parsing/crawling and a script usually can parse about 50,000 pages per day without multi-threading. Indeed, multi-process/multi-thread crawler will be even faster.
Have a nice day,
Nikolay. less
I'm the skilled developer that you need, regards...
fastnet
|
fastnet
From Bulgaria
Offline
|
Foundation EUFreelance.com Member
$150
in 2 days
over 3 years ago
I already have done very similar script.Please see PMB for more info and demo. Thanks for Your time.
Dear sir,
Thank you very much for giving us opportunity to participate your project. We possess 5 years of Experience in such operation. Please check the PM for more details.
We provide 100% perfect result. We look... more
Dear sir,
Thank you very much for giving us opportunity to participate your project. We possess 5 years of Experience in such operation. Please check the PM for more details.
We provide 100% perfect result. We look forward to hearing from you. Thank you for your consideration.
Thanking you
Wasimul Haque
less
Concoct Information Technology is a global technology solutions provider to enterprises, consumers & technology companies. Concoct follows a support-
centric model to all its services that help its clients leverag... more
Concoct Information Technology is a global technology solutions provider to enterprises, consumers & technology companies. Concoct follows a support-
centric model to all its services that help its clients leverage IT to align business objectives.
less
We have similar project and we can offer you the professional services, the team with over 10+ years working experiences, established in Jul, 2004. please kindly check PMB for details, thank you.