Find Jobs
Hire Freelancers

Crawling 50 websites using Established open source crawler

$8-15 USD / hour

Cancelled
Posted over 7 years ago

$8-15 USD / hour

I need to crawl 50 websites using an open source web search engine crawler such as scrapy, opensourceserver. We prefer to use an already established crawler with the ability to store the crawled data on the cloud and being able to access it with JSON API for calls. This project will become an add-on module to our existing project where we will use keyword to search through all the crawled data and extract information that contain those keywords. For example, if we are going to crawl [login to view URL], we want to use the crawled data and search for "breaking news" keyword. Any news, article, information that includes this keyword, whether in the title or body, will be pulled. When bidding, please suggest the following - What open source crawler you will use - What are its advantages - The total amount of TIME required to complete the work - The hourly fee If you want to get a list of the 50 websites that needs crawling, please PM me.
Project ID: 12494404

About the project

30 proposals
Remote project
Active 7 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
30 freelancers are bidding on average $11 USD/hour for this job
User Avatar
I am ready to get started right away.... Can we discuss the project details? My distinction, payment after your complete satisfaction with the resulted task.
$8 USD in 40 days
4.9 (330 reviews)
8.4
8.4
User Avatar
Dear Sir, I'm very much delighted to let you know that i did data scraping with PHP-cURL, PhantomJS, Node.js, Selenium from many sites. I just scraped the data from web site and then wrote the data in mysql database or excel or csv or xml file. I worked on many similar projects, I have big experience in data mining projects. I have written hundreds of web scrapers which scrape millions of pages each day. I'm ready to fulfill your requirement. I can finish this task in short time, with the best quality. I can assure 100% accuracy. Please give me the opportunity to do the work. With Kind Regards, Debdulal Roy Proshanta
$10 USD in 40 days
4.9 (98 reviews)
7.5
7.5
User Avatar
Hello, This is not copy/paste message. I read your requirements. I am interested for this job. I have expertise in Wordpress, Laravel, Magento, AngularJS, Ruby on Rails, Core PHP etc. technologies and can work on task and complete within deadline. I have good experience in PHP/Ruby Programming with JavaScript, jQuery, SQL and I can achieve your task. I can provide you 40 hrs/ week productive work for 1 resource and I have predefined process which can easily understandable by clients. I will directly in your contact and I will also do Chat / Audio call with you daily, so that there will not be any communication gap. Also if you will face any productivity issue, our resource will come during weekend and finish the work. Also I will daily send you progress report and will also available online, so that you can contact him whenever you want. So overall you will not face any communication gap between you and our resource. It will be my pleasure to work with you. You can go through my portfolio. I am new in freelancer but I have more then $30k official work in other freelancing site with 5/5 ratings which I can show you. Please send me message, so that I can converse further about this task. Best Regards, Mehul Javia
$12 USD in 40 days
5.0 (51 reviews)
7.0
7.0
User Avatar
Hello, We have accomplished 90% of the project which is similar of your requirement. All we need 10% customization as per your requirement set and specifications. I want to discuss in personal chat in order to explore your needs, which will yield a clear picture of implementation phase. Prior undertaking project, I want to show demo of the work done previously. Apart from demo, I will be sharing following documentation which will turn your project into Quality and Successful delivery: - Technical Project Proposal - Designs - Flow chart for this Project - Execution plan
$15 USD in 40 days
4.8 (54 reviews)
7.1
7.1
User Avatar
***WE WORK WITH GUARANTEED FIRST-PAGE RANKING ON GOOGLE WITH HUGE TRAFFIC *** Hello, Thanks for the opportunities to take part bidding on your project. I am Sam S. From WebStep Technologies, an India based NASSCOM registered, ISO 9001-2015 certified and Google Adwards partner company. We primary focus on Search Engine Optimization (SEO), Reputation Management, PPC and Social Media Marketing. We have a dedicated team of 23+ professionals to serve you in marketing your company. We are a Genuine SEO/Marketing Firm with 8+ years’ experience. Our SEO Professionals have worked for more than 200 clients. Proposal: WebStep wishes to initiate a proposal by offering our industry leading 90 days guarantee first-page ranking on Google. We develop the project with proper article and directory submission and Hi-pr link building, Social Media Optimisation. Please invite me in chat for detail discussion; I assure you that you must like our work and will give us more jobs in future. We provide weekly reports or according to you that you'll enjoy with us. Looking forward to your reply Thanks & Regards, Sam S. WebStep Technologies Pvt. Ltd.
$10 USD in 40 days
4.9 (17 reviews)
5.9
5.9
User Avatar
Hello, I am very interested. I have done many hundreds of crawler using phpcrawler. Let me know the 50 target sites? The time required will depend on the target site complexity and if it has some kind of protections and the like to get access to that site. Looking forward to working with you. Thanks and all the best, Steve
$15 USD in 40 days
4.8 (33 reviews)
6.1
6.1
User Avatar
Sir, I am well versed in this kind of jobs and can do your project as per requirement. I have over 8 years of experiences and will give reference of my online work portfolio once I heard from you. I am very much able to work on this. ***I am ready to start Waiting to hear from you. with thanks and regards
$12 USD in 40 days
4.7 (59 reviews)
5.9
5.9
User Avatar
dear, i read your requirement very carefully, i have experience in relevant field. so i need more discussion on this project. with regards jasmeet kalsi
$8 USD in 40 days
4.6 (34 reviews)
5.1
5.1
User Avatar
Hello, - What open source crawler you will use BeautifulSoup - What are its advantages Easy to implement and extend as per needs Less time consuming - The total amount of TIME required to complete the work Depends on type of websites to crawl - The hourly fee 10$ If you want to get a list of the 50 websites that needs crawling, please PM me. Would be good
$11 USD in 40 days
5.0 (12 reviews)
4.9
4.9
User Avatar
Hello, We have pool of dedicated team consisting of certified developers capable enough to handle your project exactly according to requirements and within said time frame. As per shared project description we find ourselves fit for project work, as we have developed several projects with approx. similar features and functionalities. Looking forward for small conversation to discuss your complete scope of work and suggest/recommend you complete solution. Thanks
$10 USD in 40 days
4.8 (15 reviews)
3.8
3.8
User Avatar
Hello! I specialize in extracting data from websites. I can take all the available information from virtually any website. I use python and php. I need more accurate information: what to take, where, and what volume. In what format to convey to you. Give me the information and I will do the test job.
$8 USD in 10 days
4.6 (15 reviews)
3.8
3.8
User Avatar
Hi, We have experience of 4 years and having the skills required by you for this project. We are very much flexible in terms of timing and requirement. The thing is if you work with us then we will make sure that its a long relation and the best part is we can start your project any time.
$8 USD in 40 days
5.0 (2 reviews)
3.2
3.2
User Avatar
I have a degree in Computer science and a maters in software engineering. i have worked in the field of programming with Java, C, C#, C++ AND PYTHON. given a chance to complete the assignment i will help you complete it within the time given.
$12 USD in 40 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hello, My name is Irfan, expert in web scrapping, data crawling and data mining, I have 5 years+ experience in data mining, website scraping, screen scraping I have completed 100+ WOULD LOVE TO PROVIDE YOU SCRAPPER/CRAWLER TO svrape site you wish to scrape projects on web scraping -- My key Skills are: Web scraping, Data mining, Data crawling, Ms Office(FULL) JavaScript, Go, PHP, SQL, HTML, Python, Perl, CSS. Platforms: Linux, Amazon Web Services (AWS), Google App Engine, Windows, Mac OS X I have completed more than 100+ projects in Web Scraping. You can test the quality of my work if needed, . Regards Irfan
$8 USD in 4 days
0.0 (0 reviews)
0.0
0.0
User Avatar
I've already worked for crawling data from website, for one of the my customer using node.js. I can also setup demo to you if you interested. Please contact or send message if you interested. Regards
$12 USD in 40 days
0.0 (0 reviews)
0.0
0.0
User Avatar
I have already deep experience working with scapy as a web crawler, i had used scrapy framework for 3 months on a large scale project for telecommunication company.
$8 USD in 30 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hi, I’m an experienced web developer and I am proficient in a number of frameworks, so can use the one which you’ll feel most comfortable with. Your website should aim at demonstrating your brand at its fullest. I will build you a website that works fast, is bag free and looks nice. If you are looking for quality work I’m the right person for you. Please find some of my skills below: -LAMP stack experience (esp. PHP and MySQL), -Strong understanding of MVC design principles and object-oriented programming, -Expertise in MVC frameworks: Laravel, Zend, CI, Cake, Yii2 and etc. -Expertise in HTML5, CSS3, JS, AJAX, Jquery, Bootstrap, React.js JSON, etc. - Proficiency at defining, implementing and integrating APIs (REST, XML-RPC) -Expertise in WordPress-Experience with source control (SVN) and bug tracking systems. Looking forward to hearing from you. Thanks for considering!
$12 USD in 25 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of UNITED STATES
Rockville, United States
4.8
29
Payment method verified
Member since Mar 8, 2005

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.