Web Crawler and DB Setup


This web crawler will only be used to gather URL and backlink information like the one used by SEOMoz who have over 60 billion URL’s indexed. The results will not be publicly available; they will only be used by us for a reporting suite that is in development

- The crawler needs to be run in a language that will be able to index billions of URL’s.

- The crawler needs to be built in such a way that it will not slow down when the database increases.

- The crawler needs to recognise and remove duplicate URL’s.

- The crawler needs to automatically create and index new links.

- The crawler needs to index where links come from, where links are pointing to, any anchor text that is used and if the link is follow or nofollow.

- The crawler will need to show how many outbound links are on each page.

- All information needs to be stored to an MySQL database.

We are aware that this is something that can be built fairly quickly however our we have our developer working on other projects so are looking to bring someone else in to complete the task.

Before commencing we will need to discuss this project via email or Skype messenger to ensure that all of the boxes are ticked and we are not missing anything that could be vital to the project.

Skills: AJAX, Java, Javascript, MySQL, PHP

See more: working of web crawler, web development language, web developer language, web crawler developer, web-crawler, task web developer, java language for web development, how to be java web developer, how to be development in web, how is c# used in web development, how c++ can be used in web development, DB Developer, how to create a web crawler, s+db, java messenger, email crawler, database crawler, crawler, duplicate email mysql php, web email url, recognise, mysql remove duplicate, java crawler url database, web crawler java database, php crawler email

Project ID: #2416418

9 freelancers are bidding on average $219 for this job


We can help in your project, please check PMB and our ratings/reviews to get idea of our experience.

$250 USD in 7 days
(260 Reviews)

I can deliver the project

$250 USD in 5 days
(126 Reviews)

Hi, I can do that

$200 USD in 3 days
(97 Reviews)

hi i have already worked on such project contact if interested

$200 USD in 10 days
(25 Reviews)

I can build a such crawler.

$300 USD in 10 days
(11 Reviews)

can show you demo. Its been done using open source tool in java. One of best ever crawler. with regards

$250 USD in 10 days
(13 Reviews)

Hi, I recently developed web crawler with data processing and insertion to MySQL database (check my review for details). I want to help you with your project. Best regards, Viktor

$200 USD in 5 days
(5 Reviews)

Hi, Attaching a screenvideo of my tool. If satisfied kindly get back. regards, Arun

$175 USD in 20 days
(0 Reviews)

Hi, I am interested of your project. Please, check PM.

$150 USD in 10 days
(0 Reviews)