Web Crawler and DB Setup

This project received 9 bids from talented freelancers with an average bid price of $219 USD.

Get free quotes for a project like this
Project Budget
$30 - $250 USD
Total Bids
Project Description

This web crawler will only be used to gather URL and backlink information like the one used by SEOMoz who have over 60 billion URL’s indexed. The results will not be publicly available; they will only be used by us for a reporting suite that is in development

- The crawler needs to be run in a language that will be able to index billions of URL’s.

- The crawler needs to be built in such a way that it will not slow down when the database increases.

- The crawler needs to recognise and remove duplicate URL’s.

- The crawler needs to automatically create and index new links.

- The crawler needs to index where links come from, where links are pointing to, any anchor text that is used and if the link is follow or nofollow.

- The crawler will need to show how many outbound links are on each page.

- All information needs to be stored to an MySQL database.

We are aware that this is something that can be built fairly quickly however our we have our developer working on other projects so are looking to bring someone else in to complete the task.

Before commencing we will need to discuss this project via email or Skype messenger to ensure that all of the boxes are ticked and we are not missing anything that could be vital to the project.

Skills Required

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online