Web Crawler and DB Setup

Budget $30 - $250 USD
Bids 9
Average Bid $219

This web crawler will only be used to gather URL and backlink information like the one used by SEOMoz who have over 60 billion URL’s indexed. The results will not be publicly available; they will only be used by us for a reporting suite that is in development

- The crawler needs to be run in a language that will be able to index billions of URL’s.
- The crawler needs to be built in such a way that it will not slow down when the database increases.
- The crawler needs to recognise and remove duplicate URL’s.
- The crawler needs to automatically create and index new links.
- The crawler needs to index where links come from, where links are pointing to, any anchor text that is used and if the link is follow or nofollow.
- The crawler will need to show how many outbound links are on each page.
- All information needs to be stored to an MySQL database.

We are aware that this is something that can be built fairly quickly however our we have our developer working on other projects so are looking to bring someone else in to complete the task.

Before commencing we will need to discuss this project via email or Skype messenger to ensure that all of the boxes are ticked and we are not missing anything that could be vital to the project.

Post a Project Like This

Looking to make some money?

  • Set your budget and the time frame
  • Outline your proposal
  • Get paid for your work

Bids on this Project

  • srinichal Profile Picture


    hyderabad,  India

    Perfection and dedication are the way we deal with projects on freelancer. There have been many clients all over the world that are satisfied with the solutions provided.All types of automation works along with web based solutions are our expertise. Looking to provide solutions in the Mobile markets as well. Some of the automation tools are listed in portfolio and here as well: * AutoBidder tool for Madbid like sites * Automation tools For all emails and social Bookmark sites * Social Bookmarking tools * Facebook Likes Automation * Webbased Versions of tools tailored to custom needs * Desktop apps tailored to customer needs * Proxy tools tailored to your requirements. A dedicated team that can literally deliver any project solution with high expertise. Please don't hesitate to contact us via freelancer. Satisfaction guaranteed solutions delivered all over the world. The Skill Set ranges from C# to PHP, Iphone to Andriod. We can deliver almost any solutions for your needs. We can also take up any type of work based on web design to desktop or mobile in the broad spectrum as we always strive to learn and perfect.

    PHP, Perl, Java, and JSP

  • threadnix Profile Picture


    Iasi,  Romania

    In short, I am very interested in developing new technologies in IT and the development of new technologies in this area. Mostly, I'm interested in programming in Java, C, C++ and Python. I have experience in Web programming by using HTML and PHP languages. Also, databases are interested, having experience in MySQL, Derby and SQLite.

    PHP, C Programming, Java, and XML

  • joeguo Profile Picture


    Jinan,  China

    Over 10 years Java & Python experience,2 year Go experience. Specialized in backend development, performance tuning, and domain data mining. Over 30 Joomla extensions have been developed. Some frameworks & softwares I am using: Spring, Hibernate, Lucene, Freemarker, Flask, Joomla, Linux, MySQL, Nginx, Apache etc.

    PHP, Java, JSP, and XML

  • JavaGuruLV Profile Picture


    Riga,  Latvia

    Professional developer and IT teacher. Working knowlege include: Spring Framework, EJB 2.1/3.0/3.1, Google Guice, JDBC, Hibernate, JPA, GWT, Vaadin, Servlet and JSP, JUnit, Mockito, etc. Passionate about agile development, pair programming and TDD.

    Java, JSP, Website Design, and Graphic Design

  • SigmaVisual Profile Picture


    Taxila Cantt,  Pakistan

    Web, Bots, Crawlers, and Scrapers Development. I have expertise in automation services and I can automate any manual process.

    PHP, Python, SEO, and Data Entry

  • avrshell Profile Picture


    Mohali,  India

    We have over 17+ years of experience in system analysis, design and development in Java/J2EE & Web 2.0 technologies. Key Skills: # Java , J2ee , Servlet , Struts 2, JSP , Velocity, SQL, Hibernate, Spring, Web Services, XML, Ajax, J SON,Web Designing ,HTML, CSS , PHP based CMS ,JAVASCRIPT/j Query etc. # Experience in developing Enterprise Applications on J2EE servers. #Hands on Experience in databases such as MYSQL, Microsoft SQL Server , ORACLE. #C#, SQL Server, W CF, , VB 6, Android, action script, java-script, j query, html, html 5, Spring, struts, hibernate, oracle, DB 2, MySQL, XML, xslt , JMS , TCP IP. #Web hosting, web site design, develop, Photoshop, coral draw, jasper report with any data source like XML or db and many more. #Experience on Java based CMS. #Excellent Technical Communication and Interpersonal Skills.

    PHP, Java, JSP, and Javascript

  • waelfree Profile Picture


    Cairo,  Egypt

    I start PHP developing from 2005 until now. I specialize in: - PHP - Ajax - Javascript - Mysql - JQuery - CSS - HTML - Apache - Linux I build many websites in many fields like: - Contests/prizes websites - Advertising - Company sites - Products/selling sites with Epayment - Life insurance sites

    PHP, C Programming, XML, and Script Install

  • ARUNVGOPAL Profile Picture


    Kochi,  India

    Software Engineer by profession, Logo designer by passion.

    Java, JSP, Javascript, and XML

  • csanuragjain Profile Picture


    Gurgaon,  India

    Both front and back end developer

    PHP, C Programming, Java, and JSP