This web crawler will only be used to gather URL and backlink information like the one used by SEOMoz who have over 60 billion URL’s indexed. The results will not be publicly available; they will only be used by us for a reporting suite that is in development
- The crawler needs to be run in a language that will be able to index billions of URL’s.
- The crawler needs to be built in such a way that it will not slow down when the database increases.
- The crawler needs to recognise and remove duplicate URL’s.
- The crawler needs to automatically create and index new links.
- The crawler needs to index where links come from, where links are pointing to, any anchor text that is used and if the link is follow or nofollow.
- The crawler will need to show how many outbound links are on each page.
- All information needs to be stored to an MySQL database.
We are aware that this is something that can be built fairly quickly however our we have our developer working on other projects so are looking to bring someone else in to complete the task.
Before commencing we will need to discuss this project via email or Skype messenger to ensure that all of the boxes are ticked and we are not missing anything that could be vital to the project.
Looking to make some money?
- Set your budget and the time frame
- Outline your proposal
- Get paid for your work
Bids on this Project
Perfection and dedication are the way we deal with projects on freelancer. There have been many clients all over the world that are satisfied with the solutions provided.All types of automation works along with web based solutions are our expertise. Looking to provide solutions in the Mobile markets as well. Some of the automation tools are listed in portfolio and here as well: * AutoBidder tool for Madbid like sites * Automation tools For all emails and social Bookmark sites * Social Bookmarking tools * Facebook Likes Automation * Webbased Versions of tools tailored to custom needs * Desktop apps tailored to customer needs * Proxy tools tailored to your requirements. A dedicated team that can literally deliver any project solution with high expertise. Please don't hesitate to contact us via freelancer. Satisfaction guaranteed solutions delivered all over the world. The Skill Set ranges from C# to PHP, Iphone to Andriod. We can deliver almost any solutions for your needs. We can also take up any type of work based on web design to desktop or mobile in the broad spectrum as we always strive to learn and perfect.
In short, I am very interested in developing new technologies in IT and the development of new technologies in this area. Mostly, I'm interested in programming in Java, C, C++ and Python. I have experience in Web programming by using HTML and PHP languages. Also, databases are interested, having experience in MySQL, Derby and SQLite.
Over 10 years Java & Python experience,2 year Go experience. Specialized in backend development, performance tuning, and domain data mining. Over 30 Joomla extensions have been developed. Some frameworks & softwares I am using: Spring, Hibernate, Lucene, Freemarker, Flask, Joomla, Linux, MySQL, Nginx, Apache etc.
Professional developer and IT teacher. Working knowlege include: Spring Framework, EJB 2.1/3.0/3.1, Google Guice, JDBC, Hibernate, JPA, GWT, Vaadin, Servlet and JSP, JUnit, Mockito, etc. Passionate about agile development, pair programming and TDD.
Taxila Cantt, Pakistan
Web, Bots, Crawlers, and Scrapers Development. I have expertise in automation services and I can automate any manual process.
Software Engineer by profession, Logo designer by passion.