Developing Compilable WebCrawler written in Java



Hello there,

I have a java project which I want to develop for my report. The project is simple WebCrawler with no interface included. I have managed to finish most of the parts. All I want is optimisation and changing some features of the project. The reason I want to hire a person is, I have no time to optimise this project for the following 1-3 months.

What I want is;

1st) The database kept on text file. I want to change it into MySQL database to make it more efficient and easy to understand.

2nd) Threads. I am using threads but I want them to work more efficient than current version. What threads will be doing is, they will be assigning to one job only ( for example; Start crawling: [url removed, login to view], dont pick any other job like [url removed, login to view], [url removed, login to view] until it finishes the [url removed, login to view] ). The trick point is, when the thread is crawling the links in [url removed, login to view], it should check the links in the another database ( pool DB for links ) whether it exists or not. If it exists, it shouldnt add it into database to prevent extra work and prevent repeated data.

3rd) I want this code to run properly. It doesn't work efficient currently there are some stucking points and other stuff. I have been using several testing tools but some internal code should be corrected.

4th) I am using [url removed, login to view] file, update it with the never version. ( Pretty simple.. )



I will be giving the whole source-code to you. You will have plenty of time to understand and achieve the goals stated above. As I am giving the source code to you, you would keep it and use it in the future, so please consider this before offering a price. The codes, other stuff and information will be updated as soon as I hire the professional.

The due date is : May 20th 2014 (±5 days)

Thanks in advance!

Skills: Data Mining, Data Processing, Java, MySQL, Software Development

See more: java project work, java hire testing, java 1.5 features, features of java 1.5, hire java professional, webcrawler, update java code, mysql/java-mysql-job, java thread, geoip, crawling of data, java project using threads, java mysql job, data interface java, crawling java, add text file project java, dat mysql, update code java, java add data, trick version, java simple example project, java advance, add data java, java written, dat database java

Project ID: #5813558

17 freelancers are bidding on average $220 for this job


Dear Client, I can help in your project. We have already experience of working on similar projects. Please see below to get idea of our experience: Amazon/Ebay Bots: More

$147 USD in 3 days
(87 Reviews)

Hello , We have a team of Skilled Java-J2EE professionals with experience up to 8 yrs. You will be able to directly communicate with our technical expert. Our Expertise is J2EE: 1) Frameworks: Struts More

$199 USD in 10 days
(41 Reviews)

Hello, I have 12+ years experience as Java developer. One question, did you developer the complete crawler yourself or is it based on a certain software product? gr, Jacco

$250 USD in 30 days
(27 Reviews)

Hi, For crawling are you currently making use of any prebuilt api. is it allowed to use any api or should we use the hardcoded java for this. Let me know if interested Thanks

$166 USD in 10 days
(23 Reviews)

Hello, I'm a Java, J2EE application developer with over 10 years experience. Check my profile, see how other project owners said about my service. Contact me if you were interested. Thanks, Rick

$526 USD in 40 days
(16 Reviews)

HI, I have done similar projects in the past. I will like to use my experience to serve you better. You can trust my sincere indulgence both in quality and time frame. If you will give me a chance, I will ensure that y More

$555 USD in 40 days
(8 Reviews)

hi I am interested in this programming task.I am an experienced javaProgrammer . I wrote a program to get user's info about card,phone,address,loginDate from paypol.It's mutilthread,user can defined how many threads More

$250 USD in 35 days
(3 Reviews)

Hi, i'm basicly new in Java, but your project sound interesting. I think i can manage the 1,2,4 part easily and i can learn a lot if i could get this project. I have already some work what i have done with the My More

$122 USD in 40 days
(1 Review)

A proposal has not yet been provided

$194 USD in 3 days
(0 Reviews)

Hi Scarlettj, Nice to meet you. I am a senior software engineer of an outsourcing company. I am working on a crawling Internet tool that is used to crawl all meta data of movies, series, episodes and so on. I More

$155 USD in 10 days
(0 Reviews)

This is my first bid on a freelancer project, and we all want the first things to be associated with happy memories, so that we can cherish them. I will make sure i will do anything to get the job done for you. I hav More

$166 USD in 15 days
(0 Reviews)

Hi I have done a similar kind of work in one of my project . For that i have used NoSQL for keeping track of duplicity , Also i have used executor framework for parallel processing of these websites. Let me know if More

$111 USD in 30 days
(0 Reviews)

Hi! I had seen carefully your proposal. Your proposal is very interesting and possible for me. I can help you. I think that we can go long time together. Please send me your code. Thank you.

$150 USD in 5 days
(0 Reviews)

I write clean code with TDD. So this is your chance to get a clean and working solution of your problem.

$277 USD in 7 days
(0 Reviews)

Hi Scarlett, I have been working with Java for the last 4 years. Although multi-threaded application is something I haven't worked on before, but I already have started my hands-on over the same and request this opp More

$200 USD in 10 days
(0 Reviews)

A proposal has not yet been provided

$66 USD in 3 days
(0 Reviews)

I have more than 6.5 Years of Extensive design and development experience using Java/J2EE, Spring, MySql, JPA,Hibernate and various Java derived technology and framework. I have worked on so many multi-threaded java a More

$200 USD in 12 days
(0 Reviews)