Closed

Simple Program To Remove Duplicate Domains From Text File

This project was awarded to visu14 for $30 USD.

Get free quotes for a project like this
Employer working
Awarded to:
Skills Required
Project Budget
$30 - $250 USD
Total Bids
15
Project Description

I need someone to write a simple program which can do the following -

Scan a .txt file which contains 1 URL per line. Check to see whether the domain name of any line is duplicate, and if it is, remove all instances of the URL.

For example - the program would check each line for [url removed, login to view] (tld being the domain extension such as .com, .org, .net, .info etc) - if two lines containing [url removed, login to view] exist, then the ENTIRE LINE will be deleted (not just the [url removed, login to view]).

Each URL will consist of more than just the domain and tld, for example URL's may consist of [url removed, login to view] however it's important that if [url removed, login to view] is found to exist on 2 or more lines, then the ENTIRE line is removed, not just the [url removed, login to view] portion.

Quite simply I want a program that will check a text file with a list of URL's, find any duplicate domains, and then remove the ENTIRE URL of any lines containing that domain and tld.

It should be something simple like: "if [url removed, login to view] exist in 2 or more lines then remove all lines containing word1.word2".

Further Examples:

Let's say the .txt file looks like the following -

[url removed, login to view]
[url removed, login to view]
[url removed, login to view]
[url removed, login to view]

This program should remove line 1 & line 2, since the domain and tld are the same. The text file should be left looking like -

[url removed, login to view]
[url removed, login to view]

Notice how [url removed, login to view] and [url removed, login to view] were NOT removed since there was only ONE instance of each domain and tld. If a domain and TLD only exist ONCE, then they should NOT be removed.

If you need further instructions please message me.

Thanks.

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online