Crawl a provided set of websites for email addresses

Budget $250 - $750 USD
Bids 27
Average Bid $403

You will receive a large CSV file (approx [url removed, login to view] rows) of names of professors at American universities. For each professor the URL of the university is listed as well.

Your job will be to write software that can crawl each website and look for pages on which the professor's name appears, and extract email addresses from there. The goal is to obtain one or more email addresses for each professor.

Since it's impossible to determine simply from the name and the URL which email address corresponds to the professor, one potential approach is to retrieve multiple pages on which the name appears and on which at least one email address appears as well (using a regex). Then, rank the email addresses based on how frequently they appear. The address that appears most often is likely to be the correct one. Example:

page 1: John Smith, [url removed, login to view](at)[url removed, login to view]
page 2: John Smith, [url removed, login to view](at)[url removed, login to view]
page 3: John Smith, [url removed, login to view](at)[url removed, login to view]
page 4: John Smith, [url removed, login to view](at)[url removed, login to view]

From this example it is pretty clear that is likely to be the correct address.

The output of your software, provided in CSV or other database-readable format, should contain the professor ID (from the input file) and one or more email addresses, each with a rank. Each row should also contain the URL of the page where the address was found.

Here are a few sample rows from the input file:

ID Name Department InstitutionID InstitutionName State Location URL
1 Obaid, Evelyn Computer Science 881 Obaid, Evelyn CA San Jose, CA [url removed, login to view]
2 Khuri, Sami Computer Science 881 Khuri, Sami CA San Jose, CA [url removed, login to view]
3 Beeson, Michael Computer Science 881 Beeson, Michael CA San Jose, CA [url removed, login to view]
15 Kubelka, Richard Mathematics 881 Kubelka, Richard CA San Jose, CA [url removed, login to view]
18 Lin, Ty Computer Science 881 Lin, Ty CA San Jose, CA [url removed, login to view]
29 Key, Scott Philosophy 145 Key, Scott CA Riverside, CA [url removed, login to view]
45 Lash, Jamie Foundations 1230 Lash, Jamie TX Dallas, TX [url removed, login to view]
47 Swain, John Physics 696 Swain, John MA Boston, MA [url removed, login to view]
48 Signorielli, Nancy Communication 1094 Signorielli, Nancy DE Newark, DE [url removed, login to view]
57 Frederick, Joan English 457 Frederick, Joan VA Harrisonburg, VA [url removed, login to view]

To save you time, one possibility is to query Google using their API for pages that contain the name of each professor and are on the domain provided. Example (this is from the first row above):

Query: "Khuri, Sami site:[url removed, login to view]"
[url removed, login to view]+Sami+site%[url removed, login to view]

As you can see the first result in this case is actually a very good page to collect the email from:
[url removed, login to view]

Generally speaking the first 10-20 results are very likely contain the correct address.

Once again, the deliverable of this project is a text (CSV or TSV) file containing one or more email addresses for each professor, ranked by probability of being correct.

The project must be delivered in at most 1 month.

Post a Project Like This

This project was awarded to

randombits

Matt is a phenomenal developer with excellent skills and relentless dedication to complete the project and deliver the correct results. When our initial approach didn't work he recommended alternative solutions that ended up being perfect for the job. I highly recommend him and will definitely hire him again.
About the Freelancer
randombits Profile Picture

Software Development is my passion and gives me lots of fun. Solving problems in this area is a source of constant satisfaction. I work mostly with open-source systems and software and try to know my technologies inside-out. I enjoy being on the bleeding edge of newest and greatest projects out there, but use what is tested and stable for my everyday solutions. Paying attention to detail is important for me, and this is what you can expect in my work. People I have worked with praise my can-do attitude and delivery on time. You can check out my github account: https://github.com/soquel and my blog: http://tinkeringwithstuff.com/

Looking to make some money?

  • Set your budget and the time frame
  • Outline your proposal
  • Get paid for your work

Bids on this Project

  • regeya Profile Picture

    regeya

    De Soto,  United States

    Veteran of the newspaper business looking to put my experience to work for you. As part of my work in the business I did everything from laying out print ads to proofreading and even some IT work along with Python and Ruby hacking.

    Python, Website Design, Copywriting, and Banner Design

  • goelvivek Profile Picture

    goelvivek

    Delhi,  India

    Do the best.

    PHP, C Programming, Javascript, and XML

  • sandeep25101982 Profile Picture

    sandeep25101982

    New Delhi,  India

    More than 10 yrs. working experience with Enterprise Software Application Development starting with .Net Framework 1.0 and above including Latest .Net 4.5 with C# ASP.Net MVC,WPF,Sharepoint,Exchange, Windows Phone and Android Application Development Highly technical expertise in Windows Server all versions,Linux, Hyper-V Virtualization and Windows Azure Cloud and Amazon Web Services. Have deployed several Cloud Datacenters globally for esteem clients.

    ASP, Visual Basic, .NET, and Graphic Design

  • randombits Profile Picture

    randombits

    Poznan,  Poland

    Software Development is my passion and gives me lots of fun. Solving problems in this area is a source of constant satisfaction. I work mostly with open-source systems and software and try to know my technologies inside-out. I enjoy being on the bleeding edge of newest and greatest projects out there, but use what is tested and stable for my everyday solutions. Paying attention to detail is important for me, and this is what you can expect in my work. People I have worked with praise my can-do attitude and delivery on time. You can check out my github account: https://github.com/soquel and my blog: http://tinkeringwithstuff.com/

    PHP, C Programming, Java, and Javascript

  • bhoir Profile Picture

    bhoir

    pune,  Singapore

    Software Developer with more than 10 yrs of experience in developing software in various technologies like C++, .NET,Python,Perl,PHP,Matlab and platforms like windows/unix. Domains: Experience working in different domains like Investment banking, Retail Banking, ATM,CAD and Image Processing, . Programming Languages : C++, VC++, XML, STL, Multithreading, socket programming C#, , Python, Perl, PHP, Matlab and shell scripting. Operating Systems: Solaris, Windows, Linux Configuration Management: Microsoft Visual SourceSafe, Subversion, Clear-Case API’s: AutoCAD API. CAD Tools: AutoCAD Database Technologies: Sybase, Oracle 9i, MS Access, PLSQL

    PHP, Perl, ASP, and C Programming

  • srinichal Profile Picture

    srinichal

    hyderabad,  India

    Perfection and dedication are the way we deal with projects on freelancer. There have been many clients all over the world that are satisfied with the solutions provided.All types of automation works along with web based solutions are our expertise. Looking to provide solutions in the Mobile markets as well. Some of the automation tools are listed in portfolio and here as well: * AutoBidder tool for Madbid like sites * Automation tools For all emails and social Bookmark sites * Social Bookmarking tools * Facebook Likes Automation * Webbased Versions of tools tailored to custom needs * Desktop apps tailored to customer needs * Proxy tools tailored to your requirements. A dedicated team that can literally deliver any project solution with high expertise. Please don't hesitate to contact us via freelancer. Satisfaction guaranteed solutions delivered all over the world. The Skill Set ranges from C# to PHP, Iphone to Andriod. We can deliver almost any solutions for your needs. We can also take up any type of work based on web design to desktop or mobile in the broad spectrum as we always strive to learn and perfect.

    PHP, Perl, Java, and JSP

  • freelancerj2ee Profile Picture

    freelancerj2ee

    Guangzhou,  China

    My skill set is as below: J2EE(B/S Web base application): -good experiences in J2EE architecture and familiarity with MVC design pattern. -skills in J2EE multi-tier architecture development with Servlet/JSP, JavaBeans / SessionBean, JNDI, EJB ,JDBC and and web services -Presentation layer, familiar with Spring MVC, Struts 2, Jquery framework -Business layer, familiar with Spring, EJB 3 framework -Persistence layer, familiar with Hibernate, Spring JDBC, Mybatis framework -Integration layer, familiar with JMS, ESB, Webservice technology -Workflow engine, Activiti framework -Application server, Jboss, Tomcat, Glassfish - Experience in SQL Server ,Oracle ,writing SQL of RDBMS -Nosql and clound computing: Hadoop,Hbase. Design area: -OOA,OOD,OOP -Design pattern -UML

    Java, JSP, Javascript, and XML

  • NirvanaWebDev Profile Picture

    NirvanaWebDev

    Fort Worth,  United States

    I believe that both form and function should be raised to highest degree of quality possible in order to combine into above average websites and applications. I specialize in: Python - Scripts, Web Applications, Scrapers HTML(5)/CSS(3)/Javascript - Frontend dev/design PHP/MySQL - Wordpress, Web Applications, Script Customization I also dabble in C/C++.

    Python, Linux, Data Entry, and Facebook Marketing

  • thiswraith Profile Picture

    thiswraith

    Petrosani,  Romania

    determined, hard worker, detail oriented

    C Programming, .NET, Copywriting, and Translation

  • cyptus Profile Picture

    cyptus

    Hammelburg,  Germany

    Over seven years of .net programming skills..

    PHP, C Programming, Java, and Javascript