Online News Comments and User Profile Scraping

Project Description:

I need someone who is experienced in crawling and data collection to help me with the following:

For the past two years, on a certain portion of an online newspaper (say ny times)

1. Crawl into a newspaper website (I will provide which site) and scrape (1) the text of article (2) comments to the article (written by users) for each article.

2. Collect the profile information of each commentor

If we can manage the above, I have follow up projects that I can potentially work with you given we mutually agree on it. I am looking for someone I can work with for a long term if I am happy with the work.

Here is what I need in detail:

1. Go to NY Times

[url removed, login to view]

2. From the most popular list, find the 10 articles that are most viewed for that day:

[url removed, login to view]

3. For each link, collect the data

- Date,

- Author Name,

- Text of the Article itself

- Headline

- Comments for the Article (Commentor Name and Comment Text)

4. Then for each commentor in overall the list, when their names are clickable collect information on the number of previous comments, the date of the earliest comment, number of people following and followed by this person.

[url removed, login to view]

For half the articles, there will not be comments, and for another half, the commentors' links will not be clickable. So the actual end data is likely to be smaller.

Can you conduct this data collection for the past year?

Let me know if you think this is feasible, before I agree.

Thanks.

Freelancer API Perl Python Script Install Social Networking

Project ID: #1374230

About the project