Closed

PHP or Ruby discussion board scrapers

This project received 8 bids from talented freelancers with an average bid price of $ USD.

Get free quotes for a project like this
Employer working
Skills Required
Project Budget
N/A
Total Bids
8
Project Description

We're looking for a Ruby or PHP app that will accept the name or id of a message/discussion board on the following sites and download its content into a MySQL database:

1. [url removed, login to view]
2. [url removed, login to view]
3. any Simple Machines Forum

Both sites 1 & 2 use extensive Javascript to generate views of their boards. You must be willing to license your code under GPL or a similar open license. Your code must not adversely affect the host site's performance. We are willing to break the project into milestones or individual projects for each site.

The basic structure of the DB will be (your comments and suggestions welcome):

Sites: id, name, front_page_url, other_metadata
Users: id, name, join_date. running_post_total, other_metadata, site_id
Topics: id, subject, date_opened, running_post_total, other_metadata, site_id
Messages: id, content, date_posted, other_metadata, user_id, parent_message_id, topic_id, site_id

Summary of relationships -
users:sites 1:1 (may turn out to be 1:many, for now assume 1:1),
topics:sites 1:1,
messages:sites 1:1,
messages:messages 1:many (replies),
users:topics 1:many,
users:messages 1:many,
topics:messages 1:many.

May not need to store site_id everywhere given those relationships, but DB should allow for fast querying of topics per site, for instance.

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online