Web Scraping Needed - JavaScript or [url removed, login to view] Solutions Only


Initial project: You will be given a live web forum with three data variables of interest to obtain: Topic, Absolute URL, and Total Number of Posts.

Use a JavaScript solution like [url removed, login to view] to scrape every single Topic from this forum, including all subforums and paginated history. Forum does have an RSS feed, but you cannot get the Total Number of Posts from it alone.

Each Topic is to be grouped with its Absolute URL and Total Number of Posts. Data should be further tagged with Subforum, of which there are less than 3. You could conceivably do so manually if needed.

The final deliverable of this project is twofold:

1) Provide a file with a list of all forum topics, sorted by Total Number of Posts. A CSV file will suffice for this purpose, but I am happy to let you propose a more creative solution. So long as, at the end of the day, I can sort all of this forum's posts in Descending Order by Total Number of Posts, I will be satisfied.

2) Provide the source code in the form of a Git or Mercurial repository.

The forum itself is quite small, with less than 5,000 topics in total needing to be scraped and sorted.

No content has to be scraped from the forum posts themselves. Your only requirement is to scrape the Topic, the URL, and the Total Number of Posts. That said, if you think you can get the forum posts as well (e.g. the individual threads and posts themselves), it could make you more valuable for future projects.

Once the work is finished, you will instantly have the opportunity for a second project: scrape a blog and sort all blog posts by total number of Blog Comments.

Skills: Javascript, Linux

See more: javascript scrape url, web scraping solutions, web scraping solution, web forum scraping, scraping web content, purpose of forum, purpose of a forum, node csv, live javascript, linux web, js web solutions, js do, javascript projects source code, data web scraping, at solutions, all in web solutions, work solutions, total solutions, javascript less than, if js, url scraping, scrape web, scrape a web

Project ID: #1485972

2 freelancers are bidding on average $150 for this job


I'm a professional JavaScript developer with experience in writing bots in Node.js. It would be customisable, and I would be able to add the ability to scrape posts in the future if you wanted.

$200 USD in 3 days
(0 Reviews)

I can handle this project, can u guive me more specific information like url to scrape. ty

$100 USD in 2 days
(0 Reviews)