Project Description:
Need a simple scraper created that will scrape a vbulletin forum.
Actions scraper will complete:
Every 10 minutes: Scrapes a subforum page that lists all threads in that subforum sorted by thread creation date (decending), adds new thread records to a watchlist as they are created.
Every 30 seconds: Scrapes a subforum page that lists all threads in that subforum sorted by last reply (decending), records the time, views, and replies for all threads on the watchlist that appear on that page.
Must include proxy support, rotating through a user entered list
USER INPUTS:
- user entered list of proxies
- user entered URLSs (2) to scrape
Output:
-scraper must keep a log of every time it scrapes, recording the time it scraped the page, also the number of views and replies at that time for each thread on the watchlist.
-export the log in csv format.
goal: to be able to record the rate of new posts per 30 seconds, and rate of new replies per 30 seconds, of a thread that is getting tons of views and replies.
