Script to webscrape netviz (facebook) and download result

What we want done: We want to gather the activity-data from approximately 1 100 groups on Facebook using the application Netvizz. We have 12 different facebook accounts and each account can only get data from those groups it has liked. We will provide you with the facebook accounts, username and login and the groups each account has liked. To gather data from the groups Netvizz requires an ID. We have compiled a list with these ID-numbers. In some cases the ID-numbers are outdated and have been migrated. This will show up in Netvizz, where you will be able to collect the new ID. See the attached video for an example. Why 12 groups? Facebook will lock out users from data gathering when their activity is too high. Using Netvizz with each facebook account you will get the overall activity (comments, likes, shares and so forth on their groups) over time. Start period should be January 2010 up to and including march 2016. To avoid Facebook from locking us out the data scraping should be gathered in 1 month intervals. We want to be identify the overall user activity over time for all the groups combined, as well as be able to single out the activity level for individual groups. The file should be readable in excel (csv). The tabular tsv datafile downloaded from Netvizz needs to be processed as one aggregate file when all the data from that speficic facebook group is downloaded. Each file downloaded for each individual group must be put in a named after the name of the facebook group id. At the end the tsv files are to be aggregated as one file. This tsv file is to be saved in the folder named total-[page-id]-result-[datefrom]-[dateto].tsv. How it is to be done: The webscraping script is to download the output file from netvizz, save the individual files to a folder named after the facebook group id. Once all the files for a group is downloaded all the tsv files for a group are to be aggregated together as one file. Leave all the former files intact and do not delete anything. The method might be using watir-webdriver and iOpus iMacros Scripting, as long as we can reproduce your results locally through simple means and free software. Files produced by Netvizz: Page Data Module This module gets posts (specify either last n or a date range) on a page and creates: A tabular file (tsv) that lists different metrics for each post. A tabular file (tsv) that lists basic stats per day for the period covered by the selected posts. A tabular file (tsv) that contains the text of user comments (users anonymized). A bipartite graph file in gdf format that shows posts, users (anonymized), and connections between the two. A user is connected to a post if she commented or liked it. See video showing an example of hos this is done for one facebook group for one month.