I need a software to extract data from a protected website.
The PROTECTION consists of being able to login only with a certain proxy and a user/password combination.
I have the proxy:port and user/password, there is no need to hack anything!
After succesfully logging in, the user is allowed to lookup the database, containing >100,000 entries, displayed on hundreds of pages. (50 entries/page)
The extractor should be able to reach this step and then extract WHOLE entries from the database.
Please keep in mind that I cannot disclose the private information about the login and website, we have to work out something without using that information!