We require either software of a reliable script to collect data from a well-known Australian Telephone Directory. The site consists of 2 websites that we need to collect standard telephone directory information from.
In short we want to download all the data monthly in order to update our internal database. The data will most likely take a month to collect slowly.
Known challenges are as follows:
This site is smart and when it detects too many searches from an IP that IP can be blocked. As such it needs to use many proxy’s in a random approach in order to avoid being blocked. It’s also suggested that the script not push too hard and to pause every minute so to not be easily detected and blocked.
As an added smarts the site often changes the layout slightly in order to break dumb spiders that are expecting the format not to change. Your script should use some smarts in order to acquire the required data.
In order to get all the data out of the sites it’s suggested to throw at it random dictionary words on a state by state approach.
If you are interested in this project please send a PM for exact details of the sites. Only experienced programmers will be considered. We will close bidding the minute we reach a deal with a qualified bidder and we would like to see this project start quickly.