You have chosen to sponsor your bid up to a maximum amount of .
If you have a look at the attached screen shot its from twitter.
If you see the RED area - that information I want to be pulled from twitter and stored into a database, as well as the twitter user's address. I will be able to use a cloud database.
However the challenge is that I want to be able to grab this data from as many twitter users in USA as possible (possibly around 20 million). I understand that this might have to be done via a crawler or using the twitter api and would take a few weeks of solid crawling or have to use a cloud service to do, twitter enable us to to this but limit results so would be slow to get a lot of data.
These resources might help
The rest of the API requires OAuth, but not search.
To use the search API you can just make a request against the following URL: http://search.twitter.com/search.json?q=[keywords]
For example to search for pizza: http://search.twitter.com/search.json?q=pizza
You get JSON data back that you can read in any program. If you use PHP, you can use cURL to make the request and json_decode() to convert the result into an object you can iterate through in a foreach() loop.
The issue is that they have certain limits
and so would have to make this distributed somehow to get it done - over a long time frame (maybe would take 10+ computers a month?)
If you think this would interest you please let me know! I am also interested in using the facebook, tublr, instagram and maybe linkedln API's to build up a large dataset of users that have specific jobs!
I understand this is a large data warehousing project and would also need a fast search to retrieve data back