Social Media Aggregation /Crawler

CLOSED
Bids
16
Avg Bid (USD)
$1368
Project Budget (USD)
$750 - $1500

Project Description:
If you have a look at the attached screen shot its from twitter.
If you see the RED area - that information I want to be pulled from twitter and stored into a database, as well as the twitter user's address. I will be able to use a cloud database.
However the challenge is that I want to be able to grab this data from as many twitter users in USA as possible (possibly around 20 million). I understand that this might have to be done via a crawler or using the twitter api and would take a few weeks of solid crawling or have to use a cloud service to do, twitter enable us to to this but limit results so would be slow to get a lot of data.

These resources might help
https://dev.twitter.com/docs/streaming-apis
https://dev.twitter.com/docs/faq#6861
The rest of the API requires OAuth, but not search.
To use the search API you can just make a request against the following URL: http://search.twitter.com/search.json?q=[keywords]
For example to search for pizza: http://search.twitter.com/search.json?q=pizza

You get JSON data back that you can read in any program. If you use PHP, you can use cURL to make the request and json_decode() to convert the result into an object you can iterate through in a foreach() loop.

https://dev.twitter.com/docs/api/1/get/users/search

The issue is that they have certain limits
https://dev.twitter.com/docs/rate-limiting/1.1/limits

and so would have to make this distributed somehow to get it done - over a long time frame (maybe would take 10+ computers a month?)

If you think this would interest you please let me know! I am also interested in using the facebook, tublr, instagram and maybe linkedln API's to build up a large dataset of users that have specific jobs!

I understand this is a large data warehousing project and would also need a fast search to retrieve data back

Skills required:
Data Processing, Engineering, PHP, Software Architecture, SQL
Additional Files: untitled.JPG
About the employer:
Verified
Public Clarification Board
Bids are hidden by the project creator. Log in as the employer to view bids or to bid on this project.
You will not be able to bid on this project if you are not qualified in one of the job categories. To see your qualifications click here.


$ 1500
in 30 days
$ 1600
in 10 days
$ 1500
in 20 days
Hire ithinksolutions
$ 1200
in 30 days
$ 1280
in 12 days
$ 1500
in 20 days
$ 1450
in 30 days
Hire nitrotechie
$ 1500
in 60 days
$ 750
in 15 days
$ 1400
in 30 days