Social Media Aggregation /Crawler

If you have a look at the attached screen shot its from twitter.

If you see the RED area - that information I want to be pulled from twitter and stored into a database, as well as the twitter user's address. I will be able to use a cloud database.

However the challenge is that I want to be able to grab this data from as many twitter users in USA as possible (possibly around 20 million). I understand that this might have to be done via a crawler or using the twitter api and would take a few weeks of solid crawling or have to use a cloud service to do, twitter enable us to to this but limit results so would be slow to get a lot of data.

These resources might help

[url removed, login to view]

[url removed, login to view]

The rest of the API requires OAuth, but not search.

To use the search API you can just make a request against the following URL: [url removed, login to view][keywords]

For example to search for pizza: [url removed, login to view]

You get JSON data back that you can read in any program. If you use PHP, you can use cURL to make the request and json_decode() to convert the result into an object you can iterate through in a foreach() loop.

[url removed, login to view]

The issue is that they have certain limits

[url removed, login to view]

and so would have to make this distributed somehow to get it done - over a long time frame (maybe would take 10+ computers a month?)

If you think this would interest you please let me know! I am also interested in using the facebook, tublr, instagram and maybe linkedln API's to build up a large dataset of users that have specific jobs!

I understand this is a large data warehousing project and would also need a fast search to retrieve data back

Skills: Data Processing, Engineering, PHP, Software Architecture, SQL

See more: php social media crawler, social media aggregation, php crawler instagram, usa jobs com, twitter com search, specific media, social service jobs, social service com, search twitter com, q.c. jobs, linkedln, jobs instagram, docs jobs, data warehousing jobs, data challenge, database aggregation example, cloud search api, challenge computers, warehousing jobs, social jobs, php oauth example, media jobs, jobs in the media, jobs in social media, jobs in media

About the Employer:
( 6 reviews ) new york, United States

Project ID: #3990675

16 freelancers are bidding on average $1368 for this job


Hi, We have designed and built websites for various types of businesses very effectively. We work with all of our clients individually to easily coordinate and to keep track of the requirements and scope. We fulfill More

$1500 USD in 30 days
(220 Reviews)

********************************************.. YOUR SEARCH ENDS HERE .. ********************************************** GET IT DONE RIGHT THE FIRST TIME! Hello Sir/Ma'am, I look forward to talking to you further and get More

$1600 USD in 10 days
(34 Reviews)

I have read and review your project requirement. Based on my previous experience I am absolutely suitable for this project as I have already done the different type complex projects very confidently and in well mannere More

$1500 USD in 20 days
(126 Reviews)

Thanks for inviting us to bid on this project. We have a team of highly qualified and creative professionals. Give us a chance to show our talents and we assure you quality.

$1200 USD in 30 days
(70 Reviews)

I can discuss further and deliver the project

$1280 USD in 12 days
(95 Reviews)

Hello Mate, We have 6 years of experience of core development and we have Experienced team

$1500 USD in 20 days
(28 Reviews)

We are an American company that will get it done right, the first time!! Pick us today and we will give you great service and quality. We have done plenty of projects like the one you are looking to get done. Please s More

$1450 USD in 30 days
(3 Reviews)

Hi, I love the detail in your description. It is a challenge to get that many data, but I love challenges. Looking forward to working on this.

$1500 USD in 60 days
(31 Reviews)

Hi mrlondon, I can do this work. I'm ready to start. Thanks & Best Regards

$750 USD in 15 days
(6 Reviews)

Successfully scraped more than 100 [url removed, login to view] check your PMB.

$1400 USD in 30 days
(8 Reviews)

Hello Sir we are having similar experience, Please check PMB for more details, Thanks

$1100 USD in 25 days
(6 Reviews)

Hello, I am expert Web research,data entry, editing and all type of data processing,Ms Word, excel, Mails.I read your job description very carefully.I have a large team with Various Expert [url removed, login to view] average Cost More

$750 USD in 0 days
(2 Reviews)

Hi, Veltrod Software services is a global software consulting company specialized in providing Mobile applications, Social media frameworks and eCommerce solutions. Leveraging best-in-class people, processes, and te More

$1200 USD in 28 days
(5 Reviews)

Hi, Its an interesting and tough challenge. I have experience building large data sets from twitter. I have proposed a framework in the private message. Sandeep

$2500 USD in 30 days
(0 Reviews)

We are interested in your project. Please, my PM for more details.

$1250 USD in 13 days
(0 Reviews)

This looks like a fun crawling/scraping project. I have over a decade of scraping experience.

$1400 USD in 30 days
(0 Reviews)