Social Media Aggregation /Crawler
$750-1500 USD
Paid on delivery
If you have a look at the attached screen shot its from twitter.
If you see the RED area - that information I want to be pulled from twitter and stored into a database, as well as the twitter user's address. I will be able to use a cloud database.
However the challenge is that I want to be able to grab this data from as many twitter users in USA as possible (possibly around 20 million). I understand that this might have to be done via a crawler or using the twitter api and would take a few weeks of solid crawling or have to use a cloud service to do, twitter enable us to to this but limit results so would be slow to get a lot of data.
These resources might help
[url removed, login to view]
[url removed, login to view]
The rest of the API requires OAuth, but not search.
To use the search API you can just make a request against the following URL: [url removed, login to view][keywords]
For example to search for pizza: [url removed, login to view]
You get JSON data back that you can read in any program. If you use PHP, you can use cURL to make the request and json_decode() to convert the result into an object you can iterate through in a foreach() loop.
[url removed, login to view]
The issue is that they have certain limits
[url removed, login to view]
and so would have to make this distributed somehow to get it done - over a long time frame (maybe would take 10+ computers a month?)
If you think this would interest you please let me know! I am also interested in using the facebook, tublr, instagram and maybe linkedln API's to build up a large dataset of users that have specific jobs!
I understand this is a large data warehousing project and would also need a fast search to retrieve data back
Project ID: #3990675
About the project
16 freelancers are bidding on average $1368 for this job
Hi, We have designed and built websites for various types of businesses very effectively. We work with all of our clients individually to easily coordinate and to keep track of the requirements and scope. We fulfill More
********************************************.. YOUR SEARCH ENDS HERE .. ********************************************** GET IT DONE RIGHT THE FIRST TIME! Hello Sir/Ma'am, I look forward to talking to you further and get More
Thanks for inviting us to bid on this project. We have a team of highly qualified and creative professionals. Give us a chance to show our talents and we assure you quality.
Hello Mate, We have 6 years of experience of core development and we have Experienced team
Hi, I love the detail in your description. It is a challenge to get that many data, but I love challenges. Looking forward to working on this.
Hi, Veltrod Software services is a global software consulting company specialized in providing Mobile applications, Social media frameworks and eCommerce solutions. Leveraging best-in-class people, processes, and te More
Hi, Its an interesting and tough challenge. I have experience building large data sets from twitter. I have proposed a framework in the private message. Sandeep
This looks like a fun crawling/scraping project. I have over a decade of scraping experience.
