Closed

Build automated data load from websites (automation) & APIs to Postgresql/MS-SQL

We are a startup focussing on blockchain products.

One of our project details are as below for which we are looking for a team of freelancers with various skill set who can together build the product.

Phase 1 (Duration: 7 days):

IT solution to collect, load and process data from various websites.

Data will be available :

1. Standard APIs of those websites (json format)

2. Websites crawls (by Automation to collect tabular data)

Need to build batch automated processes to collect these data and load in Postgresql/MS-SQL on defined interval of time. This would require setting of PostgresqlMS-SQL server on Ubuntu latest version and further processing to consolidate collected data in desired tables' format. Which requires processing via stored procedures.

Parameters, like, frequency of batch run and hyperlink of the API/Website to crawl should be read from a database table. Solution should be generalised, such that, similar pattern websites are read similarly.

Ex:

1. Standard APIs (samples):

[login to view URL]

[login to view URL]

[login to view URL]

2. Websites Data Crawl (samples):

[login to view URL]

[login to view URL]

[login to view URL]

Once data is collected directly from these websites, it needs to be processed and collated in target summary tables based upon few business logics.

Phase 2 (duration: 4 weeks) & 3 (duration: 3 weeks):

To be discussed once approaching finish of phase 1.

Skills needed for phase 1:

JSON processing, API processing, Automation to crawl websites and collect data, Postgresql/MS-SQL scripting, database knowledge etc.

We want this project to be worked on PostgreSQL (open source) rather MS-SQL.

Skills: API, Data Analytics, JSON, PostgreSQL, Website Design

See more: data collection websites, collect api data cgi script, data processing websites, automated data extraction, automated web login data extraction, data processing feeds automation, perl automated data mine, google kind api data, build web base app data entry form, javascript extract data website load time, capturing data 2008 websites, find truck load websites, data extraction websites, data entry websites home, automated drop shipping websites, convert data md5 hash mssql, automated data mining information websites, build custom weather api, automated data polling websites, automated forex api

About the Employer:
( 0 reviews ) India

Project ID: #17387188

14 freelancers are bidding on average ₹63770 for this job

alwaysanshuman

This is Anshuman from Atventus India Private Limited. We expertise in Blockchain and BitCoin development. We have lots of experience in setting up direct BitCoin blockchain transactions with all the wallet. features. More

₹150000 INR in 20 days
(17 Reviews)
6.6
mmadi

Dear ukrblockchains, I have experience with these skills. I do have similar experience doing something similar to yours "Build automated data load from websites (automation) & APIs to Postgresql/MS-SQL" therefore I More

₹52500 INR in 24 days
(11 Reviews)
6.2
WebnAppExperts

Hi I have gone through your project and I fulfil your all requirements.I have a expertise in MS-SQL and Web development & App development. Have a look on some similar project which you needed- WordPress: www. More

₹62500 INR in 20 days
(4 Reviews)
4.3
₹62500 INR in 20 days
(20 Reviews)
5.0
₹62500 INR in 20 days
(2 Reviews)
2.3
swappitweb

Dear Sir/Madam, Hope you are doing great, We thoroughly went through your requirement and assure you that we can deliver the best version of your it that’s too on time and at competitive price. Let’s discuss w More

₹62500 INR in 20 days
(3 Reviews)
1.0
akashbais74

Looking for Mobile App Solutions, we've got you covered. Contact us today, our developers are eager to solve your problems.. We specialize in Hybrid and Native Mobile Apps. Our prices are affordable. Contact us today.

₹62500 INR in 20 days
(0 Reviews)
0.0
amar47f

Hi, this is amar I am a Team Leader of experienced web & mobile developers with a Bachelor of software from the Pacific National University(PNU) and more than 8+ years of experience in Developing. We have a solid ex More

₹41666 INR in 3 days
(0 Reviews)
0.0
jitendrabaraiya

I can do your job.

₹66666 INR in 20 days
(0 Reviews)
0.0
redincorpindia

Dear client I saw your requirement regarding the automated data load application from website & API for crypto currency information and data. I can develop this application with in a day maximum. Because I developed More

₹37500 INR in 7 days
(2 Reviews)
0.0
nnesarikar

Hello! This appears to be similar to work I did to build data marts hence loading data and writing stored procedures is not an issue

₹62500 INR in 20 days
(0 Reviews)
0.0
freelancerqa007

Hello There, I have more than 6 years experience manual testing and automation testing both. Linkedin : [login to view URL] My skype: [login to view URL] My Website : http://perfectq More

₹62500 INR in 20 days
(0 Reviews)
0.0
sarkaramal

Hi, I have work experience related to Web Crawling, Data Extraction. Currently Working as Data Science. Have knowledge of Postgress, Distributed System.

₹62500 INR in 24 days
(0 Reviews)
0.0
mpate99

I am a junior data scientist and a software engineer. I can do this within a week. .I am working on such a project currently. So it won't take me long to produce the results.

₹44444 INR in 20 days
(0 Reviews)
0.0