Completed

Create a Python-Based Elasticsearch Input Pipeline

The script will be run against a directory of zipped CSV (tab-delimited) files. The steps of the script are as follows:

For every file:

Unzip the file

Ignore the data labels in the text file. A separate set of labels will be provided (same number of columns, just cleaned up, spaces removed, etc...)

Validate the input file to ensure it contains the appropriate number of columns

For each record contained within each file:

* Generate a UUID that will later be used as the record's unique identifier

* Transform the record into valid JSON

* PUT the json into Elasticsearch using Index API

* Ensure that the PUT is successful and the record was written

* Attempt to resubmit records that fail

* Keep a log of submissions containing UUID, Submit Status (Success/Fail), FIPS/APN, and Date/Time of the submission

* Keep track of the total numbers of successful and failed attempts

Once a file is complete:

- write the log file back to the directory the source files came from

- delete the uncompressed version of the source file

We'll be importing close to 20m records. It would be ideal if we could thread these operations - but we have to be careful to not inundate Elasticsearch. We'd have to make sure it can keep up so submissions do not fail.

Looking for a price of $225 on this initial work.

Skills: Elasticsearch, Python

See more: create div based site, create website based subscriptions, create web based course registration, create community based websites, create lan based system net, create text based mmorpg, create web based massively multiplayer, create text based rpg website, create text based online game, create tick based php mysql game, create magento based website hours, create web based app, create turn based browser game, create php based video page, create vector based die outline, create text based rpg game, create browser based mmorpg free, python based work outsourcing

About the Employer:
( 0 reviews ) Sparta, United States

Project ID: #20079217

Awarded to:

Mickelson

Hi Nice to meet you. I have enough experience in python script. Below the libraries are I used in past project. selenium, pandas, matplotlib, lxml, beautifulsoup, scipy, and other useful libraries. I have written More

$225 USD in 3 days
(41 Reviews)
5.7

5 freelancers are bidding on average $164 for this job

susanna2018

Hi, Sir!! i am a python expert and full-stack developer with full time. -):-):-) i want to help you with my python skills :-):-):-) “””” Specially Data Science , Data Analysis “””” i use sklearn , Tensorfl More

$200 USD in 3 days
(29 Reviews)
5.3
Muhammadsamran

Hello Sir/ Ma’am We are a group of Software Engineers (Programmers) having 10+ years of experience. Expert in JAVA, C, C++ , C# , Python, ANDROID,IOS, MATLAB, IONIC. Done 40+ projects here on FREELANCER.COM. More

$30 USD in 1 day
(1 Review)
2.1
pinesucceed01

Hi there, I am Python developer, having below given skills: Engineering professional with 10 years of experience in Software development. Mastering/Leading in the development of applications/tools using Python for 6 More

$140 USD in 7 days
(0 Reviews)
0.0
OlatWorld

We are a team of developers and python experts. We are dedicated to help you with your python projects. We will be honored to work with you on this project. Kindly message us to discuss better

$225 USD in 3 days
(0 Reviews)
0.0