Closed

Python Large File parser using Pandas

We need to parse a large file (20GB-25Gb) what is ';' separated and has from 2-??? columns per row

Message_type is in first column the rest is the message itself nad we need this to be in high performance

We need to generate a new fil for every message_type (24 types) and of that 20Gb of data roughly 3Gb are data with a tag we are interested in.

The logic for the tags needs to be dynamic (if we add a file load_whattoload that needs to be imported dynamically, code for that that is somewhat wokring is provided, but can be changed if needed) We don't wan to have 24 IF IFEL IFEL ELSE statements

Since we are using pandas we the file needs to parsed with pandas, unless there is a really good reason why not to.

Wokring simple code with IF IFEL is provided, with a few tags completed and a small and medium dataset sample aswell

We need this to be done ASAP and needs to be done within 12h after acceptance.

IF you know SQL that is big plus since if you are a dedicated coder, you can keep converting the current SQL based parser functions to Python.

Skills: Python

See more: parse large log file, perl script parse large file, large file parse unix, parse large csv file information need, python binary file parser, php parse large xml file mysql, linux python multi thread process large file, python analyze large file, php parse large xml file, parse large json file java, python text file parser, parse large json file python, php parse large json file, python find string in large file, parse large json file android, parse large json file javascript, python replace string in large file, parse large xml file, python shuffle large file, python split large file into smaller files

About the Employer:
( 4 reviews ) Sundsvall, Sweden

Project ID: #31522138

33 freelancers are bidding on average €509 for this job

(174 Reviews)
7.4
(64 Reviews)
6.6
abhisaini0188

Dear Client Thank you for your project. I've just checked your job description carefully. I'm senior developer with 9+ years of Python,Django and Flask. By using Python, I developed AI engine, BOT, Web Scraping Tools, More

€700 EUR in 15 days
(3 Reviews)
5.0
ExpertSoul

Hello, I've complete MS Software Engineering. I've 5-Year experience in Core Python development. My Core Skills are Machine Learning and AI - Deep Learning - Tensorflow - Pytorch - OpenCV - Deep Learning - Data Analys More

€500 EUR in 5 days
(16 Reviews)
4.7
MashoodurRehman1

PYTHON developer I have read your job description and I am pretty sure that I can complete every bit of your requirements. Further details and cost can be discussed in chat

€250 EUR in 3 days
(36 Reviews)
4.9
ntkhong

⭐⭐⭐⭐⭐ Python Expert ⭐⭐⭐⭐⭐ Dear sir! I am very familiar with data processing using python language. I've checked just your attached project and I am very interested in your project. When do you need the project finished More

€250 EUR in 1 day
(24 Reviews)
5.0
SigmaVisual

Hi, I am interested in this project and I have extensive experience in similar data processing projects. I have 15 years experience and my main area of expertise includes Data Engineering (Small or Big Data level), Da More

€500 EUR in 1 day
(5 Reviews)
4.4
josanta

Hello there! I will use threads to parse large files with pandas, I am also a pro in SQL. Can we talk I will submit it within 12 hours? Thank you!

€550 EUR in 1 day
(21 Reviews)
4.0
normanburtonfree

Dear client, I hope you are doing well As a professional developer, I have many years experiences of python and SQL I will complete the project within your time -12 hours prfelctly until now I have developed many pytho More

€500 EUR in 1 day
(9 Reviews)
4.0
DuskoDjurin

Hello Python is my main skill. I have rich experience in Python Script Development. Let's discuss the project in detail. Thanks Dusko

€500 EUR in 7 days
(1 Review)
3.8
vitaliydudar7

Hello? How are you? Thanks for posting this job. I am a Senior Python developer with 7+ years. I have been doing this kind of python projects many times. Please let me know. Best Regards

€250 EUR in 3 days
(6 Reviews)
3.5
alamineee

Hi, Dear Employer, I am Al.A. I am a Ph.D. backgrounder professional Research Writer. In my University life, I worked for much research. Every semester needs to research writing to pass the semester. In that course, I More

€300 EUR in 4 days
(9 Reviews)
3.4
dahani9091

Hello! I'm a pandas expert! All my scrapping project has a pandas feature on it. I can help you finish this task , with the best performance. Contact me. Thanks.

€500 EUR in 7 days
(7 Reviews)
3.3
sku5551ed6fd9643

Hi there! You want to build an Tour Industry application that will incorporate a fleet management system, travel booking, accommodation booking, generate itinerary for tour packages with a gateway payment system.I am c More

€500 EUR in 7 days
(1 Review)
3.0
vkavtiashvili

Hi there. No problem. I will do my best for you to provide High Quality, Fast Speed, Full Time. I'm a senior Node and Python developer with rich experience of python and MySQL. While reading your requirement, I am sure More

€500 EUR in 7 days
(2 Reviews)
3.1
l060448

Hi, I have 6+ years of experience in application development. I can parse the doc as you need with Pandas but 12 hr time is very short. At least should be 3 days.

€600 EUR in 3 days
(1 Review)
2.7
ivanovatale

Nice to meet you. I would like to work on python project such as desktop, flask, django. Already experienced in many modules such as selenium, numpy, flaks, django, tensoflow, pyqt, tkinter etc. So can help you in anyw More

€500 EUR in 1 day
(2 Reviews)
2.8
lechaantonov333

Dear Client! I have read your project description. I'm glad to say I can help you. I have a rich experiences for projects like yours. If you hire me, then I will do my best and finish it in time. We can discuss in more More

€500 EUR in 7 days
(3 Reviews)
2.3
Pankaj9810

Hey there ! Good day. I am Pankaj Ranga I would like to work with your project and I can surely provide the details you have specified on the project description. I can confidently say that I am knowledgeable enough to More

€500 EUR in 7 days
(1 Review)
2.5
spasicobrenok

Hello, I have been did the same project before and so my capabilities and experiences is the best thing for this project. Let's discuss the detail. Regard.

€500 EUR in 7 days
(1 Review)
1.7