Closed

Code that will parse and merge large csv files

I'm working with large csv files up to 50GB in size. Typically one file is around 30GB while the one that will be merged onto it is just about 100 MB. The code I'm currently using will take about 1 hour to parse and merge two files. If we can reduce the time this takes by writing new code that will be great.

Skills: Python, C++ Programming

See more: merge csv files with different columns python, merge two csv files python pandas, combine two csv files python, combine multiple csv files into one workbook separate sheets python, python merge multiple csv files, merging two csv files with a common column python, merge csv files python pandas, pandas merge csv files by column, graph large csv files, load large csv files mysql, load large csv files php, reading large csv files net, query large csv files, merge multiple csv files, merge horizontal csv files, query large csv files excel, google translate large csv files, how to open large csv files in excel 2010, free software to open large csv files, how to compare two large csv files

About the Employer:
( 0 reviews ) Santa Clara, United States

Project ID: #27364042

23 freelancers are bidding on average $175 for this job

liveexperts123

Hi there,I'm biddin on your project "Code that will parse and merge large csv files" I have read your project description and i'm confident i can do this project for you perfectly.I still have a few questions. please l More

$250 USD in 4 days
(44 Reviews)
7.0
hbxfnzwpf

I am very proficient in c/c++. I have 17 years c/c++ developing experience now, and have worked for more than 10 years. My work is online game developing, and mainly focus on server side, using c++ under Linux environm More

$100 USD in 1 day
(126 Reviews)
6.8
sodiqa32

⭐⭐Hello, there!⭐⭐ ✅I have gone through your project details carefully and I think that I am the best fit developer for your project. ✅I have rich experience in similar projects, so I have a clear way to complete the pr More

$34 USD in 1 day
(25 Reviews)
5.6
umg536

Hi there, I'm bidding on your project "Code that will parse and merge large csv files" Being an expert in C, and C++ programming I can do this project for you. please leave a message on my chat so we can discuss the b More

$250 USD in 4 days
(12 Reviews)
5.1
Shendiqerimi

Hello, Ken! I have read your description very carefully, so I'm very confident in your project. I have rich experiences in CSV parsing with python and done many projects such as your projects . In your pro More

$100 USD in 2 days
(10 Reviews)
4.6
hasankassim7

Hello I am a Python and c++ expert. I have worked on a project of sorting very large files, so I am capable of doing yours. Let me know please about the criteria of merging. Feel free to send a message.

$80 USD in 2 days
(31 Reviews)
4.7
DRG1990

Hello I have read your job description carefully. Also, I have rich experience in python So I am interested in and I am confident in your project. I always provide perfect results to my clients as you can see my profil More

$140 USD in 3 days
(10 Reviews)
4.4
CrazyProger

I'm a Python expert with rich experience in CSV. I read the job description and understood the requirements. I think I can finish this project within short period with a minimal budget. I can start work immediately. More

$140 USD in 2 days
(13 Reviews)
4.6
casper70

Hi I am fully experienced working with python and c++. Contact me to discuss further details. Thanks for your time :)

$300 USD in 3 days
(18 Reviews)
4.7
drozhkov

hi, probably i can help you 100chars100chars100chars100chars100chars100chars100chars100chars100chars

$250 USD in 7 days
(7 Reviews)
4.1
igorshapo926

Hello. Thanks for your attention. I am Machine Learning, Deep Learning, Computer Vision expert and working in these fields from the past 5 years. The major field in which I have worked on including Deep Learning, Machi More

$150 USD in 1 day
(3 Reviews)
4.0
Wellkuka

Hi I can do this job. There are several libraries in python for handling large data. I can start working right now.

$140 USD in 7 days
(4 Reviews)
3.7
Dutchstudent7750

Hello. So in order to speed up the merging there's a few things to few things to try, optimizing the query, choosing a different data structure and parallelization. I would need to take a look at what exactly it is yo More

$100 USD in 1 day
(19 Reviews)
3.9
QASIM7862001

I believe i can reduce the time by alot get in touch Hey there, i am developer from the UK with over 9 years experience in web development. Upon reading your project description this seems like a task which i can start More

$250 USD in 1 day
(3 Reviews)
3.8
Devesh23

Hello, I am kernel developer and I might have an idea what might be happening in your code. Your code is trying to read the file and work on it at runtime. Now, its a big file and if you are using C++ then you must be More

$250 USD in 7 days
(14 Reviews)
3.9
shahzain93

Hi, I can optimize code, Please share your current code so i can check and get back to you? what about language, can I use according to my choice? Thanks

$220 USD in 7 days
(2 Reviews)
3.1
KMilan27

Hello, I can reduce the time using Multi Processing in Python. Let's discuss more in chat. Thank you Milan

$200 USD in 3 days
(2 Reviews)
2.5
Krimson1991

Hi, In my previous job I did a lot of data analyzing with selfmade scripts in Python and C++. I think I can help you to create a better script for it. Can you tell me more about what kind of data needs to processed? More

$200 USD in 14 days
(1 Review)
1.4
jainrakesh1701

I have 8 years experience in Python programming. Using python I have processed huge volume of data(in GB) and extracted meaningful information from it.I have worked with various formats like Json, Csv, Excel, XML, Par More

$200 USD in 7 days
(0 Reviews)
0.0
mateocolombo95

Hey there, I imagine that it can be solved using recursion, maybe with divide and conquer, or somthing similar.

$200 USD in 7 days
(0 Reviews)
0.0