You have chosen to sponsor your bid up to a maximum amount of .
I want a program in C# / MySQL, that would be able to:
upload about 50 zip or CSV files. These files contain historical financial data and are very large for a total size of about 1.2GB zipped. I can send you a file example if you want.
1/ import files from a local computer. The files are either in ZIP or csv (a file example can be provided if necessary).
Table format (a table example can be provided if necessary).
orig_id bigint (20,0) not null primary key
id_minute bigint (20,0) not null
desc varchar(20) not null
datetime varchar(25) null
v1 decimal(10,5) null
v2 decimal(10,5) null
2/ sort data by datetime ascending
3/ remove any duplicate. duplicate if datetime & v1 & v2 in above/below records are equal
4/ add orig_id to every record
5/ group by minute (beginning & end of minute).
a) Please also allow possibility to only group by end of minute (or beginning of minute) instead of having beginning & end of minute values
b) Please also allow possibility to group by a variable number of minutes
6/ ad id_minute to every record by minute
7/ merge new data with historical data
8/ since process is very long, provide ability to text/email when each process is finished
The reason you will need to process file by file instead of importing all files together and doing the process only once is because the file size would be large to freeze the computer.
Once the 50 files are uploaded into mysql, design process to upload a monthly file to upload and do the same steps to then append to the past data.