Completed

Trigger Spark Jobs based on various conditions

Suppose there are two folders A and B. A folder will be loaded with Employee data files and B folder will be loaded with Department data files.

Every minute, the java program should monitor these folders and trigger SparkA(based on folder A) or SparkB(based on folder B) programs(Spark-Submit or equivalent). After successful completion, the java program should move the processed files to a different location. The condition is either for every five minutes or for exceeding the size of the folder to more than 2GB.(Spark 2.1.1)

You can use below code for reference:

[url removed, login to view]

Task 1:
Run the spark job(spark-submit) from java code.

Task2:
Schedule the java class based on time or size, whatever matches first. (Configurable size or configurable time)

Task3:

Java class should read the files in the folder and move them to a different folder and Spark job will read the files from this new folder.

Skills: Hadoop, Java, Spark

See more: spark jobs, online jobs based on typing from home, jobs based from home, how can i find a developer my blockchain app concept, freelancer jobs on various websites, jobs based at home in recruitment, i am three months of visiter in dubai and i need a job my qualification is i com 12 year expirience data entry in pakistan urgen, i need a full time java coder, i need a christmas and new year ecard for my prague czech republic based travel business the text needs to be editable i will ne, i have a matching algorithm which matches doctors profiles to jobs posted by employers, graphic designers looking for jobs based in jhb, dropbox https://wwwdropboxcom/s/7n2rnqzsxrd1xci/e-rauch%20storerar?dl=0dropbox https://wwwdropboxcom/s/7n2rnqzsxrd1xci/e-rauch%2, jobs based tamil language anywhere, interface based krpano navigated php folder tree, offline jobs based flash software, mmorpg jobs based uk, jobs based vb, jobs based flash, freelancing jobs based photoshop chennai, jobs based home review, typing jobs based cebu

About the Employer:
( 0 reviews ) United States

Project ID: #16753314

Awarded to:

sandeepfreax

Hi I am interested in this job. I believe that the skills and experiences I have gained at this position make me an ideal candidate for this position. I have extensive experience in implementing Spark-core, Spark-s More

$55 USD in 4 days
(0 Reviews)
0.0

6 freelancers are bidding on average $73 for this job

Gaosong2017

Hello, dear How are you? I have experience in Hadoop in pyspark with python. I think that i can help you as well. Hope your contact me. Best Regards.

$100 USD in 1 day
(18 Reviews)
5.2
AmolZinjadeP

Hi, I have more that 4+years of experience in hadoop and data mining technologies like HDFS, MapReduce, python, Spark, Hive etc. Please review my profile for skills Contact me

$100 USD in 1 day
(5 Reviews)
3.9
duym

A proposal has not yet been provided

$72 USD in 3 days
(3 Reviews)
2.6
Yuanyifei

please accept my proposal for the job of Trigger Spark Jobs based on various conditions,I have a BA in Communication and Information System Engineering. 3+ years working as programmer has given me understanding of h More

$77 USD in 1 day
(0 Reviews)
0.0
gurukarnamprasad

A proposal has not yet been provided

$35 USD in 3 days
(0 Reviews)
0.0