Tuesday, March 5, 2019

Spark Get Maximum Volume by Stock

val stocks = sc.textFile("hdfs://ip-10-10-10-10.ec2.internal:8020/user/deepakdubey/input/stocks")
val splits = stocks.map(record => record.split(","))
val symvol = splits.map(arr => (arr(1), arr(7).toInt))
val maxvol = symvol.reduceByKey((vol1, vol2) => Math.max(vol1, vol2))
maxvol.collect().foreach(println)

--Spark shell
spark-shell --master yarn