Spark Parallel Execution

Experiments around executing spark queries in parallel.

Three methods are used here: -

• ParallelWithFutures uses standard Scala Futures and callbacks.

• ParallelWithCats uses Cats Effect library to wrap the spark context in a Resource and parTraverseN to control level of parallelism.

• ParallelWithZIO uses ZIO to wrap the spark context in a ZIO Scope and use foreachPar to run the queries in parallel. The ZIOAspect.parallel aspect controls the level of parallelism.

ParallelWithFutures

Uncomment the line : -

    // Uncomment the following line to pause the code and allow the Spark UI to be viewed 
//scala.io.StdIn.readLine()

to view the Spark UI.

ParallelWithCats/ParallelWithZIO

Note: Use method closeSparkSessionWithPause to keep the Spark UI active.

Example Spark Job Output with parTraverseN(1)

Example Spark Job Output with parTraverseN(10)

Notes

Running on Java 17+

Add the following to Java options to run : -

--add-exports java.base/sun.nio.ch=ALL-UNNAMED

ToDo

1. Lift the SparkEnv config into a ZIO layer also. See zio.config package.