If you want to test your Spark program locally, you don't even need to spin up the single-node standalone Spark. Just set your master url to local[*] like this

val sc = new SparkContext("local[*]", "Simple Application", sparkConf)

Then in sbt, type > run to run your program (this should be the same as running from IntelliJ, but I used to run the program from terminal using sbt).

Since you may not want to change your master url in code between local[*] and spark://... many times, you can leave them blank

val sc = new SparkContext(new SparkConf())

and set your java properties when running, for example, in build.sbt, you can add

javaOptions := Seq("-Dspark.master=local[*]", "")

and run it using run in sbt.

To make a more comprehensive local-mode experience, you may want to add the following lines in your build.sbt

run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))
runMain in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))
fork := true
javaOptions := Seq("-Dspark.master=local[*]", s"")
outputStrategy := Some(StdoutOutput)

We have created a sbt plugin which can add these settings for you, it can also help you deploy a standalone Spark cluster on cloud system like aws ec2, give a look at spark-deployer if you are interested.


You are missing a crucial step:


which actually submits a job to the cluster. Unfortunately there is not presently a solid working wrapper for it except for spark-submit. So there presently is not a reliable way to programatically submit spark jobs. There is a jira for it that was partially addressed in Feb 2015: but it lacks documentation.

The difficulty lies in the complexity of the environmental machinations provided by spark-submit. It has not been found possible to replicate them solely within scala/java code.


Here is the Class.

    import org.apache.spark.launcher.SparkLauncher;

   public class MyLauncher {
     public static void main(String[] args) throws Exception {
       Process spark = new SparkLauncher()
         .setConf(SparkLauncher.DRIVER_MEMORY, "2g")

Related Query

More Query from same tag