[Solved]-How do you submit a job to Spark from code?-scala

If you want to test your Spark program locally, you don't even need to spin up the single-node standalone Spark. Just set your master url to local[*] like this

val sc = new SparkContext("local[*]", "Simple Application", sparkConf)

Then in sbt, type > run to run your program (this should be the same as running from IntelliJ, but I used to run the program from terminal using sbt).

Since you may not want to change your master url in code between local[*] and spark://... many times, you can leave them blank

val sc = new SparkContext(new SparkConf())

and set your java properties when running, for example, in build.sbt, you can add

javaOptions := Seq("-Dspark.master=local[*]", "-Dspark.app.name=my-app")

and run it using run in sbt.

To make a more comprehensive local-mode experience, you may want to add the following lines in your build.sbt

run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))
runMain in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))
fork := true
javaOptions := Seq("-Dspark.master=local[*]", s"-Dspark.app.name=my-app")
outputStrategy := Some(StdoutOutput)

We have created a sbt plugin which can add these settings for you, it can also help you deploy a standalone Spark cluster on cloud system like aws ec2, give a look at spark-deployer if you are interested.

You are missing a crucial step:

org.apache.spark.deploy.SparkSubmit

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

which actually submits a job to the cluster. Unfortunately there is not presently a solid working wrapper for it except for spark-submit. So there presently is not a reliable way to programatically submit spark jobs. There is a jira for it that was partially addressed in Feb 2015: but it lacks documentation.

 https://github.com/apache/spark/pull/3916/files

The difficulty lies in the complexity of the environmental machinations provided by spark-submit. It has not been found possible to replicate them solely within scala/java code.

Here is the Class.

https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/launcher/package-summary.html

    import org.apache.spark.launcher.SparkLauncher;

   public class MyLauncher {
     public static void main(String[] args) throws Exception {
       Process spark = new SparkLauncher()
         .setAppResource("/my/app.jar")
         .setMainClass("my.spark.app.Main")
         .setMaster("local")
         .setConf(SparkLauncher.DRIVER_MEMORY, "2g")
         .launch();
       spark.waitFor();
     }
   }