Thx, I will try this out. In the meanwhile I found another solution: $ curl -k --user "admin:" -v -H "Content-Type: application/json" -X POST -d @Curl-KindSpark_BrandSampleModel_SessionSetup.json " with a JSON file containing { "kind": "spark", "jars": ["adl://"] } and with the uploaded jar in the Azure Data Lake Gen1 account containing the Scala object and then post the statement $ curl -k --user "admin:myPassword" -v -H "Content-Type: application/json" -X POST -d @Curl-KindSpark_BrandSampleModel_CodeSubmit.json "" -H "X-Requested-By: admin" with the content { "code": "import AzureSparkLivy_GL01._; val brandModelSamplesFirstModel = AzureSparkLivyFunction.SampleModelOfBrand(sc, \"Honda\"); brandModelSamplesFirstModel" }.

So I told Livy to start an interactive Spark session and load the specified jar and passed some code to invoke a member of the object in the jar. It works. Will check your advice too.


  • Make a uber jar of your Spark app with sbt-assemby plugin.

  • Upload jar file from the previous step to your HDFS cluster: hdfs dfs -put /home/hduser/PiNumber.jar /user/hduser

  • Execute your job via livy: curl -X POST -d '{"conf": {"kind": "spark" , "jars": "hdfs://localhost:8020/user/hduser/PiNumber.jar"}}' -H "Content-Type: application/json" -H "X-Requested-By: user" localhost:8998/sessions

  • check it: curl localhost/sessions/0/statements/3:

{"id":3,"state":"available","output":{"status":"ok","execution_count":3,"data":{"text/plain":"Pi is roughly 3.14256"}}}


Spark Livy API for Scala/Java requires using an uber jar file. sbt-assembly doesn't make fat jar instantly, it annoys me. Usually, I use Python API of Livy for smoke tests and tweaking.

Sanity checks with Python:

  • curl localhost:sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"print(\"Sanity check for Livy\")"}'

You can put more complicated logic to field code. BTW, it's a way in which popular notebooks for Spark works - sending the source code to cluster via Livy.

Related Query

More Query from same tag