score:2

Accepted answer

it's working now for me, and just for the record, referencing @martinsenne answer.

what i did is as below:

  1. clear all compile files under folder "project"
  2. scala version 2.10.4 (previously using 2.11.4)
  3. change spark-sql to be: "org.apache.spark" %% "spark-sql" % "1.4.1" % "provided"
  4. change mllib to be: "org.apache.spark" %% "spark-mllib" % "1.4.1" % "provided"

@note:

  1. i've already started a spark cluster and i use "sh spark-submit /path_to_folder/hello/target/scala-2.10/hello_2.10-1.0.jar" to submit jar to spark master. if use sbt to run by command "sbt run" will fail.
  2. when changing from scala-2.11 to scala-2.10, remember that the jar file path and name will also change from "scala-2.11/hello_2.11-1.0.jar" to "scala-2.10/hello_2.10-1.0.jar". when i re-packaged everything, i forgot to change the submit job command for the jar name, so i package into "hello_2.10-1.0.jar" but submitting "hello_2.10-1.0.jar" which caused me extra problem...
  3. i tried both "val sqlcontext = new org.apache.spark.sql.sqlcontext(sc)" and "val sqlcontext = new org.apache.spark.sql.hive.hivecontext(sc)", both are working with method createdataframe()

Related Query

More Query from same tag