You run your application on a Spark cluster, so I guess the problem is that your code cannot find the users.txt file. Try using a path that can be reached independent of the current work directory and/or using
local as the
You need not only to compile project, but package it as assembled jar file with all dependencies. It can be a little trick to remove all dependencies conflicts. You can take a look at my sample project for Spark that uses Spark-Streaming, it can be helpful to start with
especially to Dependencies exclusion https://github.com/pellucidanalytics/tweet-driven-comparable-companies/blob/master/project/Dependency.scala
And read some articles:
http://prabstechblog.blogspot.com/2014/04/creating-single-jar-for-spark-project.html http://eugenezhulenev.com/blog/2014/11/20/twitter-analytics-with-spark/ http://eugenezhulenev.com/blog/2014/10/18/run-tests-in-standalone-spark-cluster/
- Correct build.sbt file for apache spark
- making sbt scala build file bring in apache spark
- How to define a file filter for file name patterns in Apache Spark Streaming in Java?
- Build sbt for spark with janusgraph and gremlin scala
- sbt build failed for spark scala with xgboost
- Build issues for apache spark
- Optimal way to create a ml pipeline in Apache Spark for dataset with high number of columns
- How to create correct data frame for classification in Spark ML
- Apache Spark Dataframe Groupby agg() for multiple columns
- Read an unsupported mix of union types from an Avro file in Apache Spark
- SBT build structure for a mono repository
- Using Apache Spark as a backend for web application
- Access public available Amazon S3 file from Apache Spark
- Apache Spark 2.0: java.lang.UnsupportedOperationException: No Encoder found for java.time.LocalDate
- Build stateful chain for different events and assign global ID in spark
- Including a Spark Package JAR file in a SBT generated fat JAR
- Use sbt shell for build and import
- Work with Jupyter on Windows and Apache Toree Kernel for Spark compatibility
- Writing to a file in Apache Spark
- Producing no artifact for root project with package under multi-project build in SBT
- Difference between Apache spark mllib.linalg vectors and spark.util vectors for machine learning
- Scalaz Type Classes for Apache Spark RDDs
- Read file on remote machine in Apache Spark using ftp
- dynamically changing library dependencies in sbt build file from provided etc
- Having sbt to re-run on file changes - The `~ compile` equivalent for `run`
- Apache Spark reading UTF-16 CSV file
- Is there a configuration file for Scala REPL / SBT Console?
- Configure repo for SBT launcher in Travis build
- Find size of data stored in rdd from a text file in apache spark
- Using a common test utility file for projects built in SBT
More Query from same tag
- What is this Scala construct doing?
- reg: what is the library dependency for sqlContext?
- Why can I have an impossible case in the scala match?
- Array initializing in Scala
- Spark SQL join really lazy?
- Json path to extract based on particular value
- Unpacking tuple types in Scala
- object Neo4j is not a member of package org.neo4j.spark
- Pig-Scala UDF exception- ScalaObject not found
- Convert List into dataframe spark scala
- How can I ask sbt-buildinfo to run only at (re)load time and not compile time?
- Mocking a val of a trait with scala-mock
- Scala - Ids lists of objects with duplicated values from spark dataset
- Scalatest Spec output in Eclipse
- Using contains in scala - exception
- How to pass DataSet(s) to a function that accepts DataFrame(s) as arguments in Apache Spark using Scala?
- How to exclude assembly from package in sbt?
- Why can't I use the constructor of a case class as a function for use inside map()
- ScalaTest plugin in eclipse not finding my class
- Play Framework: What happens when requests exceeds the available threads
- What's the equivalent of hoistEither in scalaz?
- How to convert String of date format "dd-MMM-yy" into DateType
- Obtaining the value from any key-value from a string representation of a json in Scala (using scala.util.parsing.json)
- Collections in scala, how to get elements in map
- What is type declaration for in definition of Nat in shapeless?
- Why is my source producing more than the buffer size?
- OneToMany Relations with Lift
- End-dating records using window functions in Spark SQL
- Storing an array of ranges of timestamps into postgresql using PlaySlick and slick-pg
- how to iterate over array[string] in spark scala?