score:0
Accepted answer
you have some spelling errors, and finally the last function count cannot be there you need to import org.apache.spark.sql.functions._
check this, i corrct the spelling errors and add the imports
import org.apache.spark.{ sparkcontext, sparkconf }
import org.apache.spark.sql.functions._
/**
* created by anquegi on 01/06/15.
*/
object qspark162015 extends app {
val conf = new sparkconf()
.setmaster("local[2]")
.setappname("qspark162015")
.set("spark.executor.memory", "2g")
val sc = new sparkcontext(conf)
val sqlcontext = new org.apache.spark.sql.sqlcontext(sc)
import sqlcontext.implicits._
val df = sc.parallelize(array((1, 30), (2, 10), (3, 20), (1, 10), (2, 30))).todf("books", "readers")
val results = df.join(
df.select($"books" as "r_books", $"readers" as "r_readers"),
$"readers" === $"r_readers" and $"books" < $"r_books"
)
.groupby($"books", $"r_books")
.agg($"books", $"r_books", count($"readers"))
results.foreach(println _)
sc.stop()
}
Source: stackoverflow.com
Related Query
- Difference between filter and where in scala spark sql
- java.lang.NoClassDefFoundError: Could not initialize class when launching spark job via spark-submit in scala code
- What is the Scala type mapping for all Spark SQL DataType
- Scala case class ignoring import in the Spark shell
- scala generic encoder for spark case class
- Why don't Scala case class fields reflect as public?
- Best way to reflect case class fields in scala 2.11?
- Spark Scala getting class not found scala.Any
- Unsupported literal type class in Apache Spark in scala
- Scala Spark : (org.apache.spark.repl.ExecutorClassLoader) Failed to check existence of class org on REPL class server at path
- Scala - Spark sql row pattern matching on struct
- Return Temporary Spark SQL Table in Scala
- Spark Internal class Kryo registration
- How to avoid automatic cast for ArrayType in Spark (2.4) SQL - Scala 2.11
- Scala 2.11 & Spark 2.0.0 Create dynamically case class to encode Dataset
- Scala Spark DataFrame SQL withColumn - how to use function(x:String) for transformations
- writing a UDF in spark sql with scala
- Scala class lazy val variables strange behaviour with Spark
- How to test a Spark SQL Query without Scala
- Spark SQL execution in scala
- Use scala reflection to initiate class with internal objects
- how to cast a spark row(StructType) to scala case class
- implement sum aggregator on custom case class on spark sql dataframe (UDAF)
- How to build Encoder for Regex class in Scala w/ Spark
- Task not serializable while using custom dataframe class in Spark Scala
- How to infer StructType schema for Spark Scala at run time given a Fully Qualified Name of a case class
- Issue converting scala case class to spark dataset with embedded options that wrap monads
- Reflect on annotation class from within scala macro annotation?
- Spark scala - Output number of rows after sql insert operation
- Execute Spark sql query within withColumn clause is Spark Scala
More Query from same tag
- Cannot run scala in IntelliJ
- How to union 2 dataframe without creating additional rows?
- Erasure of type fields
- Testing injected controllers in Play for Scala
- How to run Test Suites sequentially in ScalaTest / SBT?
- Spark Scala Delete rows in one RDD based on columns of another RDD
- How do I get type parameter from type with type parameter, inside scala macro?
- Generate Random Hexidecimal in Scala?
- Class#getInterfaces() and Class#getGenericInterfaces() return arrays of differing length
- Difference between query string and path parameter in play framework?
- what is the reason multiply default dispatcher threads
- Graphx Scala: Applying filter on Vertex with Property Inheritance
- Scala - multiple function, returns multiple function
- object xml is not a member of package scala
- What is the purpose of outer and inner function parameters in Scala?
- How can I make this method more Scalalicious
- scala - it is possible to create "Int increment extension method" that can be invoked from any package?
- found scala.Int(0) required Int 0
- Scala - Executing every element until they all have finished
- Scala: Edit/Modify json string based on internal value
- Terminal animation - is clearing screen right approach?
- Keeping the materialized value of a source passed to a chunked request entity (client-side)
- Scala Variables scope
- Play Framework 2.1.2 internationalization doesn't work with Finnish characters
- How can I create an instance of a Case Class with constructor arguments with no Parameters in Scala?
- how to convert yyyy-dd-mmThh:mm:ss in spark sql
- Scala can't infer type arguments of Java method
- Scala - compose function n times
- Cannot run sbt on redhat
- How to use ScriptEngine in ScalaTest