score:0
Accepted answer
from the manuals an example:
val windowedcounts = words
.withwatermark("timestamp", "10 minutes")
.groupby(
window($"timestamp", "10 minutes", "5 minutes"),
$"word")
.count()
try putting your window clause up front i would hazard to guess. and use of $ as in the examples for field names.
score:0
val jsondf = explodedf.withwatermark("timestamp", "1 minutes")
val aggdf = jsondf.groupby(functions.window(jsondf.col("timestamp"), "30 seconds", "30 seconds"),jsondf.col("jsondata.name")) .avg("jsondata.price").alias("averageprice")
try this, thanks me latter
Source: stackoverflow.com
Related Query
- Window Overload method cannot resolve in spark structured streaming-scala
- Spark 2.4.0 Avro Java - cannot resolve method from_avro
- Scala cannot resolve correct overloaded SLF4J method after flatmapping AnyRef*
- Cannot resolve Queries with streaming sources must be executed with writeStream.start() Scala
- Spark Structured Streaming - Custom aggregation with window time event
- How to use Scala Case Class to map Kafka source in Spark Structured Streaming
- Spark streaming scala window length by number of objects
- Scala Spark - Cannot resolve a column name
- Scala schema_of_json function fails in spark structured streaming
- How to specify structured streaming time based window in straight Spark SQL
- Window operations on date column in Spark Structured Streaming
- Creating Schema of JSON type and Reading it using Spark in Scala [Error : cannot resolve jsontostructs]
- I cannot make dataframe using streaming mode for online prediction in apache spark using scala
- Scala Named method arguments cannot resolve symbol
- scala compiling error: Cannot resolve overloaded method 'withTransaction'
- scala Spark structured streaming Receiving duplicate message
- Cannot resolve task not serializable [org.apache.spark.SparkException: Task not serializable] Spark Scala RDD
- Sliding window in Spark Structured Streaming
- Window length in Spark Structured Streaming
- Scala method cannot resolve variable
- Cannot resolve column name error for column passed as strings - Spark Scala
- Spark Structured Streaming cannot read from kafka inside docker
- How to call a method after a spark structured streaming query (Kafka)?
- Cannot Resolve Overloaded Method IntelliJ 2018.2.8 / Scala 2.11.8/Spark 2.4.5
- cannot resolve symbol apache in spark scala maven
- Scala Spark Structured Streaming Filter by TimestampType within Struct Field
- Spark streaming store method only work in Duration window but not in foreachRDD workflow in customized receiver
- Scala JavaFx -- Cannot resolve overloaded method 'add' when trying to add tree table columns
- Running a spark structured streaming application scala code written in a file using nohup
- reduceByKey method not being found in Scala Spark
More Query from same tag
- Two sets of constructor parameters in a scala class
- Throwing Exception in Foreach/Map Block
- Setting up Slick 3.xx to use different databases in different environments
- Scala library available in both compile and test configuration
- How implement a DSL for a matrix-lib in scala?
- What are the dangers of using an object as an AKKA actor?
- Spark - Task Scheduler Error while parsing JSON
- Access Request Body in essential filter Play Framework 2
- Lagom - How can I use event processor in a service integration test?
- Starting Node server from Scala app with environment variables
- How to convert scalastyle-result.xml into readable report
- Idiom for creating a scoped value only once in scala
- Spark Scala: Count Consecutive Months
- Monitoring a task in apache Spark
- Which implicit conversions in Scala are present as default when nothing is imported
- Read the input xml data from HDFS using scala
- Spark - How to deal with updates
- Pushing elements externally to a reactive stream in fs2
- Different use case for akka cluster aware router & akka cluster sharding?
- Null Pointer Exception in Spark JavaPairRDD: (JavaPairRDD.scala:1028)
- Two ways of starting a scala script, which is preferable?
- How to get the structure of JSON in scala
- Akka http streams using sink memory is growing
- How to handle recursion with monix's observable?
- Using Slick with Scala (with Playframwork 2.3)
- Scala - is it possible to write HOF that has varargs?
- Where is x from?
- Is there a way to require constructors that take particular arguments?
- Usage of javascriptRouter in Play 2.0.2
- QueryString parsing in Play