score:1
as far as i can tell all you need here is aggregate
with zerovalue
, seqop
and combop
corresponding to the operations which are performed by your accumulators.
val zerovalue: (a, b, c) = ??? // (accum1.zero, accum2.zero, accum3.zero)
def seqop(r: (a, b, c), t: t): (a, b, c) = r match {
case (a, b, c) => {
// apply operations equivalent to
// accum1.addaccumulator(a, t)
// accum2.addaccumulator(c, t))
// accum3.addaccumulator(c, t)
// and return the first argument
// r
}
}
def combop(r1: (a, b, c), r2: (a, b, c)): (a, b, c) = (r1, r2) match {
case ((a1, b1, c1), (a2, b2, c2)) => {
// apply operations equivalent to
// acc1.addinplace(a1, a2)
// acc2.addinplace(b1, b2)
// acc3.addinplace(c1, c2)
// and return the first argument
// r1
}
}
val rdd: rdd[t] = ???
val accums: (a, b, c) = rdd.aggregate(zerovalue)(seqop, combop)
Source: stackoverflow.com
Related Query
- aggregating multiple values at once
- Scala Spark aggregating values from multiple columns into new column
- Is there a way to add literals as columns to a spark dataframe when reading the multiple files at once if the column values depend on the filepath?
- How to count occurrences of different values in multiple columns all at once where number or name of columns is not known?
- How to pattern match multiple values in Scala?
- Catching multiple exceptions at once in Scala
- Aggregating multiple columns with custom function in Spark
- Assign multiple variables at once in scala
- In Scala, is there a neat and simple way to compare one value with multiple values
- Overriding multiple config values in Typesafe config when using an uberjar to deploy
- Can I import from multiple packages all at once in Scala?
- Multiple Option values in Scala
- Scala collections, single key multiple values
- Setting up sbt environment to hack on multiple libraries at once
- Scala & Spark: Cast multiple columns at once
- a function that returns multiple values in Scala
- scala hashmap multiple values
- How to do aggregation on multiple columns at once in Spark
- Converting multiple optional values in Scala
- Scala multiple implicit parameters with defaults resulting in ambiguous values
- How to write a condition based on multiple values for a DataFrame in Spark
- Adding a new column to a Dataframe by using the values of multiple other columns in the dataframe - spark/scala
- Single key contains Multiple values in Map in scala
- Inserting multiple values into table with anorm
- Gatling - Check multiple values with JsonPath
- Scala Ordering by multiple values
- Vectorize Scala function with multiple return values
- Idiomatic way to update multiple values in scala.immutable.Map
- Are there any difference calling `run.db` once and multiple times in Slick
- Scala assigning multiple returned values from a function
More Query from same tag
- Scala getting a collection back when the compiler says that it has type Any
- How rename S3 files not HDFS in spark scala
- passing map containing user defined object as value to play framework2 scala template
- How to convert RDD to DF in spark scala?
- Scala, iterating a collection, working out 10% points
- higher kinded types and WrappedString
- How to split a Play application into several parts
- error in initSerDe : java.lang.ClassNotFoundException class org.apache.hive.hcatalog.data.JsonSerDe not found
- Spark levenshtein left join on string, match one data then break
- Scala Code snippet - What is the Java 8 equivalent?
- Scala Yeild returning Try[Either[]] rather then Either
- Scala define type from multiple generic traits
- scala inline function - "ar.reduce((x,y) => x)" works but "ar.reduce{case(x,y) => x}" doesnt
- How to set thread timeout
- Using traits with a factory
- Location track items must be given as pairs of comma separated lat/longs Twitter4J Scala
- how to concatenate option in scala
- Why do exceptions not stop Akka Stream flow
- how to create range column based on a column value?
- akka: sharing mutable state
- Convert List to akka.japi.Function[akka.http.javadsl.model.HttpRequest, java.util.concurrent.CompletionStage[akka.http.javadsl.model.HttpResponse]]*
- How to calculate join in Spark with top N failed matches efficiently?
- How can Scala receive multiple parameters in a method definition?
- How can I use generics for a Scala (2.12) macro?
- Mock not recognized in IntelliJ while using Scala
- Are sealed classes enforced in Java and, if yes, how?
- Get previous 12months with month end date from given month using Scala
- Reaching JWT Token in play2 silhouette JWTAuthenticator
- How to implement lazy sequence (iterable) in scala?
- import implicit conversions without instance of SparkSession