score:1
Accepted answer
aggregate
should be used inside a spark sql expr
for spark 2.4. also it should be better to add a type cast to ensure there is no type mismatch:
df.withcolumn("amount", expr("aggregate(list_val, 0, (x, y) -> (x + int(y)))")
// for float type; for double type, replace "float" with "double"
df.withcolumn("amount", expr("aggregate(list_val, float(0), (x, y) -> (x + float(y)))")
in scala api that would be
df.withcolumn("amount", aggregate($"list_val", lit(0), (x, y) => (x + int(y))))
df.withcolumn("amount", aggregate($"list_val", lit(0f), (x, y) => (x + float(y))))
df.withcolumn("amount", aggregate($"list_val", lit(0.0), (x, y) => (x + double(y))))
Source: stackoverflow.com
Related Query
- add values from struct keys spark
- Create a dataframe from a hashmap with keys as column names and values as rows in Spark
- Fetch all values irrespective of keys from a column of JSON type in a Spark dataframe using Spark with scala
- Spark aggregate keys from JSON input files and output values to a new JSON file
- Filter out struct of null values from an array of structs in spark dataframe
- Comparing values from different keys in scala / spark
- Spark extracting values from a Row
- How to get keys and values from MapType column in SparkSQL DataFrame
- Getting values from Map given list of keys in Scala
- How can I create a Spark DataFrame from a nested array of struct element?
- Null values from a csv on Scala and Apache Spark
- Using regex to access values from a map in keys
- How do add values of selective rows from a list in an functional style?
- How to remove keys with null values from Argonaut Json objects
- How to add null values in an array in spark scala
- How to efficiently find distinct values from each column in Spark
- Spark Scala - How do I iterate rows in dataframe, and add calculated values as new columns of the data frame
- Split an Spark dataframe by some column values and then rotate each generated dataframe independently from the others
- Sum up into a new column from a range of values of a column in Spark using Scala
- Spark & Scala - Cannot Filter null Values from RDD
- In scala, how to get an array of keys and values from map, with the correct order (i-th key is for the i-th value)?
- Spark Dataframe - How to get a particular field from a struct type column
- Spark DataFrame - drop null values from column
- Convert row values into columns with its value from another column in spark scala
- how to extract the column name and data type from nested struct type in spark
- Lookup values from a MapType column with keys from another column
- Spark Streaming - Calculating stats from key-value pairs grouped by keys
- create a Spark DataFrame from a nested array of struct element?
- How to sum values of a struct in a nested array in a Spark dataframe?
- How to add null columns to complex array struct in Spark with a udf
More Query from same tag
- Scala : function to measure the runtime of any other function
- scala - modeling methods as classes
- How to write in Scala shortly "filter first/last n elements satisfying a given function"?
- Getting out of memory error while reading parquet file in spark submit job
- how is scalaz.Equal resolved
- Unable to pass the spark developer code check, encountered binary compatibility error
- hasDefiniteSize and knownSize
- Sorting 2D String array in Scala
- Is this an abuse of Scala pattern matching?
- Implicit class holding mutable variable in multithreaded environment
- Getting a stack trace from SBT with Scala
- Scala: ResultSet translation into a different kinds of multimaps
- How to dockerize my play scala app?
- Which Toolkit to develop Mac OSX/Windows Daemon service
- Why does BitSet require an explicit cast to be considered as an instance of Set[Int]?
- Scala : append Array to List
- Extract specific JSON field with Scala and Argonaut
- scala loop to add date string into a seq
- Get an implicit instance by class name
- Spark 2.0 Timestamp Difference in Milliseconds using Scala
- scala range split missing the last one
- Scala concatenate Map - one with Option and other without Option
- Scala Unable to pimp NaturalTransformation in scalaz or cats
- How to clean a CSV file for reading text in double quotes as one column
- How can I see in what [Java/Scala?] code does Scala compiler rewrites original Scala-code
- playframework scala Seq[Future[Model]] removing the Future
- List[List[String]] in Scala
- Is there a way to get proper report of runtime compilation errors in scala 2.10?
- Getting `Any` instead of Seq[DataFrame]
- Skuber delete items in a specific namespace