score:6
Accepted answer
because null means no value and shouldn't be checked like this.
use isnull
function:
val df3 = df2.withcolumn("a1", when($"a1".isnull, $"a2"))
or coalesce
, which returns first non-null value:
val df3 = df2.withcolumn("a1", coalesce($"a1", $"a2"))
Source: stackoverflow.com
Related Query
- In DataFrame.withColumn, how can I check if the column's value is null as a condition for the second parameter?
- How to merge the value of several columns into a map in Spark Dataframe
- How to add column in Dataframe base on the value of other Columns spark
- How to fill the null value in dataframe to uuid?
- How to select specific columns from Spark DataFrame based on the value of another column?
- How to put values to the new separate DataFrame columns based on the value of specific already existing column? I.e transpose the DataFrame
- How can I add several columns (still not populated) to the DataFrame in Spark Structured Streaming
- How can I filter spark Dataframe according to the value that column contains?
- how to filter out a null value from spark dataframe
- How can I override a typesafe config list value on the command line?
- How can I find the index of the maximum value in a List in Scala?
- How can I fix the missing implicit value for parameter ta: TildeArrow in a test spec
- How can I obtain the default value for a type in Scala?
- How can I syntax check a Scala script without executing the script and generating any class files?
- Scala Dataframe null check for columns
- How to check the number of partitions of a Spark DataFrame without incurring the cost of .rdd
- Given a value of some path-dependent type, how can I get an instance of the "container"?
- Spark Dataframes: How can I change the order of columns in Java/Scala?
- How to implicitly wrap a value that can be null or an array into an Scala Option
- Scala - Spark - How to transform a dataframe containing one string column to a DF with columns with the rigth type?
- How can I get a value from a the ScalaFX thread?
- How to get the row from a dataframe that has the maximum value in a specific column?
- How to get the set of rows which contains null values from dataframe in scala using filter
- how to create DataFrame if I can not use the SparkContext?
- In DataFrame.withColumn, how can I use the column's value as a condition for the second parameter?
- How can find size of each Row in Apache spark sql dataframe and discrad the rows having size more than a threshold size in Kilobyte
- How can i check for empty values on spark Dataframe using User defined functions
- How can I check if Any value isEmpty?
- How can I write dataframe to csv file using one partition although the file size exceeds executors memory
- How to group by a column on a dataframe and applying single value to columns of all rows grouped?
More Query from same tag
- Difference between file and class file in IntelliJ
- How is this exactly a curried function?
- One and two hop citation in scala
- Print a variable as part of a multiline string?
- How to overwrite NaN or Infinity result when performing arithmetic on Double?
- Scala - How to group a list of tuples without pattern matching?
- Custom input reader in spark
- Workaround for Scala macro annotation bug
- How to use multiple counters in Flink
- how to call function of scala class from main method?
- How to get a single defined element of a Coproduct instance?
- Composing scalaz io effect objects
- Persistence Layer Abstraction [Slick]
- Render json data to the view with play! scala 2.2.3
- Scala Future not printing results
- Creating an instance of the self type in Scala
- Passing a type parameter for instantiation
- Spark Structured Streaming - Custom aggregation with window time event
- Spark - Iterating through all rows in dataframe comparing multiple columns for each row against another
- Play Framework - migration from version 2.3.x to 2.4.1: how to determine whether a JSON value is JsUndefined
- GADT Type as Shapeless Coproduct -- how to build an Interpreter with an arbitrary number of Algebras
- Split or extract a Value from a vector SCALA
- Java Class not Found Exception while doing Spark-submit Scala using sbt
- How to use json object from Javascript in Scala to connect with database?
- How to set an empty value in a lift select element?
- ClassNotFoundException when Loading Custom Class
- SOAP based WebServices and WSDL
- Scala equivalent for ActiveSupport's Object.try in Ruby
- How to return a list all paths(Branches) of nodes that are in a binary tree using Scala?
- How to call Scala's HashMap.toArray() from Java?