score:6

Accepted answer

because null means no value and shouldn't be checked like this.

use isnull function:

val df3 = df2.withcolumn("a1", when($"a1".isnull, $"a2"))

or coalesce, which returns first non-null value:

val df3 = df2.withcolumn("a1", coalesce($"a1", $"a2"))

Related Query

More Query from same tag