score:4
Accepted answer
you will have to separate the person
struct
column into separate columns and then use drop
new_df.select("car", "person.*").drop("name")
if you want person.age
back then you can construct it back as struct
import org.apache.spark.sql.functions._
new_df
.select("car", "person.*")
.drop("name")
.withcolumn("person", struct("age"))
.drop("age")
root
|-- car: string (nullable = true)
|-- person: struct (nullable = false)
| |-- age: long (nullable = true)
as @raphaelroth has pointed out in the comments below that you can just use
new_df.select($"car",struct($"person.age").as("person"))
or even shorter as
new_df.withcolumn("person", struct("person.age"))
udf way
you can even do it in udf
way (is not recommended though) (just for your information)
import org.apache.spark.sql.functions._
def removestruct = udf((p: personold)=> person(p.age))
new_df.withcolumn("person", removestruct(col("person")))
for that you would need two case classes
though
case class personold(age: long, name: string)
case class person(age: long)
Source: stackoverflow.com
Related Query
- DataFrame drop column not working
- 2 dataframe column values not working in where clause
- Spark 1.6: drop column in DataFrame with escaped column names
- Filter dataframe by value NOT present in column of other dataframe
- Group by column "grp" and compress DataFrame - (take last not null value for each column ordering by column "ord")
- Self-join not working as expected with the DataFrame API
- where clause not working in spark sql dataframe
- Spark DataFrame column names not passed to slave nodes?
- Spark DataFrame - drop null values from column
- drop last column of a dataframe using spark-scala
- Spark is pushing down a filter even when the column is not in the dataframe
- Order By Timestamp is not working for Date time column in Scala Spark
- Spark DataFrame filter not working as expected with Random
- Add new column to dataframe with modified schema or drop nested columns of array type in Scala
- Add a column to spark dataframe which contains list of all column names of the current row whose value is not null
- Spark dataframe ".as" function does not drop columns not present in matched case class
- Drop rows of Spark DataFrame that contain specific value in column using Scala
- Getting latest based on column condition in Spark Scala is not working
- column names not present in dataframe spark
- Spark dataframe column names not being recognized
- DataFrame user-defined function not applied unless I change column name
- date_trunc function does not work with the spark dataframe while adding new column
- Spark withColumn working for modifying column but not adding a new one
- spark.time() not working for dataframe query
- Spark SQL DataFrame join with filter is not working
- Spark/Scala flatten and flatMap is not working on DataFrame
- Drop list of Column from a single dataframe in spark
- Creating Empty DF and adding column is NOT working
- Not able to apply function to Spark Dataframe Column
- Like clause not working with int column in slick
More Query from same tag
- Empty folder in HDFS using Spark Streaming+Kafka
- Working with Some() and Option() in Scala
- initialise a var in scala
- Akka stream batching
- Apache beam / Scala Combine perkey
- Getting error when instantiating a trait within a class (method became private)
- Serial consumption of Kafka topics from Spark
- Is there any action in RDD keeps the order?
- PlayFramework - How to use geolocation to redirect client to correct url based on country?
- Scala public becomes private when accessed from Java?
- Default generic type on method in scala
- Retrive subkey values of all the keys in json spark dataframe
- Scala Code — I fail to understand
- Unable to use for comprehension to resolve Future
- How to use JDBC to read datasets from Oracle?
- What is the best Scala thread-safe way to write to a BufferedWriter?
- How to Shuffle a List in Scala
- How to execute list of string SQL statements against a PostgreSQL db in Scala using doobie?
- How to avoid re-running an actor in Akka?
- How to prove type equality over an implicit?
- Security in Play 2.2.x
- Get request with Rapture Http
- Avro Serialization from generics
- Iterator of Strings to Inputstream of bytes
- Compiling Spark Scala Program into jar file using installed spark and maven
- Why does jersey doesn't pick up my PATH and give me a 404?
- Convert BigDecimal to BigInt Scala
- Read excel files with apache spark
- Enforce front-end validation of email field in PlayFramework 2.6 with Bootstrap 4
- Calling Polymorphic Scala Methods from Java