score:3
Accepted answer
- parse the JSON using
from_json
- cast it to
MapType(StringType, StringType)
- extract only values using
map_values
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types.{MapType, StringType}
movieDF.select("wiki_mv_id","mv_nm","mv_genre")
.withColumn("genre_frmttd",map_values(from_json(col("mv_genre"),MapType(StringType, StringType))))
.show(1,false)
Source: stackoverflow.com
Related Query
- Fetch all values irrespective of keys from a column of JSON type in a Spark dataframe using Spark with scala
- Create a dataframe from a hashmap with keys as column names and values as rows in Spark
- Spark aggregate keys from JSON input files and output values to a new JSON file
- spark withColumn value generation from all column values
- How to merge all unique values of a spark dataframe column into single row based on id and convert the column into json format
- Replace values from a nested json string column in Spark dataframe
- Retrive subkey values of all the keys in json spark dataframe
- How to get keys and values from MapType column in SparkSQL DataFrame
- Transform all keys from `underscore` to `camel case` of json objects in circe
- write a spark Dataset to json with all keys in the schema, including null columns
- Map column values to a a numeric type in spark
- How to remove keys with null values from Argonaut Json objects
- Spark SQL: Select with arithmetic on column values and type casting?
- How to efficiently find distinct values from each column in Spark
- Split an Spark dataframe by some column values and then rotate each generated dataframe independently from the others
- Sum up into a new column from a range of values of a column in Spark using Scala
- Spark Dataframe - How to get a particular field from a struct type column
- Spark DataFrame - drop null values from column
- Convert row values into columns with its value from another column in spark scala
- how to extract the column name and data type from nested struct type in spark
- Lookup values from a MapType column with keys from another column
- Spark - Get from a directory with nested folders all filenames of a particular data type
- Aggregating all Column values within a Map after groupBy in Apache Spark
- Using lag function in Spark Scala to bring values from another column
- How to select Keys from Json Object{} (complex data type) DataFrame in Spark
- Spark get all rows with same values in array in column
- Spark Dataframe has duplicated values in all rows when adding column with random double
- Create new column in Spark DataFrame with diff of previous values from another column
- How to get column values from list which contains column names in spark scala dataframe
- Get top values from a spark dataframe column in Scala
More Query from same tag
- How to obtain coefficient values from Spark-MLlib Linear Regression model (Scala)?
- Is ther any way to check what is the actual corrupt record in a json file
- Spark Graphx : class not found error on EMR cluster
- Conflicting method when paramter is limited to AnyRef and AnyVal in Scala
- In Lift, how do you bind to instances instead of classes for comet?
- Actor Thread Safety
- Don't return same type from filter for efficiency
- How do you unit test Scala in Eclipse?
- Scala Future and Either Transformation
- how to detect duplicated line using scala akka stream
- How to call T eq(Object) method of Java interface from Scala?
- NotSerializableException with Neo4j Spark Streaming Scala
- Forward reference extends over definition of value problem
- Reason for Scala's Map.unzip returning (Iterable, Iterable)
- PlayFramework ScalaTest Testing webpages with selenium
- scala rdd flatmap to generate multiple row from one row to en-fill gap of rows issue
- Is the play.api.cache.Cache.getOrElse function Thread safe?
- Serving local images with Play 2 to be displayed inline
- Type mismatch with If statement containing bitwise operators
- Running ApsctJ in Intellij IDEA with Scala
- Scala flatMap filter elements in array instance of Type
- Performance of typeclasses in Scala
- ClassNotFoundException when reading object
- Use implicit value from one module in another in Scala/Spark
- How to access object in a list - Scala
- Match list with two values in Scala
- Parse HTML in Scala
- using Either to parse failure and success
- ERROR SparkContext: Error initializing SparkContext(run mnist example with tensorflow on spark)
- Spark 2.1.0, cannot resolve column name when doing second join