score:1
Accepted answer
the main thing here is that generate a query
list
from given map
that you can do like below
//input df
val df=seq(("john","non hf","new york"),("steav","non hf","mumbai"),("ram","hf","boston")).todf("srccolumnz", "srccolumny", "srccolumnr")
//input list
val maplist=list(map("targetcolumn" -> "columnnamex", "sourcecolumn" -> list("srccolumnx", "srccolumny", "srccolumnz", "srccolumnp", "srccolumnq", "srccolumnr")), map("targetcolumn" -> "columnnamey", "sourcecolumn" -> list("srccolumny")), map("targetcolumn" -> "columnnamez", "selectvalue" -> 5))
//get all the columns of df as list
val dfcols=df.columns.tolist
//then generate query list like below
val query = maplist.map { mp =>
if (mp.contains("sourcecolumn")) {
val srccolumn = mp.getorelse("sourcecolumn", "sourcecolumn key not found").tostring.replace("list(", "").replace(")", "").split(",").map(_.trim).tolist
val srccol = srccolumn.filter(dfcols.contains(_)).head
df.col(srccol.tostring).alias(mp.getorelse("targetcolumn", "no target column found").tostring)
} else {
lit(mp.getorelse("selectvalue", "no target column found").tostring.replace("(", "").replace(")", "").trim).alias(mp.getorelse("targetcolumn", "no target column found").tostring)
}
}
//finally , fire the query
df.select(query:_*).show
//sample output:
+-----------+-----------+-----------+
|columnnamex|columnnamey|columnnamez|
+-----------+-----------+-----------+
| non hf| non hf| 5|
| non hf| non hf| 5|
| hf| hf| 5|
+-----------+-----------+-----------+
Source: stackoverflow.com
Related Query
- How to create dataframe from list[Map] based on condition
- How to create dataframe from list[Map] based on condition
- How do I nest data from one Spark dataframe in another dataframe based on a condition
- how to create an extra column in DataFrame based on a simple condition in spark
- How to create DataFrame from Scala's List of Iterables?
- How to create a DataFrame from a text file in Spark
- How to create Spark Dataset or Dataframe from case classes that contains Enums
- how to create DataFrame from multiple arrays in Spark Scala?
- How can I create a Spark DataFrame from a nested array of struct element?
- Scala Spark : How to create a RDD from a list of string and convert to DataFrame
- How to create a Dataframe from existing Dataframe and make specific fields as Struct type?
- How to create a distributed sparse matrix in Spark from DataFrame in Scala
- How to write a condition based on multiple values for a DataFrame in Spark
- Spark - Scala - Remove Columns from a dataframe based on condition
- Create a Map from the elements of the same set based on condition in Scala
- How to update Spark dataframe based on Column from other dataframe with many entries in Scala?
- How to retrieve a subset of rows from DataFrame based on condition?
- How to count number of rows in a spark dataframe based on a value (primary key) from another dataframe?
- How to create a dataframe from Array[Strings]?
- How would I extract numbers from column names based on a match condition in spark scala?
- how to create a dataframe based on the first appearing date and based on additional columns each id column
- How do I create a 3 column dataframe from 3 ListBuffers in Scala?
- How to create new columns in dataframe using Spark Scala based on different string patterns
- How to create spark dataframe from list
- How to create json object from dataframe column of type map(key:Int,value:struct)
- how to create multiple list from dataframe in spark?
- How to create DataFrame from fixed-length text file given field lengths?
- How to create a dataframe from a list of Scala custom objects
- How to create a DataFrame from List?
- Spark DataFrame - How to partition the data based on condition
More Query from same tag
- How to post tweets using scribe?
- Scala testing mocking implicit parameters?
- Is it possible to execute tasks on the Namenode?
- How to add/append a new row to a DataFrame in Scala without using a SQL insert?
- How to run the Scala code in Eclipse
- How to log a variable onlineID to Gatling logs
- Serialisation/deserialisation for sealed traits with Upickle
- Can't access extra methods of trait implementation
- How to add object to unparameterized Java List?
- Is there a better way to pick the first element of a List in Scala
- Object not serializable error while reading multiple AVRO files
- Scala type mismatch while doing pattern matching
- How can I add my parentheses back into my expression when printing?
- Using Spark-Submit to write to S3 in "local" mode using S3A Directory Committer
- Why are traits instantiable?
- scala play action composition response header
- Scala: Reading data from scylla throws exception
- Play2 and Flash Scope, why can't I print my message?
- Scala import class in same folder
- Akka http losing sender reference
- how to read CSV file in scala
- How to get nested json object/value in scala
- How Scala Traits behave?
- Spark select column based on row values
- Not found: object system
- Can I use a type bound on a Scala abstract method and then "tighten up" the definition in a subclass?
- Efficient copy of a Map to a new Map in Scala
- Variance under Inheritance scala
- Option is both a functor and a monad?
- Scala CQRS Framework