score:1
Accepted answer
you get this number of columns doesn't match
error because your erdf
dataframe contains only one column, that contains an array:
+----------------------------+
|value |
+----------------------------+
|[a, b, c, d, e, f, g] |
|[a2, b2, c2, d2, e2, f2, g2]|
+----------------------------+
you can't match this unique column with the seven columns contained in your header.
the solution here is, given this erdf
dataframe, to iterate over your header columns list to build columns one by one. your complete code thus become:
val spark = sparksession.builder.appname("er").master("local").getorcreate()
import spark.implicits._
val erresponse = response.body.tostring.split("\\\n")
val header = erresponse(0).split(", ") // build header columns list
val body = erresponse.drop(1).map(x => x.split(",").tolist).tolist
val erdf = header
.zipwithindex
.foldleft(body.todf())((acc, elem) => acc.withcolumn(elem._1, col("value")(elem._2)))
.drop("value")
that will give you the following erdf
dataframe:
+-------+-------+-------+-------+-------+-------+-------+
|column1|column2|column3|column4|column5|column6|column7|
+-------+-------+-------+-------+-------+-------+-------+
| a| b| c| d| e| f| g|
| a2| b2| c2| d2| e2| f2| g2|
+-------+-------+-------+-------+-------+-------+-------+
Source: stackoverflow.com
Related Query
- Converting dataframe columns into list of tuples
- null pointer exception while converting dataframe to list inside udf
- Converting List of List to Dataframe
- Converting List of List or RDD to DataFrame in Spark-Scala
- Converting a Spark Dataframe to a Scala Map collection list
- Converting a dataframe into a hashmap where Key is int and Value is a list in Scala
- Converting a dataframe column with values to a list using spark and scala
- Spark SQL converting a scala DataFrame into a column list
- Extract column values of Dataframe as List in Apache Spark
- How to create DataFrame from Scala's List of Iterables?
- Converting a Scala Map to a List
- Fetch Spark dataframe column list
- Adding a column of rowsums across a list of columns in Spark Dataframe
- Dropping multiple columns from Spark dataframe by Iterating through the columns from a Scala List of Column names
- How to add a column to Dataset without converting from a DataFrame and accessing it?
- Convert scala list to DataFrame or DataSet
- Converting multiple different columns to Map column with Spark Dataframe scala
- How can I print nulls when converting a dataframe to json in Spark
- Convert List into dataframe spark scala
- Scala Spark : How to create a RDD from a list of string and convert to DataFrame
- Scala, converting multiple lists to list of tuples
- Converting a list of either to a cats ValidatedNel
- Converting a vector column in a dataframe back into an array column
- How do I groupby and concat a list in a Dataframe Spark Scala
- Converting pattern of date in spark dataframe
- Spark DataFrame filtering: retain element belonging to a list
- How to groupBy an iterator without converting it to list in scala?
- Converting a Heterogeneous List to and from Json in Play for Scala
- In Scala - Converting a case class List into a List of tuples
- Count of List values in spark - dataframe
More Query from same tag
- How to fallback Scala version for SBT dependencies?
- complete a future within other future
- Empty Dataset Returned while performing Join
- In scala shapeless, how to use Record.updateWith inside another method?
- Option operators with access to Option container
- Fastest way to reduce a list of objects
- Scala existentials placeholder translation for M[_,_] where M[X,Y <: N[X]]
- How to specify schema for CSV file without using Scala case class?
- How do I instantiate a Scala case class using all the attributes of a smaller case class?
- What is the #:: operator in a scala Stream?
- Build, using Shapeless, generic default instances for case classes with parameters defining a common createValue method
- Scala reflection to get List field value by universe apply
- Creating a cookie if it doesn't exist inside of my Action that I created with ActionBuilder
- When are classes extending AnyVal not unboxed?
- when do we need to close MongoConnection
- akka camel sending case classes as messages
- What is the proper way of using coffeescript in play framework?
- How to convert map to dataframe?
- Configuration error[Application secret not set]
- How to count number of total items where a class references itself
- Boostrap css and JS not behaving as expected with Play framework
- Why does the sbt ~compile command do not recompile on code change?
- Generic method type erasure and pattern matching
- Converting List to Map using keys in Scala
- Spray routing works for single slash but nothing else
- How can I send messages to a remote actor via CLI with Akka remoting?
- type erasure and Inheritance in scala
- How to pass in session variable to a form in play framework/scala to be stored in database
- Cannot implement representation type as type member
- Not able to write csv file by taking data from kafka topic in spark Streaming with scala