score:1
package spark
import org.apache.spark.sql.sparksession
import org.apache.spark.sql.functions.col
object arraytocol extends app {
val spark = sparksession.builder()
.master("local")
.appname("dataframe-example")
.getorcreate()
import spark.implicits._
val inptdf = seq(seq("a", "b", "c"), seq("x", "y", "z")).todf("value")
val d = inptdf
.withcolumn("0", col("value").getitem(0))
.withcolumn("1", col("value").getitem(1))
.withcolumn("2", col("value").getitem(2))
.drop("value")
d.show(false)
}
// variant 2
val res = inptdf.select(
$"value".getitem(0).as("col0"),
$"value".getitem(1).as("col1"),
$"value".getitem(2).as("col2")
)
// variant 3
val res1 = inptdf.select(
col("*") +: (0 until 3).map(i => col("value").getitem(i).as(s"$i")): _*
)
.drop("value")
score:1
another way to create a dataframe ---
df1 = spark.createdataframe([(1,[4,2, 1]),(4,[3,2])], [ "col2","col4"])
output---------
+----+---------+
|col2| col4|
+----+---------+
| 1|[4, 2, 1]|
| 4| [3, 2]|
+----+---------+
Source: stackoverflow.com
Related Query
- How write code that creates a Dataset with columns that have the elements of an array column as values and their names being positions?
- i have wriiten a logout code in welcome.scala.html page but getting errors in that page how can i write the code for it
- How do I write a JSON Format for an object in the Java library that doesn't have an apply method?
- write a spark Dataset to json with all keys in the schema, including null columns
- How to implement a trait with a generic case class that creates a dataset in Scala
- How to extract the elements of an Array[String] that starts with particular value in Scala?
- how to delete the elements that have the same prefix String (2 to 5 Chars) in a List?
- How do I write a path matcher that matches any part of the remaining path with a regex?
- How do I explode multiple columns of arrays in a Spark Scala dataframe when the columns contain arrays that line up with one another?
- How to write a function that takes a list of column names of a DataFrame, reorders selected columns the left and preserves unselected columns
- Scala - how to make the SortedSet with custom ordering hold multiple different objects that have the same value by which we sort?
- What is the efficient way to create Spark DataFrame in Scala with array type columns from another DataFrame that does not have an array column?
- How to collect an array of HTML elements that have the same ID using jQuery?
- Understanding the code that creates a mutable.Map with mutable.MultiMap
- How to write pointcut with customized annotation to print the global variable value that used in a function?
- how to select all columns that starts with a common label
- How to parse a csv that uses ^A (i.e. \001) as the delimiter with spark-csv?
- How to write Scala 2.9 code that will allow dropping into an interpreter
- How to write a zipWith method that returns the same type of collection as those passed to it?
- Scala spark: how to use dataset for a case class with the schema has snake_case?
- How to have a Scala standalone application that uses the playframework libraries
- How can I write a function have a polymorphic return type based on the type argument of its type parameter?
- How to bundle bash completion with a program and have it work in the current shell?
- How can I improve Scala's type inference with type parameters that don't show up in the first parameter list?
- How to interact with the compiler in Scala code itself?
- how to write an sbt plugin to launch the app with an agent
- How can I write a function that performs a case statement with a generic type?
- How to write clear code with logging?
- How to make compiler check that 2 method arguments have the same type?
- How to write first class functions with context bounds, that can be called cleanly at use site?
More Query from same tag
- Apple postion in snake game scala
- sbt-android: unable to use scala-reflect on android: java.rmi.Remote not found
- Scala: How to check if at least one of the fields of a class is defined?
- Why can't I rename the import of @BeanProperty in Scala
- Object identity of listeners
- Human-readable XML output from Scala?
- Map Over the Keys/Values from an Option[Map]
- Reading Structs from a spark data frame
- Scala JSON If key matches value return string
- How do I retrieve the 2nd value from the last entry in a list of 2tuples?
- Can I write some of the code in Scala (using AndroidStudio)?
- Several case classes with the same behavior in Scala
- Writing JSON depending on instance type
- inputRadioButton - When options attribute contains HTML code
- Run a SBT task with arguments from command line
- Play Framework: How to serialize/deserialize an enumeration to/from JSON
- Return type from specs2 helper method
- How get Class[_ <: A]] with lower bound
- List of String implicit conversions like +=
- export many files from a table
- Why am I getting OutOfMemoryError compilation error in Scala?
- How run glue job locally?
- Flink job cant use savepoint in a batch job
- What is principle of Spark sample method with replacement?
- Scala try and catch Error
- Processing (OSM) PBF files in Spark
- Why does null.asInstanceOf[<some CPS annotated type>] fail?
- Defining multiple modules at once in the build definition
- Get max term and number
- Shapeless: Iterate over the types in a Coproduct