score:1
Not tested but should work:
import spark.implicits._
import org.apache.spark.ml.linalg.Vector
df.as[(String, Vector)].flatMap {
case (j, ps) => ps.toArray.zipWithIndex.map {
case (p, ti) => JournalDis(j, ti, p)
}
}
Source: stackoverflow.com
Related Query
- explode a row of spark dataset into several rows with added column using flatmap
- Split Map type column with huge values into multiple rows using Scala and Spark
- Mapping Spark DataSet row values into new hash column
- Spark dataframe select rows with at least one null or blank in any column of that row
- Spark / Scala: Split row into several rows based on value change in current row
- Attach column names to elements with Spark and Scala using FlatMap
- Convert row values into columns with its value from another column in spark scala
- Rolling up multiple rows into a single row and column in spark
- How to compare one row with all other rows in spark using scala
- Transforming all new rows into new column in Spark with Scala
- transform the RDD with list column , into multiple rows in Spark
- Compare rows of an array column with the headers of another data frame using Scala and Spark
- how to split row into multiple rows on the basis of date using spark scala?
- How to split column into multiple rows using Spark JavaRDD
- Spark SQL using a window - collect data from rows after current row based on a column condition
- Split single row into two rows in a spark dataframe using scala
- How to fill column with value taken from a (non-adjacent) previous row without natural partitioning key using Spark Scala DataFrame
- How to create a new column for dataset using ".withColumn" with many conditions in Scala Spark
- Converting row into column using spark scala
- How to explode column with multiple records into multiple Columns in Spark
- Spark combine multiple rows to Single row base on specific Column with out groupBy operation
- Apache Spark concatenate several rows into a list in one row
- Spark dataframe - Replace tokens of a common string with column values for each row using scala
- Unable to get the value of first column from a row in dataset using spark scala
- Spark dataframe concat multiple columns into one with optional columns in some row using Java
- How to pass dataset column value to a function while using spark filter with scala?
- Spark Scala - How to explode a column into multiple rows in spark scala
- Split one row into several rows in Spark Scala
- Encode an ADT / sealed trait hierarchy into Spark DataSet column
- How to convert a string column with milliseconds to a timestamp with milliseconds in Spark 2.1 using Scala?
More Query from same tag
- wrapping different datatype into a generic, in scala
- Better type checking on match in Scala
- Spark: programmatically populate columns from array values
- How to define partially parameterize generic implicit class?
- Serializing sequences of AnyVal with json4s
- PlayFramework java.lang.OutOfMemoryError: unable to create new native thread
- Different field instances in class and parent/Call super constructor with method
- Word will not be added to dictionary
- How to add a sort condition to a Spark Dataframe
- Counting lines of a file in Scala
- Running my program leads to java.lang.NoSuchMethodError: scala.Predef$.augmentString(Ljava/lang/String;)Lscala/collection/immutable/StringOps;
- Get largest Map from an RDD of Array of Maps
- scala: enforce immutable type in closure
- How to add to scala.collection.mutable.Buffer
- Extending Seq.sortBy in Scala
- PartialFunction in Scala
- How to sort an RDD of tuples with 5 elements in Spark Scala?
- scala string interpolation with a variable length
- Scala HashMap: iterate in insertion order?
- ScalaJson: Traversing JSValue structure (JSONPath syntax) where key might be one of two different strings
- Config RestEasy service using Spring
- flatMapping in scala/spark
- How to put data from various tables into separate lists via one request
- Parse Json using Circe and monocle lens
- program hangs when using multiple futures with multiple remote actors
- Read JSON Tree structure in Scala Play Framework
- spark shell (spark 2.4, scala 2.11.12 ) can not recognize imported class
- Best way to convert a Java Future to a Twitter Future
- OPAL: Manually creating an annotated method
- Play Framework 2.3 handling Future in Controllers