score:1
you can use a udf to turn the start and end years into an array of all the years in the range, and then use explode
to turn each item in that array into a row:
import org.apache.spark.sql.functions._
val torange = udf { (y1: int, y2: int) => (y1 to y2).toarray }
input
.withcolumn("years", torange($"start_year", $"end_year"))
.select($"name", explode($"years") as "year")
Source: stackoverflow.com
Related Query
- Splliting rows in spark shell
- Spark shell - How can I retrieve rows from my dataset based on a time period or in between 2 given dates or two years
- Is there a way to take the first 1000 rows of a Spark Dataframe?
- Spark : how to run spark file from spark shell
- Flattening Rows in Spark
- Iterate rows and columns in Spark dataframe
- Spark shell command lines
- How to shuffle the rows in a Spark dataframe?
- How to use s3 with Apache spark 2.2 in the Spark shell
- Spark Shell "Failed to Initialize Compiler" Error on a mac
- Spark Scala Split dataframe into equal number of rows
- How do I filter rows based on whether a column value is in a Set of Strings in a Spark DataFrame
- Aggregate rows of Spark DataFrame to String after groupby
- Filtering rows based on column values in spark dataframe scala
- Collect rows as list with group by apache spark
- Apache Spark shell crashes when trying to start executor on worker
- Read ORC files directly from Spark shell
- Spark scala data frame udf returning rows
- Increase memory available to Spark shell
- Spark shell : How to copy multiline inside?
- Union does not remove duplicate rows in spark data frame
- How can I select a stable subset of rows from a Spark DataFrame?
- Scala case class ignoring import in the Spark shell
- Merging rows into a single struct column in spark scala has efficiency problems, how do we do it better?
- How to transpose/pivot the rows data to column in Spark Scala?
- How to log malformed rows from Scala Spark DataFrameReader csv
- Difference between two rows in Spark dataframe
- Spark remove duplicate rows from DataFrame
- Regrouping / Concatenating DataFrame rows in Spark
- Merge rows in a spark scala Dataframe
More Query from same tag
- Immutable copy-update in abstract class
- Writing a Custom Class to HDFS in Apache Flink
- Update Spark Streaming Variable During Execution
- Scala Curly braces with and without return statement
- How to create a Dataframe programmatically that isn't StringType
- How to pattern match types in Scala macros?
- Get certain key from a Map out of a List use scala
- How to remove duplicates from collection (without creating new ones in-between)?
- Play 2.0 Scala and async callbacks
- Odersky's Coursera Lecture: Why can't the type-checker infer value type Int?
- Iteratively update a scalaz Tree
- Scala macro to create field and method pointers
- How to Split a State Machine in multiple Classes or Traits in Chisel?
- How do you group elements from Enumerator[A] to Enumerator[Seq[A]]?
- Scala implicit final class
- Converting a spark dataframe that contains Vector as a feature into CSV file
- Play Framework (Scala) Template
- What is the equivalent log4j statement in scala for the below java code
- scala Future processing depth-first not breadth-first
- Why can I call += with a single parameter on a Scala Set?
- Weird behavior in Scala
- Scala Function syntax
- How to group messages by username?
- What is Event Driven Concurrency?
- Can I make json4s's extract method case insensitive?
- Create JSON from the List using Json4s
- Spark: Can I get the case class from a data set?
- java.lang.ClassCastException: models.Task cannot be cast to models.Task on Play Framework
- Returning subclass of parameterized type in Scala
- Cannot keep the scala application up and running on docker