score:0

to get your desired output rename some columns then join everything. you probably need to avoid records with the same start end.


  import org.apache.spark.sql._
  import org.apache.spark.sql.functions._

  val input: dataframe =
    ???
    
  val start: dataframe =
    input.select(
      col("name") as "startname",
      col("lat") as "startlat",
      col("long") as "startlong"
    )

  val end: dataframe =
    input.select(
      col("name") as "endname",
      col("lat") as "endlat",
      col("long") as "endlong"
    )

  start.join(end, col("startname") =!= col("endname"))


Related Query

More Query from same tag