score:0
to get your desired output rename some columns then join everything. you probably need to avoid records with the same start end.
import org.apache.spark.sql._
import org.apache.spark.sql.functions._
val input: dataframe =
???
val start: dataframe =
input.select(
col("name") as "startname",
col("lat") as "startlat",
col("long") as "startlong"
)
val end: dataframe =
input.select(
col("name") as "endname",
col("lat") as "endlat",
col("long") as "endlong"
)
start.join(end, col("startname") =!= col("endname"))
Source: stackoverflow.com
Related Query
- How to compare the distance of every row from each other using lat long in Spark-scala
- How to compare one row with all other rows in spark using scala
- How to find the next occurring item from current row in a data frame using Spark Windowing?
- How to create a Row from a List or Array in Spark using Scala
- How to modify previous line in REPL - scala to modify the typing errors to save time compare to entering each line using up/down arrows
- How to compare every element in the RDD with every other element in the RDD ?
- How to assign a category to each row based on the cumulative sum of values in spark dataframe?
- How can find size of each Row in Apache spark sql dataframe and discrad the rows having size more than a threshold size in Kilobyte
- Using Spark on Dataproc, how to write to GCS separately from each partition?
- In spark streaming job, how to collect error messages from executors to drivers and log these at the end of each streaming batch?
- How to use withColumn with condition for the each row in Scala / Spark data frame
- How to Find the maximum consecutive years for each ID using Scala / Spark
- how to split row into multiple rows on the basis of date using spark scala?
- How to read Avro files (generated from Java class) using Spark shell when the source Java class is loaded?
- How to calculate the ApproxQuanitiles from list of Integers into Spark DataFrame column using scala
- How to access the last two character of each cell of Spark DataFrame to do some calculations on its value using Scala
- How to create a column of row id in Spark dataframe for each distinct column value using Scala
- how to concat multiple columns in spark while getting the column names to be concatenated from another table (different for each row)
- How to fill column with value taken from a (non-adjacent) previous row without natural partitioning key using Spark Scala DataFrame
- How can I detach from Mongo in Spark when using the Mongo Spark Connector?
- How do I compare each column in a table using DataFrame by Scala without caring what the column is?
- Using scala spark how to print just the response body value returned from a HTTP post call
- Getting the row count by key from dataframe / RDD using spark
- How to redirect Apache Spark logs from the driver and the slaves to the console of the machine that launchs the Spark job using log4j?
- how to merge the multiple columns in single columns using UDF and remove the 0 value row from the column in pyspark
- Unable to get the value of first column from a row in dataset using spark scala
- How to copy a file from local to s3 using spark as a single file with the given name?
- How to split text in each row when getting the data from kafka topic?
- How to filter Scala Spark DataFrame if row matches ID in another DataFrame and timestamp is below the other frames timestamp
- How to filter the data from Rdd and save it to text file using scala in spark
More Query from same tag
- Use a saved model to transform another data without fitting again Spark
- How to change code using Scala Parser Combinators to take operator precedence into account?
- sbt share dependencies across projects
- Send json part in play-ws multipart request
- SBT gives java.lang.NullPointerException when trying to run simple "Hello!" example script
- Scala Print Map in Order
- spark/scala drop row with nan in any column
- How to use "static final constant" in a Scala Constructor?
- What happens if you edit a file during compilation?
- How to Enable Source.Queue Backpressure
- Condition on rows content of dataframe in Spark scala
- Can I perform matching on a type parameter in Scala to see if it implements a trait?
- How to change all columns data types in StructType or ArrayType columns?
- Scala - Behaviour of hashCode & equals when using mutable HashSet
- Passing a jenkins build parameter to sbt plugin
- Is apply and unapply a constructor in scala
- PersistentFSM doesn't persist to journal
- How to create a map key from a variable?
- Scala: const method in covariant class
- What's the fundamental difference between those two constructs?
- Using UUID.randomUUID().toString for setting a key in request body giving issues- scala
- type mismatch in scala when using reduce
- Spark count parameter "mentions" in the rows
- Pass implicit parameter through multiple objects
- Scala on Mac - Cannot figure out how to run target?
- Overview of Scala method and function syntax
- What (exactly) are "First Class" modules?
- Implementing SQL logic via Dataframes using Spark and Scala
- Scala Skinny ORM Soft Deletion get deleted records
- Controlling spawning of actors in Akka who consume noticeable amounts of memory