Do you need all 55 columns ?
You can create a case class that just holds that columns you need and save this subset.
Of course you keep the original file that has all the data if you need that in the future.
You are getting the Tuple50 error because you are hitting the 22 Tuple limit in Scala - see Why does the Scala library only defines tuples up to Tuple22?
- Remapping columns from a schemaRDD
- Derive multiple columns from a single column in a Spark DataFrame
- Select Specific Columns from Spark DataFrame
- Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the referenced columns only include the internal corrupt record column
- Create array of literals and columns from List of Strings in Spark
- Dropping multiple columns from Spark dataframe by Iterating through the columns from a Scala List of Column names
- Reading JSON files into Spark Dataset and adding columns from a separate Map
- How to get date from different year, month and day columns in spark (scala)
- How to remove backslash from all columns in a Spark dataframe?
- How to explode StructType to rows from json dataframe in Spark rather than to columns
- Create a map column in Apache Spark from other columns
- How to get columns from an org.apache.spark.sql row by name?
- adding two columns from a data frame in scala
- Selecting several columns from spark dataframe with a list of columns as a start
- Spark sum columns from different dataframes
- Remove constant columns from an RDD and compute the covariance matrix
- Convert row values into columns with its value from another column in spark scala
- How does $ symbol working when selecting columns from DataFrame?
- Spark - Scala - Remove Columns from a dataframe based on condition
- How to read from textfile(String type data) map and load data into parquet format(multiple columns with different datatype) in Spark scala dynamically
- using spark to read specific columns data from hbase
- Create separate columns from array column in Spark Dataframe in Scala when array is large
- How to select columns that exist in case classes from DataFrame
- Create SOAP XML REQUEST from selected dataframe columns in Scala
- Update column value from another columns based on multiple conditions in spark structured streaming
- Remove Null from Array Columns in Dataframe in Scala with Spark (1.6)
- How to groupby columns from two dataframes and then apply aggregate difference function between rows?
- Breeze extract columns from DenseMatrix based on list
- Slick 3.2: Filtering on columns from left-joined table
- How to drop multiple columns from JSON body using scala
More Query from same tag
- What class is this Builder pattern extending?
- Window Overload method cannot resolve in spark structured streaming-scala
- How do I get color coded console output from SBT on OSX?
- How to write a reads method of a trait
- Scala: groupBy (identity) of List Elements
- Syncing speed of reading from DB and Writing to elasticsearch using Akka grpc stream
- mocking a request with a payload using wiremock
- Why do we need both Future and Promise?
- How to know which implicit fails to be resolved?
- How to get the full URL from an Http4s Request
- Using scala.collection.immutable.Map inside my java class
- Scala Circular Map
- A hard way to get rid of everything generated by sbt
- Traversing lists and streams with a function returning a future
- How to get datatype of column in spark dataframe dynamically
- How can I change version of Scala that is used by Play, SBT and its plugins?
- Running sbt inside Intellij got wrong bytecode version
- Spark program stucks because of Jedis Pool
- Joining 2 RDDs when one having a Option type as key
- "Stable identifer required" error during companion object import
- how can I pass KafkaAvroSerializer into a Kafka ProducerRecord?
- Storing Scala data structures on disk
- Scala split string on whitespace excluding certain sections
- Function analog for .map for a collection that changes during processing?
- How to get substring using patterns and replace quotes in json value field using scala?
- How does sbt integrate with IntelliJ?
- How get return value when using actor in akka-stream
- Scala implicit conversion blocked somehow
- Scala Breeze does not find my file on a webserver, while Java does
- How can I resolve conflicting actor systems while testing akka-http and akka actors at the same spec file?