score:1
Accepted answer
scala syntax for selecting column is different from python. try using parenthesis instead of square brackets:
result.withcolumn("convertedvalue", result("count")*result("converted")).show
score:1
you can use the operators as below:
import spark.implicits._
result.withcolumn("convertedvalue", $"count" * $"converted")
score:1
import spark.implicits._
result.withcolumn("convertedvalue", 'count * 'converted)
Source: stackoverflow.com
Related Query
- spark shell column multiplication and updating same dataframe
- Spark Scala: create and use column in the dataframe on the same line
- How to PartitionBy a column in spark and drop the same column before saving the dataframe in spark scala
- Extract emojis from Dataframe column and add them into a different Column of the same Dataframe Scala Spark
- get min and max from a specific column scala spark dataframe
- SPARK DataFrame: How to efficiently split dataframe for each group based on same column values
- Add Number of days column to Date Column in same dataframe for Spark Scala App
- spark dataframe trim column and convert
- Adding StringType column to existing Spark DataFrame and then applying default values
- how to join two DataFrame and replace one column conditionally in spark
- Split an Spark dataframe by some column values and then rotate each generated dataframe independently from the others
- Spark : Parse a Date / Timestamps with different Formats (MM-dd-yyyy HH:mm, MM/dd/yy H:mm ) in same column of a Dataframe
- Changing several Spark DataFrame column types, dynamically and configurable
- Spark Dataframe schema definition using reflection with case classes and column name aliases
- Updating Dataframe Column name in Spark - Scala while performing Joins
- Create a dataframe from a hashmap with keys as column names and values as rows in Spark
- Scala/Apache Spark Converting DataFrame column values and type, multiple when otherwise
- Extract a column value from a spark dataframe and add it to another dataframe
- Retrieve spark dataframe array column value and reuse it as a column name in a UDF
- Spark and Scala, add new column with value from another dataframe by mapping common key
- dataframe columns as key and column data as value group by id in spark scala
- Sequential Dynamic filters on the same Spark Dataframe Column in Scala Spark
- Spark Dataframe with pivot and different aggregation, based on the column value (measure_type) - Scala
- Fetch the partial value from a column having key value pairs and assign it to new column in Spark Dataframe
- Scala Apache Spark and dynamic column list inside of DataFrame select method
- scala spark dataframe explode is slow - so, alternate method - create columns and rows from arrays in a column
- Different sum between dataframe column and its array in spark scala
- spark merge datasets based on the same input of one column and concat the others
- Scala Spark - copy data from 1 Dataframe into another DF with nested schema & same column names
- Spark scala - parse json from dataframe column and return RDD with columns
More Query from same tag
- How to remove duplicate tuples using scala?
- Scala Cake Pattern and Dependency Collisions
- Is it possible to create a Flow with Akka-stream that can switch between 2 different inner Shapes?
- BigtableConfiguration v BigtableOptions when running Bigtable emulator
- The java "Object" Class equivalent in Scala
- scala list read value of given string
- Scala: Xtream complains object not serializable
- Is there a Spark built-in that flattens nested arrays?
- How to run batch hive queries in spark scala
- Array of case class as parameter T
- not found: https://repo1.maven.org/maven2/org/jetbrains/sbt-idea-shell_2.12_1.0/2017.2/sbt-idea-shell-2017.2.pom
- Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found
- Cast Numeric to Float in Scala
- In scala, how to find all occurrence of \\$[0-9]+ in a string and replace them
- Connecting to remote master on standalone Spark
- Dependency error when running a Spark application that instantiates a NiFi receiver
- Running a process as sudo in Scala
- How to load newer version of json4s in Spark 2.1 executor?
- What are the differences between Either and Option?
- Scala Type Mismatch with Nested function call
- Is it possible to define companion classes/modules in the Scala interpreter?
- Playframework: Call the controller's action by location.href through Javascript
- Scala using self as a Type
- smoothing array and keep the array length unchanged in Scala
- Scala + EclipseLink: JPA annotations treated as abstract traits, cannot be instantiated
- Is using Scala's Iterator ++ (concat) recursively stack-safe?
- Fork Join Pool vs Fixed Thread Pool for akka http client-focused workload?
- Play Scala Anorm One to Many relationship
- creating dataframe from nested json
- How to calculate z-score on dataframe API in ApacheSpark stucured streaming?