score:10
Accepted answer
Indeed, the VectorAssembler Transformer does not take strings. So you need to make sure that your columns match numerical, boolean, vector types. Make sure that your udf is doing the right thing and be sure that none of the columns has StringType.
To convert a column in a Spark DataFrame to another type, make it simple and use the cast() DSL function like so:
val analysisData = dataframe_mysql.withColumn("Event", dataframe_mysql("Event").cast(DoubleType))
It should work!
Source: stackoverflow.com
Related Query
- VectorAssembler does not support the StringType type scala spark convert
- What is the efficient way to create Spark DataFrame in Scala with array type columns from another DataFrame that does not have an array column?
- Why does this Scala function compile when the argument does not conform to the type constraint?
- Why does Scala warn about type erasure in the first case but not the second?
- Why does Scala maintain the type of collection not return Iterable (as in .Net)?
- Scala with spark - "javax.servlet.ServletRegistration"'s signer information does not match signer information of other classes in the same package
- Why does Scala not infer the type parameters when pattern matching with @
- spark write: CSV data source does not support null data type
- Why does the Scala compiler give "value registerKryoClasses is not a member of org.apache.spark.SparkConf" for Spark 1.4?
- Why does mapping over a Scala enumeration's ValueSet not change the type of the values?
- Why the following Scala code does not compile unless explicit type parameters are added?
- How does the Scala program compile, given that bindAndHandle method takes the first parameter of type Flow not Route?
- Spark scala - convert an array into values from the same table in a Hierarchy type table
- Scala type inference does not work with type bounds, unless the type is specified explicitly
- Spark Scala VectorAssembler IllegalArgumentException: XX does not exist. Available: number
- Why does scala not infer the right type when wrapping EitherT?
- getOrElse for an Option(null) does not return a None type or the default value when null in Scala
- Any reason why scala does not explicitly support dependent types?
- What special rules does the scala compiler have for the unit type within the type system
- Why does Haskell's foldr NOT stackoverflow while the same Scala implementation does?
- Why does Scala choose the type 'Product' for 'for' expressions involving Either and value definitions
- Scala Spark contains vs. does not contain
- What is the Scala type mapping for all Spark SQL DataType
- In Scala static value initialization does not appear to be happening before the main method is called
- Why does scalac not believe that a method does not fit the required type signature?
- @throws in Scala does not allow calling Java to catch correct exception type
- How does the Scala type checker know to prevent calling flatten on a List that is already flat
- Why is my Scala function returning type Unit and not whatever is the last line?
- Why scala does not unify this type lambda with underlying type?
- Why does the Scala compiler say that copy is not a member of my case class?
More Query from same tag
- How to group objects using a classifier function in FS2?
- Memory Issues in Akka/Spray application
- Xtend: Is there a counterpart to the Scala singleton object in Xtend
- Scala Object From JSON Request
- Count spaces between words in a string
- How to install scodec library in scala/sbt?
- start/stop lifecycle of application with sbt-native-packager
- Lift - JE.Call() question
- How to reference a val in a case statement?
- Breeze extract columns from DenseMatrix based on list
- Unable to add a breakpoint in IntelliJ 15
- Recursive function not returning Int
- Is an intermediate List eligible for Garbage Collection when toStream is used?
- Spark: Different output with different number of cores
- docker container based library to support elastic4s
- Pure functional Random number generator - State monad
- Scala SBT throwing IncompatibleClassChangeError exception
- How to generate rules from a string values in scala?
- Reading result of Formula Cells in Apache POI 3.15
- Scala suffix notation is compiling
- how to make the element contains '$' char as key, and behind the key's numbers as a List?
- Cannot run jar file created from Scala file
- How do I combine Playframework evolutions with Slick Tables autogeneration?
- Any command in sbt shell induces `expected ';'` error
- Porting python-twisted based code to scala: framework advice needed
- How to use Mockito to create a mock api in scala
- Scala / Cats: How to unzip an NonEmptyList
- Using scala Maps to accumulate values
- Spark Excel Write options
- Creating a process from a queue in scalaz
