score:0
the findmean
method is a method of the object motitordetection
. the object motitordetection
has a sparkcontext
on-board, which is not serializable. thus, the task used in rdd.map
is not serializable.
move all the matrix-related functions into a separate serializable object, matrixutils
, say:
object matrixutils {
def findmean(a: bdm[double]): bdv[double] = {
var c = mean(a(*, ::))
c
}
def tomatrix(x: bdv[double], y: bdv[double], c: int): bdm[double]={
val m = bdm.zeros[double](c,2)
m(::, 0) := x
m(::, 1) := y
m
}
...
}
and then use only those methods from rdd.map(...)
:
object motitordetection {
val sc = ...
def safezones(stream: dstream[(int, bdm[double])]){
import matrixutils._
... = rdd.map( ... )
}
}
Source: stackoverflow.com
Related Query
- Is using function in transformation causing Not Serializable exceptions?
- Task not serializable when using a function with map in Spark (Scala)
- Spark Task not serializable with lag Window function
- Is there a way to call a function defined using `val` in Scala with the whole curly brace block as an argument and not the final result of that block?
- Spark function not serializable
- Using apply ("()") on function passed in by-name parameter: evaluation is not forced?
- How can I return two DStreams in a function after using the filter transformation in spark streaming?
- Not Serializable Exception when using Scalaz with Spark
- Streaming from HBase using Spark not serializable
- DataFrame using UDF giving Task not serializable Exception
- Task not serializable while using custom dataframe class in Spark Scala
- Scala: why using `_` as a lambda expression body to be used as a function argument does not work?
- What is causing the sort Function serializable in spark?
- How to Convert Seq to Json using Circe inside a function - keep getting "implicit value not found" error
- Task not serializable - foreach function spark
- Spark dataframe missing columns when data are not available while using pivot function
- Applying Complex transformation in configurable manner using scala function and spark dataframe
- Task not serializable when moving simple function out of UDF
- org.apache.spark.SparkException: Task not serializable for date function
- Using variable second times in function not return the same value?
- Jupyter Scala notebook not recognizing Scala anonymous function using Java List
- Scala: Task not serializable when using closure
- Calling UDF function and get Task not serializable Exception
- LabelledGenerid.Aux implicit not available when using generic type in function body
- Using Class Methods in Spark RDD Operations Returns Task not serializable Exception
- Task not serializable issue at window rank function in scala spark
- How to aggregate sum on multi columns on Dataframe using reduce function and not groupby?
- Using rscala eval function for returning values to Scala not working
- How to use a function or method on a Spark data frame column for transformation using Scala
- use function or class of scala in spark: org.apache.spark.SparkException: Task not serializable
More Query from same tag
- How to sort a dataset by the hash of a column in Scala?
- Create child actors based on the request pattern
- Regex for getting the number after the last delimeter of the line in Scala
- Akka remote: is it possible to work with classes from two differents app which don't have the same package name?
- System.getenv not finding envrionment variables
- Can SparkContext.setCheckpointDir(hdfsPath) set same hdfsPath in different spark apps?
- Scala pairwise traversal of list elements
- How do I split a Spark rdd Array[(String, Array[String])] to a single RDD
- Parsing scalaz.Validation in Java
- Hidden features of Scala
- What does the number mean in "java.lang.ArrayIndexOutOfBoundsException: -1" in Scala?
- Changing appends programmatically
- underscore "class needs to be abstract since method is not defined" error
- Is LISP's code-as-data ideology basically the same thing as higher order functions?
- error while fetching data from column of type case class in spark scala
- Spark dataframe explode deprecation warning in 2.4.4
- converting to lowercase a username before info goes to controller
- Why spark (scala API) agg function takes expr and exprs arguments?
- Is there a Implicit value conflicts in Ordering[A] in scala's source code?
- Kotlin zipAll alternative
- Is there a reason why the akka local tcp port is set to 0?
- Should I report this assertion failure? What is $asInstanceOf anyway?
- Start external console application from Scala in interactive mode
- Exception after Setting property 'spark.sql.hive.metastore.jars' in 'spark-defaults.conf'
- Find average value from a column of stream dataframe with array values using spark scala
- Scala: java.lang.UnsupportedOperationException: Primitive types are not supported
- IntelliJ Scala cannot build a simple project (SBT can)
- What's the idiomatic way to take top n values according to some criteria?
- Swapping statements in for-comprehension in Scala shows strange behavior
- Am I Using (Scala) Generics Correctly?