score:0
you are missing the configuration for your spark driver
. this causes the operator to not be able to run any workloads, and ultimately all the processes are hogged up. you need to take your spark driver configuration, convert into a configmap
resource and make sure this configmap
is applied to the same namespace as your spark driver pods are running before you run any pods.
the pod scheduler makes sure that any volumes
, secrets
and configmaps
etc are available before it schedules the pods to run, that's why you're observing this.
here is an example from google's spark operator
repository on how to write a spark driver configuration as a configmap
.
simply take all the --conf
parameters from your command and change them to a configmap
and apply it. it should work.
note: since the pod controller will poll for the configmap
availability periodically on its own, once you apply the configmap
, with the correct name, the pod should start working.
Source: stackoverflow.com
Related Query
- Spark Driver Pod Getting Stuck in init state due to no driver configmap found
- java.sql.SQLException: No suitable driver found when loading DataFrame into Spark SQL
- Spark Scala getting class not found scala.Any
- Apache Spark stuck in "loading" state
- spark scala : jnibasedunixgroupsmapping: error getting groups for yyy: the user name could not be found
- Why is my spark job getting stuck when collect() is called?
- reduceByKey method not being found in Scala Spark
- Why so many tasks in my spark job? Getting 200 Tasks By Default
- Getting the value of a DataFrame column in Spark
- Spark Streaming mapWithState seems to rebuild complete state periodically
- spark error RDD type not found when creating RDD
- Apache Spark 2.0: java.lang.UnsupportedOperationException: No Encoder found for java.time.LocalDate
- SPARK : failure: ``union'' expected but `(' found
- Spark Scala : Getting Cumulative Sum (Running Total) Using Analytical Functions
- Spark driver disassociated and removed by the master
- Spark job is failed due to java.io.NotSerializableException: org.apache.spark.SparkContext
- Spark Streaming historical state
- Geting messages of Offset is getting reset in structured streaming mode in Spark
- Spark SQL's Scala API - TimestampType - No Encoder found for org.apache.spark.sql.types.TimestampType
- Getting HDFS Location of Hive Table in Spark
- Using groupBy in Spark and getting back to a DataFrame
- "error: type mismatch" in Spark with same found and required datatypes
- The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached
- Spark JoinWithCassandraTable on TimeStamp partition key STUCK
- spark 2.2 cache() cause the driver OutOfMemoryerror
- log from spark udf to driver
- spark implicit encoder not found in scope
- Getting covariance matrix in Spark Linear Regression
- No suitable driver found for jdbc:mysql when running from single jar
- Launching Apache Spark SQL jobs from multi-threaded driver
More Query from same tag
- Unable to write method in Scala working both for Double and Float
- Convert DStream[Double,Double] to RDD[(Double, Double)]
- Scala pattern match json
- Using berkeley hardfloat
- Getting the number of rows in a Spark dataframe without counting
- Scala REPL. First it says "illegal start of definition", then "error: not found: type" when I try it a different way
- Selectively disable subsumption in Scala? (correctly type List.contains)
- Unable to read variable annotations from Scala macro
- How to implement instance sharing for case classes
- Is it possible to decorate methods in scala?
- How to test file upload in Play 2.2.2?
- GraphX Pagerank Implementation
- playframework wildcard matching
- Aerospike - Scans with “done(abandoned-unknown)” status
- How to convert scala immutable map to JSON using Java?
- indexOf for 2D Vector in Scala
- Finding the first child of an XML Elem in Scala
- Gatling session variable is returning same value in a loop
- Filling gaps in timeseries Spark
- Spark caching difference between 2.0.2 and 2.1.1
- Spark - use dataframe many times without many unloads
- Scala RAD tools
- java.lang.NoSuchMethodError: main when starting HelloWorld with Eclipse Scala plugin
- Scala 2.10 Futures with readLine in SBT
- Why val is stable identifier and def is not?
- Parser Combinators: Does repsep allows back-tracking?
- How to let Spark parse a JSON-escaped String field as a JSON Object to infer the proper structure in DataFrames?
- Testing actor crash using TestActorRef
- Not able to locate element when running chrome in headless mode
- SSL in a REST Lift project, where to start?