score:0
Accepted answer
turns out the problem was in my creation of the warc files that i was using,
val warcs = sc.newapihadoopfile(
warcfile,
classof[warcgzinputformat], // inputformat
classof[nullwritable], // key
classof[warcwritable] // value
).cache()
turns out removing .cache()
stops the exceptions. i don't know why though, so an explanation would still be welcome.
Source: stackoverflow.com
Related Query
- How can you view the result of RDD.join() in Scala?
- How can I access the last result in Scala REPL?
- How can I view the code that Scala uses to automatically generate the apply function for case classes?
- How can I use the last result from a scala map as input to the next function?
- How can I use aggregate with join in the same query result with Spark?
- How can I filter a spark RDD by the result of mapping?
- How to create new commands that you can run after running the sbt command Scala
- in Scala how do you join 2 RDD
- in Scala using RDD , how do you get apply function in the Iterable if RDD[(k,Iterable[v])
- How do I view the type of a scala expression in IntelliJ
- Scala how can I count the number of occurrences in a list
- How do I replace the fork join pool for a Scala 2.9 parallel collection?
- How can I idiomatically "remove" a single element from a list in Scala and close the gap?
- How can I syntax check a Scala script without executing the script and generating any class files?
- How can I get the actual object referred to by Scala 2.10 reflection?
- Can you override the stream writers in scala @serializable objects?
- Can anyone explain how the symbol "=>" is used in Scala
- How can I invoke the constructor of a Scala abstract type?
- How can an implicit be unimported from the Scala repl?
- Using Scala 2.10 reflection how can I list the values of Enumeration?
- How to show images using Play framework and Scala in the view page
- How do you uninstall the Scala Eclipse plugin?
- How can one list all csv files in an HDFS location within the Spark Scala shell?
- How can you get ScalaFX to play nice in the SBT console?
- How can I find the version of Scala installed in Eclipse IDE?
- How do you attach the Scala Intellij debugger for tests?
- scala: how to create a generic type which is subtype of all the number classes in scala so that it can include compare method
- How can I add scala actors to an existing program without interfering with the normal termination behavior?
- How can you make custom function types in Scala with named parameters?
- How can I keep -Xcheckinit from interfering with the deserialization of Scala objects?
More Query from same tag
- Instantiate generic case class without "new"
- What's the difference between shouldBe vs shouldEqual in Scala?
- Appending a label to immutable case classes in Scala
- scala or java library that will read .ssh/config
- akka-http + spray-json client side json marshalling
- Caused by: java.lang.IllegalArgumentException: Can't get JDBC type for null
- Unable to infer SQL type for my user-defined function which return BigInt
- Scala JsoupBrowser set UserAgent
- Scala: expanded syntax of trampolining function breaks tail recursion
- Unit testing of tls enabled in akka http
- how to generate spark time series data
- Abstract types puzzler
- Spray testing basicauth from js html
- LazyList in Scala
- Work on list of tuples in Scala - part 4
- How to split column into multiple columns in Spark 2?
- Scala unpickle with missing field
- Scala List with only subclasses
- Behaviour of Lag function in scala when the column is null
- Exiting Spark-shell from the scala script
- Extracting an object from a nested list of objects
- flatMap in Generator from online scala course by Martin Odersky
- Functional programming applied
- What are the tradeoffs, besides being more or less functional, in using a var List or val mutable.List
- Migrating from Maven to SBT
- scala.collection.mutable.PriorityQueue: Comparing with 2 or more attributes of a case class
- Read local/linux files in Spark Scala code executing in Yarn Cluster Mode
- How to find unique elements from list of tuples based on some elements using scala?
- Scala group Stream elements without evaluating the whole Stream
- Strange pattern matching behaviour with AnyRef