score:0
simply put, if you want to run spark locally (v2.4.x), you must specify hadoop as 2.6.5. you can use any version of the aws java sdk, but hadoop is specifically locked to that version. if you wish to circumvent this, it would be wise to upload files to s3 in one of 2 ways:
- from your jar, using
transfermanager
- with v1.11.600, there is a bug that is causing it to use
org.apache.httpcomponents.httpclient
4.5.8, which can cause logs to flood
- with v1.11.600, there is a bug that is causing it to use
- from a bash script using
aws s3 sync
<-- recommended
Source: stackoverflow.com
Related Query
- Unmodifiable view of a mutable Scala collection
- Can you sort a mutable Scala collection in place?
- NoSuchMethodError when attempting to implicitly convert a java to scala collection
- Yield an ArrayBuffer (or other mutable Collection type) from a for loop in Scala
- Convert a mutable collection to an immutable collection in Scala
- Scala mutable collection with sorting on insert
- HashCode in scala mutable collection
- Scala Seq[Seq[SomeClass]] type mismatch for mutable collection
- Converting from Java List to Scala mutable Buffer
- Mutable Map Collection in Scala can NOT update a single element
- Scala and akka switch to mutable data collection in this case?
- Scala best way of turning a Collection into a Map-by-key?
- Converting a Java collection into a Scala collection
- How to transform Scala collection of Option[X] to collection of X
- How are Scala collections able to return the correct collection type from a map operation?
- Scala - mutable (var) method parameter reference
- mutable vs. immutable in Scala collections
- Checking that all items in a collection match a predicate in Scala
- Converting mutable collection to immutable
- Which scala mutable list to use?
- Conversion from scala parallel collection to regular collection
- Google guava vs Scala collection framework comparison
- Scala collection standard practice
- Scala - Mutable thread safe collections
- What is the fastest way to sum a collection in Scala
- Type based collection partitioning in Scala
- How to use mutable collections in Scala
- Scala collection type for filter
- Scala Collection sorted, sortWith and sortBy Performance
- scala append to a mutable LinkedList
More Query from same tag
- Scala play with reactivemongo, not working
- How does the @main annotation in Scala 3 work?
- How to search for an Exchange appointment for an iCalUid using Java EWS API?
- how to fix task not serializable exception in sparkstreaming
- Is it possible to rewrite this function tail-recursively?
- How to proxy a call to a hashmap without losing the type
- Understanding of '_' (underscore) in Scala as a type variable
- Enable sbt plugin in IntelliJ IDEA 14
- convert RDD Array[Any] = Array(List([String], ListBuffer([string])) to RDD(String, Seq[String])
- Scala/Play: JSON serialization issue
- How do I access the fields of a link-table with Lift Mapper?
- spark-scala: download a list of URLs from a particular column
- Method returning Future[Unit] containing nested future that is expected to complete
- How to consolidate two json result into one dataframe in scala spark
- How can I group the a list into tuple grouped items in Scala?
- scala: idiomatic and functional way to add errors to a list of errors
- Play Scala Activator Server having errors while serving requests because of @Inject not working
- sparkSession.sparkContext is null with local spark cluster
- SBT Native Packager change output location of ZIP
- Produce an error with scala and spray
- Custom regex validation in Play Framework - Scala
- Inferred List Type with Either and Option
- Scala - How do I modify an input array passed to a method by reference?
- Scala Macros: Accessing members with quasiquotes
- How can I convert Long from String in scala
- How to get Map-like sugar in another constructor
- Scala: Set a value based on parameter-type
- Understanding `apply` and `unapply` with Singleton Type
- How to add a json object in to a json array using scala play?
- sys.env.getOrElse Throw Exception