score:4
after troubleshooting some previous methods i had attempted before, i've come across the following fix:
in my pom.xml
i excluded the hadoop-client
dependency automatically imported by the spark-core
jar. this dependency was version 2.6.5
which conflicted with the cluster's version of hadoop. instead, i import the version i require.
<dependency>
<groupid>org.apache.spark</groupid>
<artifactid>spark-core_${scala.version.major}</artifactid>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupid>org.apache.hadoop</groupid>
<artifactid>hadoop-client</artifactid>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupid>org.apache.hadoop</groupid>
<artifactid>hadoop-client</artifactid>
<version>${hadoop.version}</version>
</dependency>
</dependency>
after making this change, i encountered the error java.lang.unsatisfiedlinkerror: org.apache.hadoop.io.nativeio.nativeio$windows.access0
. further research revealed this was due to a problem with the hadoop configuration on my local machine. per this article's advice, i modified the winutils.exe
version i had under c://winutils/bin
to be the version i required and also added the corresponding hadoop.dll
. after making these changes, i was able to successfully read data from blob storage as expected.
tldr
issue was the auto imported hadoop-client
dependency which was fixed by excluding it & adding the new winutils.exe
and hadoop.dll
under c://winutils/bin
.
this no longer required downgrading the hadoop versions within the hdinsight cluster or changing my downloaded spark version.
score:2
problem: i was facing same issue while running fat jar with hadoop 2.7 and spark 2.4 on cluster with hadoop 3.x , i was using maven shade plugin.
observation: while building fat jar it was including jar org.apache.hadoop:hadoop-hdfs:jar:2.6.5 which has class class org.apache.hadoop.hdfs.web.hftpfilesystem. which was causing problem in hadoop 3
solution: i have excluded this jar while building fat jar as below.issue got resolved.
Source: stackoverflow.com
Related Query
- Error through remote Spark Job: java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem
- MVCE with Apache Spark / Java Reflection Error on static constructor?
- apache spark streaming kafka integration error JAVA
- Using remote standalone cluster in local Spark application error "scala.Option; local class incompatible"
- Custom Accumulator Spark error using Java class type in mix Scala and Java Spark project
- Error while creating Wordcount project using Spark & Java on Eclipse in Cloudera through Vmware
- Apache Spark job calling java program and showing different behavior in RDD and Dataframe
- strange error message: bad symbolic reference. A signature in package.class refers to term apache in package org which is not available
- Case class equality in Apache Spark
- Spark case class - decimal type encoder error "Cannot up cast from decimal"
- Running a Job on Spark 0.9.0 throws error
- Apache Spark error while start
- Spark Dataframe to Dataset of Java class
- Error in running job on Spark 1.4.0 with Jackson module with ScalaObjectMapper
- Error in running Spark in Intellij : "object apache is not a member of package org"
- java.lang.NoClassDefFoundError: Could not initialize class when launching spark job via spark-submit in scala code
- Connecting to a remote Spark master - Java / Scala
- Apache Spark: ERROR local class incompatible when initiating a SparkContext class
- How can I obtain the DAG of an Apache Spark job without running it?
- Read file on remote machine in Apache Spark using ftp
- Apache Spark - How does internal job scheduler in spark define what are users and what are pools
- Unsupported literal type class in Apache Spark in scala
- Getting "how can getCommonSuperclass() do its job if different class symbols get the same bytecode-level internal name" compile error
- Spark Session returned an error : Apache NiFi
- RandomForestClassifier was given input with invalid label column error in Apache Spark
- Scala Spark : (org.apache.spark.repl.ExecutorClassLoader) Failed to check existence of class org on REPL class server at path
- Spark job fails on java 9 NumberFormatException for input string ea
- How to pass Scala DataSet to Java Class in Apache Flink
- Apache Spark MLLib - Running KMeans with IDF-TF vectors - Java heap space
- Maven build ERROR (Scala + Spark):object apache is not a member of package org
More Query from same tag
- What is the precedence of type operators (bounds) in scala?
- scala: How to convert an anonymous function to val
- Map Slick oneToMany results to tree format
- new Maven dependency not recognized
- Sort by multiple values with sortBy
- How to mix in two traits when instantiating a case class instance?
- Lazy vals and implicit parameters in Scala
- Sample: Calendar for Play2.0- scala
- Manifest and abstract type resolution
- Scala MapReduce Framework giving Type Mismatch
- Scala compilation into machine code
- Instead of Scala Map objects, can we use any other proper objects for below type of structure?(Case Class)
- Mapping case class with companion object
- Reading from Kafka with Scala Spark2 Streaming
- Java Zookeeper API weird ZNode behavior. Unable to delete ZNode properly. It has unexpected results
- How to paginate external calls while deserialise a Trait with type parameters using Spray?
- Scala: Type mismatch required _$1 for lower context bound generics
- scala rapids using an opaque UDF for a single column dataframe that produces another column
- Setting H20 Dependency in Intellij and run on spark-submit
- Resolving spark dependencies using sbt
- String to Timestamp Pyspark / Spark
- Change language of text in template in play framework 2.1.1
- Scala - How to parse many-one relationships?
- How to take out Nothing out of inferred type
- How to call a scala function taking a value of Void type
- Scala Covariance and Lower Type Bounds Explanation
- Partially applied generic function "cannot be cast to Nothing"
- scala spark use udf function in spark shell for array manipulation in dataframe column
- join spark df with list
- How do I compare abstract WeakTypeTag's unrepresented Generics