score:1

Accepted answer

the most recent version of the emr at this time (may 2019) is 5.23.0 and it still uses spark 2.4.0 (https://docs.aws.amazon.com/emr/latest/releaseguide/emr-release-5x.html#emr-5200-release)

https://spark.apache.org/docs/2.4.0/ :

spark runs on java 8+, python 2.7+/3.4+ and r 3.1+. for the scala api, spark 2.4.0 uses scala 2.11. you will need to use a compatible scala version (2.11.x).

i believe non-experimental support for scala 2.12.x is only added in spark 2.4.3, which is not yet available on emr: https://spark.apache.org/docs/2.4.3/ :

spark runs on java 8+, python 2.7+/3.4+ and r 3.1+. for the scala api, spark 2.4.3 uses scala 2.12. you will need to use a compatible scala version (2.12.x).

score:3

emr 5.24.0 has spark 2.4.2 which supposedly uses scala 2.12 as a default, but aws still ships a spark version compiled against scala 2.11 only, by this time they should have at least provided a config flag to choose a spark version for scala 2.12


Related Query

More Query from same tag