So, it all points to that after the file is inside JAR, it can only be accessed as a inputstream to read the chunk of data from within the compressed file.

I arrived at a solution, even though its not pretty it does what I need, that is to read a csv file, take the 2 first columns and make it into a dataframe and after load it inside a key-value structure (in this case i created a case class to hold these pairs).

I am considering migrating these lookups to a HOCON file, that may make the process less convoluted to load these lookups

import sparkSession.implicits._
val fileStream ="/lookup01.csv")
val input = sparkSession.sparkContext.makeRDD(

val myRdd = {
      line =>
        val col = utils.Utils.splitCSVString(line.getString(0))
        KeyValue(col(0), col(1))

val myDF = => (x.key, x.value)).collectAsMap()



You have to get the correct path from classPath

Considering that your file is under src/main/resources:

val path = getClass.getResource("/lookup01.csv")

val v_lookup = "header", true ).csv(path)

Related Query

More Query from same tag