Accepted answer

You can read as a text file and get the path of file as below.

import org.apache.spark.sql.functions.input_file_name
val spark = SparkSession
  .appName("Test App")
import spark.implicits._

val data ="/parent_dir/*")
  .select(input_file_name().as("path"), $"value")

Now you get the data as dataframe that consist of path of file and data as

|path                            |value  |

Now you can parse the path and get only the required directory.

Hope this helps!

Related Query

More Query from same tag