put like this in your code

configuration configuration = new configuration(); configuration.set("fs.hdfs.impl",org.apache.hadoop.hdfs.distributedfilesystem.class.getname()); configuration.set("fs.file.impl",org.apache.hadoop.fs.localfilesystem.class.getname());


i had this problem when my maven repository contained corrupted jar files. same as you i could see the hadoop-common-x.x.x.jar existed in eclipse when viewing the "maven dependencies" of my java project. however when expanding the jar file in eclipse and selecting the class named org.apache.hadoop.fs.fsdatainputstream eclipse was reporting a message something like "invalid loc header".

deleting all files from my local maven repository and executing mvn install again resolved my issue


if you are using the configuration to run your app for debugging. make sure you have the checkbox checked for include dependencies with provided scope if you have any of the dependencies and you have mentioned its scope to provided. it worked for me by following this approach


my experience with eclipse ide : from package explorer

my basic path for ubuntu installation is usr/hadoop/hadoop-2.7.1 (lets' say conf) i've added two jar files,from conf/share/hadoop/common/lib and from conf/share/hadoop/common. and this is the java code (from the book hadoop in action) :

import org.apache.hadoop.conf.configuration;
import org.apache.hadoop.fs.fsdatainputstream;
import org.apache.hadoop.fs.fsdataoutputstream;
import org.apache.hadoop.fs.filestatus;
import org.apache.hadoop.fs.filesystem;
import org.apache.hadoop.fs.path;

public class putmerge {

public static void main(string[] args) throws ioexception {
        configuration conf = new configuration();


        org.apache.hadoop.fs.filesystem hdfs = org.apache.hadoop.fs.filesystem.get(conf);
        filesystem local = org.apache.hadoop.fs.filesystem.getlocal(conf);
        path inputdir = new path(args[0]);
        path hdfsfile = new path(args[1]);
        try {
            filestatus[] inputfiles = local.liststatus(inputdir);
            fsdataoutputstream out = hdfs.create(hdfsfile);
            for (int i=0; i<inputfiles.length; i++) {
                fsdatainputstream in =[i].getpath());
                byte buffer[] = new byte[256];
                int bytesread = 0;
                while( (bytesread = > 0) {
                    out.write(buffer, 0, bytesread);
        } catch (ioexception e) {

the solution for me was to export the .jar file from this code, and this what i did : right click on putmerge project, then export (from the pop-up menu) :

always from package explorer

and saved the jar file in a folder named putmerge on home/hduser directory from filesystem, putmerge.jar

in another folder named input (path /home/hduser/input) there are three .txt files as input for putmerge procedure : three input files

and now we are ready to launch the command from a terminal session : hadoop jar /home/hduser/putmerge/putmerge.jar putmerge /home/hduser/input output4/all

and the command /usr/hadoop/hadoop-2.7.1$ hdfs dfs -cat /output4/all

will contain all the text of the three single files.

Related Query

More Query from same tag