Java HDFS打包成client jar

一、首先利用Maven下載相依套件，包括hadoop-common、hadoop-hdfs、jdk.tools

<dependencies>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.6.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-hdfs</artifactId>
        <version>2.6.0</version>
    </dependency>
    <dependency>
        <groupId>jdk.tools</groupId>
        <artifactId>jdk.tools</artifactId>
        <scope>system</scope>
        <version>1.6</version>
        <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
    </dependency>
</dependencies>

二、打包的部分：

<build>
    <plugins>
        <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration>
                <appendAssemblyId>false</appendAssemblyId>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
                <archive>
                    <manifest>
                        <mainClass>HDFSTest</mainClass>
                    </manifest>
                </archive>
            </configuration>
            <executions>
                <execution>
                    <id>make-assembly</id>
                    <phase>package</phase>
                    <goals>
                        <goal>assembly</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

三、View->Tool Windows->Maven Projects->Plugins->assembly->assembly.assembly

四、這樣子在Debug的時候都會是正常沒問題，但是真正Build出來時會發現如下錯誤：

C:\Users\will\Desktop\HDFSProject\target>java -jar HDFSProject-1.0-SNAPSHOT.jar
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
java.io.IOException: No FileSystem for scheme: hdfs
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:138)
        at HarOperation.downloadFromHar(HarOperation.java:21)
        at HDFSTest.main(HDFSTest.java:11)

五、這是因為我們引進的客戶端library在core-default.xml這個文件並無配置下列屬性，所以我們可以在程式加上這個config。

private Configuration conf = new Configuration();
conf.set("fs.hdfs.impl","org.apache.hadoop.hdfs.DistributedFileSystem");

六、當然最好的作法是將hadoop源碼下載後再core-default.xml加入配置並重新編譯使用。