org.apache.hadoop.dfs
Class DistributedFileSystem

java.lang.Object
  extended by org.apache.hadoop.conf.Configured
      extended by org.apache.hadoop.fs.FileSystem
          extended by org.apache.hadoop.dfs.DistributedFileSystem
All Implemented Interfaces:
Configurable

public class DistributedFileSystem
extends FileSystem

Implementation of the abstract FileSystem for the DFS system. This object is the way end-user code interacts with a Hadoop DistributedFileSystem.

Author:
Mike Cafarella

Field Summary
 
Fields inherited from class org.apache.hadoop.fs.FileSystem
LOG
 
Constructor Summary
DistributedFileSystem(InetSocketAddress namenode, Configuration conf)
          Construct a client for the filesystem at namenode.
 
Method Summary
 void close()
          No more filesystem operations are needed.
 void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile)
          Move completed local data to DFS destination
 void copyFromLocalFile(Path src, Path dst)
          The src file is on the local disk.
 void copyToLocalFile(Path src, Path dst)
          The src file is under FS, and the dst is on the local disk.
 FSOutputStream createRaw(Path f, boolean overwrite, short replication)
          Opens an OutputStream at the indicated Path.
 boolean deleteRaw(Path f)
          Get rid of Path f, whether a true file or dir.
 boolean exists(Path f)
          Check if exists.
 long getBlockSize()
          Return the number of bytes that large input files should be optimally be split into to minimize i/o time.
 DataNodeReport[] getDataNodeStats()
          Return statistics for each datanode.
 String[][] getFileCacheHints(Path f, long start, long len)
          Return a 2D array of size 1x1 or greater, containing hostnames where portions of the given file can be found.
 long getLength(Path f)
          The number of bytes in a file.
 String getName()
          Returns a name for this filesystem, suitable to pass to FileSystem.getNamed(String,Configuration).
 long getRawCapacity()
          Return the total raw capacity of the filesystem, disregarding replication .
 long getRawUsed()
          Return the total raw used space in the filesystem, disregarding replication .
 short getReplication(Path f)
          Get replication.
 long getUsed()
          Return the total size of all files in the filesystem.
 Path getWorkingDirectory()
          Get the current working directory for the given file system
 boolean isDirectory(Path f)
          True iff the named path is a directory.
 Path[] listPathsRaw(Path f)
          List files in a directory.
 void lock(Path f, boolean shared)
          Obtain a lock on the given Path
 boolean mkdirs(Path f)
          Make the given file and all non-existent parents into directories.
 void moveFromLocalFile(Path src, Path dst)
          The src file is on the local disk.
 FSInputStream openRaw(Path f)
          Opens an InputStream for the indicated Path, whether local or via DFS.
 void release(Path f)
          Release the lock
 boolean renameRaw(Path src, Path dst)
          Rename files/dirs
 void reportChecksumFailure(Path f, FSInputStream in, long start, long length, int crc)
          Report a checksum error to the file system.
 boolean setReplicationRaw(Path src, short replication)
          Set replication for an existing file.
 void setWorkingDirectory(Path dir)
          Set the current working directory for the given file system.
 Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile)
          Returns a local File that the user can write output to.
 String toString()
           
 
Methods inherited from class org.apache.hadoop.fs.FileSystem
completeLocalOutput, create, create, create, create, create, createNewFile, createNewFile, delete, delete, exists, get, getChecksumFile, getLength, getNamed, isChecksumFile, isDirectory, isFile, isFile, listFiles, listFiles, listPaths, listPaths, lock, mkdirs, open, open, open, parseArgs, release, rename, rename, setReplication, startLocalOutput
 
Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

DistributedFileSystem

public DistributedFileSystem(InetSocketAddress namenode,
                             Configuration conf)
                      throws IOException
Construct a client for the filesystem at namenode.

Throws:
IOException
Method Detail

getName

public String getName()
Description copied from class: FileSystem
Returns a name for this filesystem, suitable to pass to FileSystem.getNamed(String,Configuration).

Specified by:
getName in class FileSystem

getWorkingDirectory

public Path getWorkingDirectory()
Description copied from class: FileSystem
Get the current working directory for the given file system

Specified by:
getWorkingDirectory in class FileSystem
Returns:
the directory pathname

setWorkingDirectory

public void setWorkingDirectory(Path dir)
Description copied from class: FileSystem
Set the current working directory for the given file system. All relative paths will be resolved relative to it.

Specified by:
setWorkingDirectory in class FileSystem

getFileCacheHints

public String[][] getFileCacheHints(Path f,
                                    long start,
                                    long len)
                             throws IOException
Description copied from class: FileSystem
Return a 2D array of size 1x1 or greater, containing hostnames where portions of the given file can be found. For a nonexistent file or regions, null will be returned. This call is most helpful with DFS, where it returns hostnames of machines that contain the given file. The FileSystem will simply return an elt containing 'localhost'.

Specified by:
getFileCacheHints in class FileSystem
Throws:
IOException

openRaw

public FSInputStream openRaw(Path f)
                      throws IOException
Description copied from class: FileSystem
Opens an InputStream for the indicated Path, whether local or via DFS.

Specified by:
openRaw in class FileSystem
Throws:
IOException

createRaw

public FSOutputStream createRaw(Path f,
                                boolean overwrite,
                                short replication)
                         throws IOException
Description copied from class: FileSystem
Opens an OutputStream at the indicated Path.

Specified by:
createRaw in class FileSystem
Parameters:
f - the file name to open
overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
replication - required block replication for the file.
Throws:
IOException

setReplicationRaw

public boolean setReplicationRaw(Path src,
                                 short replication)
                          throws IOException
Description copied from class: FileSystem
Set replication for an existing file.

Specified by:
setReplicationRaw in class FileSystem
Parameters:
src - file name
replication - new replication
Returns:
true if successful; false if file does not exist or is a directory
Throws:
IOException

renameRaw

public boolean renameRaw(Path src,
                         Path dst)
                  throws IOException
Rename files/dirs

Specified by:
renameRaw in class FileSystem
Throws:
IOException

deleteRaw

public boolean deleteRaw(Path f)
                  throws IOException
Get rid of Path f, whether a true file or dir.

Specified by:
deleteRaw in class FileSystem
Throws:
IOException

exists

public boolean exists(Path f)
               throws IOException
Description copied from class: FileSystem
Check if exists.

Specified by:
exists in class FileSystem
Throws:
IOException

isDirectory

public boolean isDirectory(Path f)
                    throws IOException
Description copied from class: FileSystem
True iff the named path is a directory.

Specified by:
isDirectory in class FileSystem
Throws:
IOException

getLength

public long getLength(Path f)
               throws IOException
Description copied from class: FileSystem
The number of bytes in a file.

Specified by:
getLength in class FileSystem
Throws:
IOException

getReplication

public short getReplication(Path f)
                     throws IOException
Description copied from class: FileSystem
Get replication.

Specified by:
getReplication in class FileSystem
Parameters:
f - file name
Returns:
file replication
Throws:
IOException

listPathsRaw

public Path[] listPathsRaw(Path f)
                    throws IOException
Description copied from class: FileSystem
List files in a directory.

Specified by:
listPathsRaw in class FileSystem
Throws:
IOException

mkdirs

public boolean mkdirs(Path f)
               throws IOException
Description copied from class: FileSystem
Make the given file and all non-existent parents into directories.

Specified by:
mkdirs in class FileSystem
Throws:
IOException

lock

public void lock(Path f,
                 boolean shared)
          throws IOException
Description copied from class: FileSystem
Obtain a lock on the given Path

Specified by:
lock in class FileSystem
Throws:
IOException

release

public void release(Path f)
             throws IOException
Description copied from class: FileSystem
Release the lock

Specified by:
release in class FileSystem
Throws:
IOException

moveFromLocalFile

public void moveFromLocalFile(Path src,
                              Path dst)
                       throws IOException
Description copied from class: FileSystem
The src file is on the local disk. Add it to FS at the given dst name, removing the source afterwards.

Specified by:
moveFromLocalFile in class FileSystem
Throws:
IOException

copyFromLocalFile

public void copyFromLocalFile(Path src,
                              Path dst)
                       throws IOException
Description copied from class: FileSystem
The src file is on the local disk. Add it to FS at the given dst name and the source is kept intact afterwards

Specified by:
copyFromLocalFile in class FileSystem
Throws:
IOException

copyToLocalFile

public void copyToLocalFile(Path src,
                            Path dst)
                     throws IOException
Description copied from class: FileSystem
The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name.

Specified by:
copyToLocalFile in class FileSystem
Throws:
IOException

startLocalOutput

public Path startLocalOutput(Path fsOutputFile,
                             Path tmpLocalFile)
                      throws IOException
Description copied from class: FileSystem
Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.

Specified by:
startLocalOutput in class FileSystem
Throws:
IOException

completeLocalOutput

public void completeLocalOutput(Path fsOutputFile,
                                Path tmpLocalFile)
                         throws IOException
Move completed local data to DFS destination

Specified by:
completeLocalOutput in class FileSystem
Throws:
IOException

close

public void close()
           throws IOException
Description copied from class: FileSystem
No more filesystem operations are needed. Will release any held locks.

Specified by:
close in class FileSystem
Throws:
IOException

toString

public String toString()
Overrides:
toString in class Object

reportChecksumFailure

public void reportChecksumFailure(Path f,
                                  FSInputStream in,
                                  long start,
                                  long length,
                                  int crc)
Description copied from class: FileSystem
Report a checksum error to the file system.

Specified by:
reportChecksumFailure in class FileSystem
Parameters:
f - the file name containing the error
in - the stream open on the file
start - the position of the beginning of the bad data in the file
length - the length of the bad data in the file
crc - the expected CRC32 of the data

getBlockSize

public long getBlockSize()
Description copied from class: FileSystem
Return the number of bytes that large input files should be optimally be split into to minimize i/o time.

Specified by:
getBlockSize in class FileSystem

getRawCapacity

public long getRawCapacity()
                    throws IOException
Return the total raw capacity of the filesystem, disregarding replication .

Throws:
IOException

getRawUsed

public long getRawUsed()
                throws IOException
Return the total raw used space in the filesystem, disregarding replication .

Throws:
IOException

getUsed

public long getUsed()
             throws IOException
Return the total size of all files in the filesystem.

Throws:
IOException

getDataNodeStats

public DataNodeReport[] getDataNodeStats()
                                  throws IOException
Return statistics for each datanode.

Throws:
IOException


Copyright © 2006 The Apache Software Foundation