org.apache.hadoop.dfs
Class DistributedFileSystem

java.lang.Object
  extended by org.apache.hadoop.conf.Configured
      extended by org.apache.hadoop.fs.FileSystem
          extended by org.apache.hadoop.dfs.DistributedFileSystem
All Implemented Interfaces:
Configurable

public class DistributedFileSystem
extends FileSystem

Implementation of the abstract FileSystem for the DFS system. This object is the way end-user code interacts with a Hadoop DistributedFileSystem.

Author:
Mike Cafarella

Field Summary
 
Fields inherited from class org.apache.hadoop.fs.FileSystem
LOG
 
Constructor Summary
DistributedFileSystem(InetSocketAddress namenode, Configuration conf)
          Construct a client for the filesystem at namenode.
 
Method Summary
 void close()
          No more filesystem operations are needed.
 void completeLocalInput(File localFile)
          We're done with the local stuff, so delete it
 void completeLocalOutput(File fsOutputFile, File tmpLocalFile)
          Move completed local data to DFS destination
 void copyFromLocalFile(File src, File dst)
          The src file is on the local disk.
 void copyToLocalFile(File src, File dst)
          The src file is under FS2, and the dst is on the local disk.
 FSOutputStream createRaw(File f, boolean overwrite)
          Opens an OutputStream at the indicated File.
 boolean deleteRaw(File f)
          Get rid of File f, whether a true file or dir.
 boolean exists(File f)
          Check if exists
 long getBlockSize()
          Return the number of bytes that large input files should be optimally be split into to minimize i/o time.
 DataNodeReport[] getDataNodeStats()
          Return statistics for each datanode.
 String[][] getFileCacheHints(File f, long start, long len)
          Return a 2D array of size 1x1 or greater, containing hostnames where portions of the given file can be found.
 long getLength(File f)
          The number of bytes in a file.
 String getName()
          Returns a name for this filesystem, suitable to pass to FileSystem.getNamed(String,Configuration).
 long getRawCapacity()
          Return the total raw capacity of the filesystem, disregarding replication .
 long getRawUsed()
          Return the total raw used space in the filesystem, disregarding replication .
 long getUsed()
          Return the total size of all files in the filesystem.
 File getWorkingDirectory()
          Get the current working directory for the given file system
 boolean isAbsolute(File f)
          True iff the named path is absolute.
 boolean isDirectory(File f)
          True iff the named path is a directory.
 File[] listFilesRaw(File f)
          List files in a directory.
 void lock(File f, boolean shared)
          Obtain a lock on the given File
 void mkdirs(File f)
          Make the given file and all non-existent parents into directories.
 void moveFromLocalFile(File src, File dst)
          The src file is on the local disk.
 FSInputStream openRaw(File f)
          Opens an InputStream for the indicated File, whether local or via DFS.
 void release(File f)
          Release the lock
 boolean renameRaw(File src, File dst)
          Rename files/dirs
 void reportChecksumFailure(File f, FSInputStream in, long start, long length, int crc)
          Report a checksum error to the file system.
 void setWorkingDirectory(File dir)
          Set the current working directory for the given file system.
 File startLocalInput(File fsInputFile, File tmpLocalFile)
          Fetch remote DFS file, place at tmpLocalFile
 File startLocalOutput(File fsOutputFile, File tmpLocalFile)
          Returns a local File that the user can write output to.
 String toString()
           
 
Methods inherited from class org.apache.hadoop.fs.FileSystem
create, create, createNewFile, delete, get, getChecksumFile, getNamed, isChecksumFile, isFile, listFiles, listFiles, open, open, parseArgs, rename
 
Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

DistributedFileSystem

public DistributedFileSystem(InetSocketAddress namenode,
                             Configuration conf)
                      throws IOException
Construct a client for the filesystem at namenode.

Throws:
IOException
Method Detail

getName

public String getName()
Description copied from class: FileSystem
Returns a name for this filesystem, suitable to pass to FileSystem.getNamed(String,Configuration).

Specified by:
getName in class FileSystem

getWorkingDirectory

public File getWorkingDirectory()
Description copied from class: FileSystem
Get the current working directory for the given file system

Specified by:
getWorkingDirectory in class FileSystem
Returns:
the directory pathname

setWorkingDirectory

public void setWorkingDirectory(File dir)
Description copied from class: FileSystem
Set the current working directory for the given file system. All relative paths will be resolved relative to it.

Specified by:
setWorkingDirectory in class FileSystem

getFileCacheHints

public String[][] getFileCacheHints(File f,
                                    long start,
                                    long len)
                             throws IOException
Description copied from class: FileSystem
Return a 2D array of size 1x1 or greater, containing hostnames where portions of the given file can be found. For a nonexistent file or regions, null will be returned. This call is most helpful with DFS, where it returns hostnames of machines that contain the given file. The FileSystem will simply return an elt containing 'localhost'.

Specified by:
getFileCacheHints in class FileSystem
Throws:
IOException

openRaw

public FSInputStream openRaw(File f)
                      throws IOException
Description copied from class: FileSystem
Opens an InputStream for the indicated File, whether local or via DFS.

Specified by:
openRaw in class FileSystem
Throws:
IOException

createRaw

public FSOutputStream createRaw(File f,
                                boolean overwrite)
                         throws IOException
Description copied from class: FileSystem
Opens an OutputStream at the indicated File.

Specified by:
createRaw in class FileSystem
Parameters:
f - the file name to open
overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
Throws:
IOException

renameRaw

public boolean renameRaw(File src,
                         File dst)
                  throws IOException
Rename files/dirs

Specified by:
renameRaw in class FileSystem
Throws:
IOException

deleteRaw

public boolean deleteRaw(File f)
                  throws IOException
Get rid of File f, whether a true file or dir.

Specified by:
deleteRaw in class FileSystem
Throws:
IOException

exists

public boolean exists(File f)
               throws IOException
Description copied from class: FileSystem
Check if exists

Specified by:
exists in class FileSystem
Throws:
IOException

isDirectory

public boolean isDirectory(File f)
                    throws IOException
Description copied from class: FileSystem
True iff the named path is a directory.

Specified by:
isDirectory in class FileSystem
Throws:
IOException

isAbsolute

public boolean isAbsolute(File f)
Description copied from class: FileSystem
True iff the named path is absolute.

Specified by:
isAbsolute in class FileSystem

getLength

public long getLength(File f)
               throws IOException
Description copied from class: FileSystem
The number of bytes in a file.

Specified by:
getLength in class FileSystem
Throws:
IOException

listFilesRaw

public File[] listFilesRaw(File f)
                    throws IOException
Description copied from class: FileSystem
List files in a directory.

Specified by:
listFilesRaw in class FileSystem
Throws:
IOException

mkdirs

public void mkdirs(File f)
            throws IOException
Description copied from class: FileSystem
Make the given file and all non-existent parents into directories.

Specified by:
mkdirs in class FileSystem
Throws:
IOException

lock

public void lock(File f,
                 boolean shared)
          throws IOException
Description copied from class: FileSystem
Obtain a lock on the given File

Specified by:
lock in class FileSystem
Throws:
IOException

release

public void release(File f)
             throws IOException
Description copied from class: FileSystem
Release the lock

Specified by:
release in class FileSystem
Throws:
IOException

moveFromLocalFile

public void moveFromLocalFile(File src,
                              File dst)
                       throws IOException
Description copied from class: FileSystem
The src file is on the local disk. Add it to FS at the given dst name, removing the source afterwards.

Specified by:
moveFromLocalFile in class FileSystem
Throws:
IOException

copyFromLocalFile

public void copyFromLocalFile(File src,
                              File dst)
                       throws IOException
Description copied from class: FileSystem
The src file is on the local disk. Add it to FS at the given dst name and the source is kept intact afterwards

Specified by:
copyFromLocalFile in class FileSystem
Throws:
IOException

copyToLocalFile

public void copyToLocalFile(File src,
                            File dst)
                     throws IOException
Description copied from class: FileSystem
The src file is under FS2, and the dst is on the local disk. Copy it from FS control to the local dst name.

Specified by:
copyToLocalFile in class FileSystem
Throws:
IOException

startLocalOutput

public File startLocalOutput(File fsOutputFile,
                             File tmpLocalFile)
                      throws IOException
Description copied from class: FileSystem
Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.

Specified by:
startLocalOutput in class FileSystem
Throws:
IOException

completeLocalOutput

public void completeLocalOutput(File fsOutputFile,
                                File tmpLocalFile)
                         throws IOException
Move completed local data to DFS destination

Specified by:
completeLocalOutput in class FileSystem
Throws:
IOException

startLocalInput

public File startLocalInput(File fsInputFile,
                            File tmpLocalFile)
                     throws IOException
Fetch remote DFS file, place at tmpLocalFile

Specified by:
startLocalInput in class FileSystem
Throws:
IOException

completeLocalInput

public void completeLocalInput(File localFile)
                        throws IOException
We're done with the local stuff, so delete it

Specified by:
completeLocalInput in class FileSystem
Throws:
IOException

close

public void close()
           throws IOException
Description copied from class: FileSystem
No more filesystem operations are needed. Will release any held locks.

Specified by:
close in class FileSystem
Throws:
IOException

toString

public String toString()
Overrides:
toString in class Object

reportChecksumFailure

public void reportChecksumFailure(File f,
                                  FSInputStream in,
                                  long start,
                                  long length,
                                  int crc)
Description copied from class: FileSystem
Report a checksum error to the file system.

Specified by:
reportChecksumFailure in class FileSystem
Parameters:
f - the file name containing the error
in - the stream open on the file
start - the position of the beginning of the bad data in the file
length - the length of the bad data in the file
crc - the expected CRC32 of the data

getBlockSize

public long getBlockSize()
Description copied from class: FileSystem
Return the number of bytes that large input files should be optimally be split into to minimize i/o time.

Specified by:
getBlockSize in class FileSystem

getRawCapacity

public long getRawCapacity()
                    throws IOException
Return the total raw capacity of the filesystem, disregarding replication .

Throws:
IOException

getRawUsed

public long getRawUsed()
                throws IOException
Return the total raw used space in the filesystem, disregarding replication .

Throws:
IOException

getUsed

public long getUsed()
             throws IOException
Return the total size of all files in the filesystem.

Throws:
IOException

getDataNodeStats

public DataNodeReport[] getDataNodeStats()
                                  throws IOException
Return statistics for each datanode.

Throws:
IOException


Copyright © 2006 The Apache Software Foundation