org.apache.hadoop.fs.s3
Class S3FileSystem

java.lang.Object
  extended by org.apache.hadoop.conf.Configured
      extended by org.apache.hadoop.fs.FileSystem
          extended by org.apache.hadoop.fs.s3.S3FileSystem
All Implemented Interfaces:
Configurable

public class S3FileSystem
extends FileSystem

A FileSystem backed by Amazon S3.

Author:
Tom White

Field Summary
 
Fields inherited from class org.apache.hadoop.fs.FileSystem
LOG
 
Constructor Summary
S3FileSystem()
           
S3FileSystem(org.apache.hadoop.fs.s3.FileSystemStore store)
           
 
Method Summary
 void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile)
          Called when we're all done writing to the target.
 void copyFromLocalFile(Path src, Path dst)
          The src file is on the local disk.
 void copyToLocalFile(Path src, Path dst, boolean copyCrc)
          The src file is under FS, and the dst is on the local disk.
 FSOutputStream createRaw(Path file, boolean overwrite, short replication, long blockSize)
          Opens an OutputStream at the indicated Path.
 FSOutputStream createRaw(Path file, boolean overwrite, short replication, long blockSize, Progressable progress)
          Opens an OutputStream at the indicated Path with write-progress reporting.
 boolean deleteRaw(Path path)
          Deletes Path
 boolean exists(Path path)
          Check if exists.
 long getBlockSize(Path path)
          Get the size for a particular file.
 long getDefaultBlockSize()
          Return the number of bytes that large input files should be optimally be split into to minimize i/o time.
 short getDefaultReplication()
          Get the default replication.
 String[][] getFileCacheHints(Path f, long start, long len)
          Return 1x1 'localhost' cell if the file exists.
 long getLength(Path path)
          The number of bytes in a file.
 String getName()
           
 short getReplication(Path path)
          Replication is not supported for S3 file systems since S3 handles it for us.
 URI getUri()
          Returns a URI whose scheme and authority identify this FileSystem.
 Path getWorkingDirectory()
          Get the current working directory for the given file system
 void initialize(URI uri, Configuration conf)
          Called after a new FileSystem instance is constructed.
 boolean isDirectory(Path path)
          True iff the named path is a directory.
 boolean isFile(Path path)
          True iff the named path is a regular file.
 Path[] listPathsRaw(Path path)
          List files in a directory.
 void lock(Path path, boolean shared)
          Deprecated.  
 boolean mkdirs(Path path)
          Make the given file and all non-existent parents into directories.
 void moveFromLocalFile(Path src, Path dst)
          The src file is on the local disk.
 FSInputStream openRaw(Path path)
          Opens an InputStream for the indicated Path, whether local or via DFS.
 void release(Path path)
          Deprecated.  
 boolean renameRaw(Path src, Path dst)
          Renames Path src to Path dst.
 void reportChecksumFailure(Path f, FSInputStream in, long inPos, FSInputStream sums, long sumsPos)
          Report a checksum error to the file system.
 boolean setReplicationRaw(Path path, short replication)
          Replication is not supported for S3 file systems since S3 handles it for us.
 void setWorkingDirectory(Path dir)
          Set the current working directory for the given file system.
 Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile)
          Returns a local File that the user can write output to.
 
Methods inherited from class org.apache.hadoop.fs.FileSystem
checkPath, close, copyToLocalFile, create, create, create, create, create, create, create, create, createNewFile, delete, get, get, getChecksumFile, getChecksumFileLength, getContentLength, getLocal, getNamed, globPaths, globPaths, isChecksumFile, listPaths, listPaths, listPaths, listPaths, makeQualified, open, open, parseArgs, rename, setReplication
 
Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

S3FileSystem

public S3FileSystem()

S3FileSystem

public S3FileSystem(org.apache.hadoop.fs.s3.FileSystemStore store)
Method Detail

getUri

public URI getUri()
Description copied from class: FileSystem
Returns a URI whose scheme and authority identify this FileSystem.

Specified by:
getUri in class FileSystem

initialize

public void initialize(URI uri,
                       Configuration conf)
                throws IOException
Description copied from class: FileSystem
Called after a new FileSystem instance is constructed.

Specified by:
initialize in class FileSystem
Parameters:
uri - a uri whose authority section names the host, port, etc. for this FileSystem
conf - the configuration
Throws:
IOException

getName

public String getName()
Specified by:
getName in class FileSystem

getWorkingDirectory

public Path getWorkingDirectory()
Description copied from class: FileSystem
Get the current working directory for the given file system

Specified by:
getWorkingDirectory in class FileSystem
Returns:
the directory pathname

setWorkingDirectory

public void setWorkingDirectory(Path dir)
Description copied from class: FileSystem
Set the current working directory for the given file system. All relative paths will be resolved relative to it.

Specified by:
setWorkingDirectory in class FileSystem

exists

public boolean exists(Path path)
               throws IOException
Description copied from class: FileSystem
Check if exists.

Specified by:
exists in class FileSystem
Throws:
IOException

mkdirs

public boolean mkdirs(Path path)
               throws IOException
Description copied from class: FileSystem
Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:
mkdirs in class FileSystem
Throws:
IOException

isDirectory

public boolean isDirectory(Path path)
                    throws IOException
Description copied from class: FileSystem
True iff the named path is a directory.

Specified by:
isDirectory in class FileSystem
Throws:
IOException

isFile

public boolean isFile(Path path)
               throws IOException
Description copied from class: FileSystem
True iff the named path is a regular file.

Overrides:
isFile in class FileSystem
Throws:
IOException

listPathsRaw

public Path[] listPathsRaw(Path path)
                    throws IOException
Description copied from class: FileSystem
List files in a directory.

Specified by:
listPathsRaw in class FileSystem
Throws:
IOException

createRaw

public FSOutputStream createRaw(Path file,
                                boolean overwrite,
                                short replication,
                                long blockSize)
                         throws IOException
Description copied from class: FileSystem
Opens an OutputStream at the indicated Path.

Specified by:
createRaw in class FileSystem
Parameters:
file - the file name to open
overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
replication - required block replication for the file.
Throws:
IOException

createRaw

public FSOutputStream createRaw(Path file,
                                boolean overwrite,
                                short replication,
                                long blockSize,
                                Progressable progress)
                         throws IOException
Description copied from class: FileSystem
Opens an OutputStream at the indicated Path with write-progress reporting.

Specified by:
createRaw in class FileSystem
Parameters:
file - the file name to open
overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
replication - required block replication for the file.
Throws:
IOException

openRaw

public FSInputStream openRaw(Path path)
                      throws IOException
Description copied from class: FileSystem
Opens an InputStream for the indicated Path, whether local or via DFS.

Specified by:
openRaw in class FileSystem
Throws:
IOException

renameRaw

public boolean renameRaw(Path src,
                         Path dst)
                  throws IOException
Description copied from class: FileSystem
Renames Path src to Path dst. Can take place on local fs or remote DFS.

Specified by:
renameRaw in class FileSystem
Throws:
IOException

deleteRaw

public boolean deleteRaw(Path path)
                  throws IOException
Description copied from class: FileSystem
Deletes Path

Specified by:
deleteRaw in class FileSystem
Throws:
IOException

getLength

public long getLength(Path path)
               throws IOException
Description copied from class: FileSystem
The number of bytes in a file.

Specified by:
getLength in class FileSystem
Throws:
IOException

getReplication

public short getReplication(Path path)
                     throws IOException
Replication is not supported for S3 file systems since S3 handles it for us.

Specified by:
getReplication in class FileSystem
Parameters:
path - file name
Returns:
file replication
Throws:
IOException

getDefaultReplication

public short getDefaultReplication()
Description copied from class: FileSystem
Get the default replication.

Specified by:
getDefaultReplication in class FileSystem

setReplicationRaw

public boolean setReplicationRaw(Path path,
                                 short replication)
                          throws IOException
Replication is not supported for S3 file systems since S3 handles it for us.

Specified by:
setReplicationRaw in class FileSystem
Parameters:
path - file name
replication - new replication
Returns:
true if successful; false if file does not exist or is a directory
Throws:
IOException

getBlockSize

public long getBlockSize(Path path)
                  throws IOException
Description copied from class: FileSystem
Get the size for a particular file.

Specified by:
getBlockSize in class FileSystem
Parameters:
path - the filename
Returns:
the number of bytes in a block
Throws:
IOException

getDefaultBlockSize

public long getDefaultBlockSize()
Description copied from class: FileSystem
Return the number of bytes that large input files should be optimally be split into to minimize i/o time.

Specified by:
getDefaultBlockSize in class FileSystem

getFileCacheHints

public String[][] getFileCacheHints(Path f,
                                    long start,
                                    long len)
                             throws IOException
Return 1x1 'localhost' cell if the file exists. Return null if otherwise.

Specified by:
getFileCacheHints in class FileSystem
Throws:
IOException

lock

@Deprecated
public void lock(Path path,
                            boolean shared)
          throws IOException
Deprecated. 

Description copied from class: FileSystem
Obtain a lock on the given Path

Specified by:
lock in class FileSystem
Throws:
IOException

release

@Deprecated
public void release(Path path)
             throws IOException
Deprecated. 

Description copied from class: FileSystem
Release the lock

Specified by:
release in class FileSystem
Throws:
IOException

reportChecksumFailure

public void reportChecksumFailure(Path f,
                                  FSInputStream in,
                                  long inPos,
                                  FSInputStream sums,
                                  long sumsPos)
Description copied from class: FileSystem
Report a checksum error to the file system.

Specified by:
reportChecksumFailure in class FileSystem
Parameters:
f - the file name containing the error
in - the stream open on the file
inPos - the position of the beginning of the bad data in the file
sums - the stream open on the checksum file
sumsPos - the position of the beginning of the bad data in the checksum file

moveFromLocalFile

public void moveFromLocalFile(Path src,
                              Path dst)
                       throws IOException
Description copied from class: FileSystem
The src file is on the local disk. Add it to FS at the given dst name, removing the source afterwards.

Specified by:
moveFromLocalFile in class FileSystem
Throws:
IOException

copyFromLocalFile

public void copyFromLocalFile(Path src,
                              Path dst)
                       throws IOException
Description copied from class: FileSystem
The src file is on the local disk. Add it to FS at the given dst name and the source is kept intact afterwards

Specified by:
copyFromLocalFile in class FileSystem
Throws:
IOException

copyToLocalFile

public void copyToLocalFile(Path src,
                            Path dst,
                            boolean copyCrc)
                     throws IOException
Description copied from class: FileSystem
The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. If src and dst are directories, the copyCrc parameter determines whether to copy CRC files.

Specified by:
copyToLocalFile in class FileSystem
Throws:
IOException

startLocalOutput

public Path startLocalOutput(Path fsOutputFile,
                             Path tmpLocalFile)
                      throws IOException
Description copied from class: FileSystem
Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.

Specified by:
startLocalOutput in class FileSystem
Throws:
IOException

completeLocalOutput

public void completeLocalOutput(Path fsOutputFile,
                                Path tmpLocalFile)
                         throws IOException
Description copied from class: FileSystem
Called when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.

Specified by:
completeLocalOutput in class FileSystem
Throws:
IOException


Copyright © 2006 The Apache Software Foundation