org.apache.hadoop.fs
Class InMemoryFileSystem

java.lang.Object
  extended by org.apache.hadoop.conf.Configured
      extended by org.apache.hadoop.fs.FileSystem
          extended by org.apache.hadoop.fs.InMemoryFileSystem
All Implemented Interfaces:
Configurable

public class InMemoryFileSystem
extends FileSystem

An implementation of the in-memory filesystem. This implementation assumes that the file lengths are known ahead of time and the total lengths of all the files is below a certain number (like 100 MB, configurable). Use the API reserveSpaceWithCheckSum(Path f, int size) (see below for a description of the API for reserving space in the FS. The uri of this filesystem starts with ramfs:// .

Author:
ddas

Field Summary
 
Fields inherited from class org.apache.hadoop.fs.FileSystem
LOG
 
Constructor Summary
InMemoryFileSystem()
           
InMemoryFileSystem(URI uri, Configuration conf)
           
 
Method Summary
 void close()
          No more filesystem operations are needed.
 void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile)
          Called when we're all done writing to the target.
 void copyFromLocalFile(Path src, Path dst)
          copy/move operations are not supported
 void copyToLocalFile(Path src, Path dst, boolean copyCrc)
          The src file is under FS, and the dst is on the local disk.
 FSOutputStream createRaw(Path f, boolean overwrite, short replication, long blockSize)
          Opens an OutputStream at the indicated Path.
 FSOutputStream createRaw(Path f, boolean overwrite, short replication, long blockSize, Progressable progress)
          Opens an OutputStream at the indicated Path with write-progress reporting.
 FSOutputStream createRaw(Path f, org.apache.hadoop.fs.InMemoryFileSystem.FileAttributes fAttr)
           
 boolean deleteRaw(Path f)
          Deletes Path
 boolean exists(Path f)
          Check if exists.
 long getBlockSize(Path f)
          Get the size for a particular file.
 int getChecksumFileLength(int size)
           
 long getDefaultBlockSize()
          Return the number of bytes that large input files should be optimally be split into to minimize i/o time.
 short getDefaultReplication()
          Get the default replication.
 String[][] getFileCacheHints(Path f, long start, long len)
          Return 1x1 'inmemory' cell if the file exists.
 Path[] getFiles(PathFilter filter)
          This API getClosedFiles could have been implemented over listPathsRaw but it is an overhead to maintain directory structures for this impl of the in-memory fs.
 int getFSSize()
           
 long getLength(Path f)
          The number of bytes in a file.
 String getName()
          Deprecated.  
 float getPercentUsed()
           
 short getReplication(Path src)
          Replication is not supported for the inmemory file system.
 URI getUri()
          Returns a URI whose scheme and authority identify this FileSystem.
 Path getWorkingDirectory()
          Get the current working directory for the given file system
 void initialize(URI uri, Configuration conf)
          Called after a new FileSystem instance is constructed.
 boolean isDirectory(Path f)
          Directory operations are not supported
 Path[] listPathsRaw(Path f)
          Directory operations are not supported
 void lock(Path f, boolean shared)
          lock operations are not supported
 boolean mkdirs(Path f)
          Make the given file and all non-existent parents into directories.
 void moveFromLocalFile(Path src, Path dst)
          The src file is on the local disk.
 FSInputStream openRaw(Path f)
          Opens an InputStream for the indicated Path, whether local or via DFS.
 void release(Path f)
          Release the lock
 boolean renameRaw(Path src, Path dst)
          Renames Path src to Path dst.
 void reportChecksumFailure(Path p, FSInputStream in, long inPos, FSInputStream sums, long sumsPos)
          Report a checksum error to the file system.
 boolean reserveSpaceWithCheckSum(Path f, int size)
          Register a path with its size.
 boolean setReplicationRaw(Path src, short replication)
          Set replication for an existing file.
 void setWorkingDirectory(Path new_dir)
          Set the current working directory for the given file system.
 Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile)
          Returns a local File that the user can write output to.
 
Methods inherited from class org.apache.hadoop.fs.FileSystem
checkPath, copyToLocalFile, create, create, create, create, create, create, create, create, createNewFile, delete, get, get, getChecksumFile, getChecksumFileLength, getContentLength, getLocal, getNamed, globPaths, globPaths, isChecksumFile, isFile, listPaths, listPaths, listPaths, listPaths, makeQualified, open, open, parseArgs, rename, setReplication
 
Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

InMemoryFileSystem

public InMemoryFileSystem()

InMemoryFileSystem

public InMemoryFileSystem(URI uri,
                          Configuration conf)
Method Detail

initialize

public void initialize(URI uri,
                       Configuration conf)
Description copied from class: FileSystem
Called after a new FileSystem instance is constructed.

Specified by:
initialize in class FileSystem
Parameters:
uri - a uri whose authority section names the host, port, etc. for this FileSystem
conf - the configuration

getUri

public URI getUri()
Description copied from class: FileSystem
Returns a URI whose scheme and authority identify this FileSystem.

Specified by:
getUri in class FileSystem

getName

public String getName()
Deprecated. 

Specified by:
getName in class FileSystem

getFileCacheHints

public String[][] getFileCacheHints(Path f,
                                    long start,
                                    long len)
                             throws IOException
Return 1x1 'inmemory' cell if the file exists. Return null if otherwise.

Specified by:
getFileCacheHints in class FileSystem
Throws:
IOException

openRaw

public FSInputStream openRaw(Path f)
                      throws IOException
Description copied from class: FileSystem
Opens an InputStream for the indicated Path, whether local or via DFS.

Specified by:
openRaw in class FileSystem
Throws:
IOException

createRaw

public FSOutputStream createRaw(Path f,
                                boolean overwrite,
                                short replication,
                                long blockSize)
                         throws IOException
Description copied from class: FileSystem
Opens an OutputStream at the indicated Path.

Specified by:
createRaw in class FileSystem
Parameters:
f - the file name to open
overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
replication - required block replication for the file.
Throws:
IOException

createRaw

public FSOutputStream createRaw(Path f,
                                boolean overwrite,
                                short replication,
                                long blockSize,
                                Progressable progress)
                         throws IOException
Description copied from class: FileSystem
Opens an OutputStream at the indicated Path with write-progress reporting.

Specified by:
createRaw in class FileSystem
Parameters:
f - the file name to open
overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
replication - required block replication for the file.
Throws:
IOException

createRaw

public FSOutputStream createRaw(Path f,
                                org.apache.hadoop.fs.InMemoryFileSystem.FileAttributes fAttr)
                         throws IOException
Throws:
IOException

close

public void close()
           throws IOException
Description copied from class: FileSystem
No more filesystem operations are needed. Will release any held locks.

Overrides:
close in class FileSystem
Throws:
IOException

getReplication

public short getReplication(Path src)
                     throws IOException
Replication is not supported for the inmemory file system.

Specified by:
getReplication in class FileSystem
Parameters:
src - file name
Returns:
file replication
Throws:
IOException

setReplicationRaw

public boolean setReplicationRaw(Path src,
                                 short replication)
                          throws IOException
Description copied from class: FileSystem
Set replication for an existing file.

Specified by:
setReplicationRaw in class FileSystem
Parameters:
src - file name
replication - new replication
Returns:
true if successful; false if file does not exist or is a directory
Throws:
IOException

renameRaw

public boolean renameRaw(Path src,
                         Path dst)
                  throws IOException
Description copied from class: FileSystem
Renames Path src to Path dst. Can take place on local fs or remote DFS.

Specified by:
renameRaw in class FileSystem
Throws:
IOException

deleteRaw

public boolean deleteRaw(Path f)
                  throws IOException
Description copied from class: FileSystem
Deletes Path

Specified by:
deleteRaw in class FileSystem
Throws:
IOException

exists

public boolean exists(Path f)
               throws IOException
Description copied from class: FileSystem
Check if exists.

Specified by:
exists in class FileSystem
Throws:
IOException

isDirectory

public boolean isDirectory(Path f)
                    throws IOException
Directory operations are not supported

Specified by:
isDirectory in class FileSystem
Throws:
IOException

getLength

public long getLength(Path f)
               throws IOException
Description copied from class: FileSystem
The number of bytes in a file.

Specified by:
getLength in class FileSystem
Throws:
IOException

listPathsRaw

public Path[] listPathsRaw(Path f)
                    throws IOException
Directory operations are not supported

Specified by:
listPathsRaw in class FileSystem
Throws:
IOException

setWorkingDirectory

public void setWorkingDirectory(Path new_dir)
Description copied from class: FileSystem
Set the current working directory for the given file system. All relative paths will be resolved relative to it.

Specified by:
setWorkingDirectory in class FileSystem

getWorkingDirectory

public Path getWorkingDirectory()
Description copied from class: FileSystem
Get the current working directory for the given file system

Specified by:
getWorkingDirectory in class FileSystem
Returns:
the directory pathname

mkdirs

public boolean mkdirs(Path f)
               throws IOException
Description copied from class: FileSystem
Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:
mkdirs in class FileSystem
Throws:
IOException

lock

public void lock(Path f,
                 boolean shared)
          throws IOException
lock operations are not supported

Specified by:
lock in class FileSystem
Throws:
IOException

release

public void release(Path f)
             throws IOException
Description copied from class: FileSystem
Release the lock

Specified by:
release in class FileSystem
Throws:
IOException

copyFromLocalFile

public void copyFromLocalFile(Path src,
                              Path dst)
                       throws IOException
copy/move operations are not supported

Specified by:
copyFromLocalFile in class FileSystem
Throws:
IOException

moveFromLocalFile

public void moveFromLocalFile(Path src,
                              Path dst)
                       throws IOException
Description copied from class: FileSystem
The src file is on the local disk. Add it to FS at the given dst name, removing the source afterwards.

Specified by:
moveFromLocalFile in class FileSystem
Throws:
IOException

copyToLocalFile

public void copyToLocalFile(Path src,
                            Path dst,
                            boolean copyCrc)
                     throws IOException
Description copied from class: FileSystem
The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. If src and dst are directories, the copyCrc parameter determines whether to copy CRC files.

Specified by:
copyToLocalFile in class FileSystem
Throws:
IOException

startLocalOutput

public Path startLocalOutput(Path fsOutputFile,
                             Path tmpLocalFile)
                      throws IOException
Description copied from class: FileSystem
Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.

Specified by:
startLocalOutput in class FileSystem
Throws:
IOException

completeLocalOutput

public void completeLocalOutput(Path fsOutputFile,
                                Path tmpLocalFile)
                         throws IOException
Description copied from class: FileSystem
Called when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.

Specified by:
completeLocalOutput in class FileSystem
Throws:
IOException

reportChecksumFailure

public void reportChecksumFailure(Path p,
                                  FSInputStream in,
                                  long inPos,
                                  FSInputStream sums,
                                  long sumsPos)
Description copied from class: FileSystem
Report a checksum error to the file system.

Specified by:
reportChecksumFailure in class FileSystem
Parameters:
p - the file name containing the error
in - the stream open on the file
inPos - the position of the beginning of the bad data in the file
sums - the stream open on the checksum file
sumsPos - the position of the beginning of the bad data in the checksum file

getBlockSize

public long getBlockSize(Path f)
                  throws IOException
Description copied from class: FileSystem
Get the size for a particular file.

Specified by:
getBlockSize in class FileSystem
Parameters:
f - the filename
Returns:
the number of bytes in a block
Throws:
IOException

getDefaultBlockSize

public long getDefaultBlockSize()
Description copied from class: FileSystem
Return the number of bytes that large input files should be optimally be split into to minimize i/o time.

Specified by:
getDefaultBlockSize in class FileSystem

getDefaultReplication

public short getDefaultReplication()
Description copied from class: FileSystem
Get the default replication.

Specified by:
getDefaultReplication in class FileSystem

reserveSpaceWithCheckSum

public boolean reserveSpaceWithCheckSum(Path f,
                                        int size)
Register a path with its size. This will also register a checksum for the file that the user is trying to create. This is required since none of the FileSystem APIs accept the size of the file as argument. But since it is required for us to apriori know the size of the file we are going to create, the user must call this method for each file he wants to create and reserve memory for that file. We either succeed in reserving memory for both the main file and the checksum file and return true, or return false.


getChecksumFileLength

public int getChecksumFileLength(int size)

getFiles

public Path[] getFiles(PathFilter filter)
This API getClosedFiles could have been implemented over listPathsRaw but it is an overhead to maintain directory structures for this impl of the in-memory fs.


getFSSize

public int getFSSize()

getPercentUsed

public float getPercentUsed()


Copyright © 2006 The Apache Software Foundation