org.apache.hadoop.fs
Class InMemoryFileSystem
java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.FilterFileSystem
org.apache.hadoop.fs.ChecksumFileSystem
org.apache.hadoop.fs.InMemoryFileSystem
- All Implemented Interfaces:
- Configurable
public class InMemoryFileSystem
- extends ChecksumFileSystem
An implementation of the in-memory filesystem. This implementation assumes
that the file lengths are known ahead of time and the total lengths of all
the files is below a certain number (like 100 MB, configurable). Use the API
reserveSpaceWithCheckSum(Path f, int size) (see below for a description of
the API for reserving space in the FS. The uri of this filesystem starts with
ramfs:// .
- Author:
- ddas
Method Summary |
void |
completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Called when we're all done writing to the target. |
void |
copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
copy/move operations are not supported |
void |
copyToLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is under FS, and the dst is on the local disk. |
Path[] |
getFiles(PathFilter filter)
|
int |
getFSSize()
|
int |
getNumFiles(PathFilter filter)
|
float |
getPercentUsed()
|
void |
reportChecksumFailure(Path p,
FSDataInputStream in,
long inPos,
FSDataInputStream sums,
long sumsPos)
Report a checksum error to the file system. |
boolean |
reserveSpaceWithCheckSum(Path f,
int size)
Register a file with its size. |
Path |
startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to. |
Methods inherited from class org.apache.hadoop.fs.ChecksumFileSystem |
copyToLocalFile, create, delete, getBytesPerSum, getChecksumFile, getChecksumFileLength, getRawFileSystem, isChecksumFile, listPaths, listPaths, lock, mkdirs, open, release, rename, setReplication |
Methods inherited from class org.apache.hadoop.fs.FilterFileSystem |
checkPath, exists, getBlockSize, getConf, getDefaultBlockSize, getDefaultReplication, getFileCacheHints, getLength, getName, getReplication, getUri, getWorkingDirectory, initialize, isDirectory, makeQualified, setWorkingDirectory |
Methods inherited from class org.apache.hadoop.fs.FileSystem |
close, copyFromLocalFile, copyToLocalFile, create, create, create, create, create, create, create, createNewFile, get, get, getContentLength, getLocal, getNamed, getUsed, globPaths, globPaths, isFile, listPaths, listPaths, moveFromLocalFile, moveToLocalFile, open, parseArgs |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
InMemoryFileSystem
public InMemoryFileSystem()
InMemoryFileSystem
public InMemoryFileSystem(URI uri,
Configuration conf)
copyFromLocalFile
public void copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
throws IOException
- copy/move operations are not supported
- Overrides:
copyFromLocalFile
in class ChecksumFileSystem
- Throws:
IOException
copyToLocalFile
public void copyToLocalFile(boolean delSrc,
Path src,
Path dst)
throws IOException
- Description copied from class:
ChecksumFileSystem
- The src file is under FS, and the dst is on the local disk.
Copy it from FS control to the local dst name.
- Overrides:
copyToLocalFile
in class ChecksumFileSystem
- Throws:
IOException
startLocalOutput
public Path startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
throws IOException
- Description copied from class:
FilterFileSystem
- Returns a local File that the user can write output to. The caller
provides both the eventual FS target name and the local working
file. If the FS is local, we write directly into the target. If
the FS is remote, we write into the tmp local area.
- Overrides:
startLocalOutput
in class ChecksumFileSystem
- Throws:
IOException
completeLocalOutput
public void completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
throws IOException
- Description copied from class:
FilterFileSystem
- Called when we're all done writing to the target. A local FS will
do nothing, because we've written to exactly the right place. A remote
FS will copy the contents of tmpLocalFile to the correct target at
fsOutputFile.
- Overrides:
completeLocalOutput
in class ChecksumFileSystem
- Throws:
IOException
reportChecksumFailure
public void reportChecksumFailure(Path p,
FSDataInputStream in,
long inPos,
FSDataInputStream sums,
long sumsPos)
- Description copied from class:
ChecksumFileSystem
- Report a checksum error to the file system.
- Specified by:
reportChecksumFailure
in class ChecksumFileSystem
- Parameters:
p
- the file name containing the errorin
- the stream open on the fileinPos
- the position of the beginning of the bad data in the filesums
- the stream open on the checksum filesumsPos
- the position of the beginning of the bad data in the checksum file
reserveSpaceWithCheckSum
public boolean reserveSpaceWithCheckSum(Path f,
int size)
- Register a file with its size. This will also register a checksum for the
file that the user is trying to create. This is required since none of
the FileSystem APIs accept the size of the file as argument. But since it
is required for us to apriori know the size of the file we are going to
create, the user must call this method for each file he wants to create
and reserve memory for that file. We either succeed in reserving memory
for both the main file and the checksum file and return true, or return
false.
getFiles
public Path[] getFiles(PathFilter filter)
getNumFiles
public int getNumFiles(PathFilter filter)
getFSSize
public int getFSSize()
getPercentUsed
public float getPercentUsed()
Copyright © 2006 The Apache Software Foundation