|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hadoop.conf.Configured
org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.InMemoryFileSystem
public class InMemoryFileSystem
An implementation of the in-memory filesystem. This implementation assumes that the file lengths are known ahead of time and the total lengths of all the files is below a certain number (like 100 MB, configurable). Use the API reserveSpaceWithCheckSum(Path f, int size) (see below for a description of the API for reserving space in the FS. The uri of this filesystem starts with ramfs:// .
Field Summary |
---|
Fields inherited from class org.apache.hadoop.fs.FileSystem |
---|
LOG |
Constructor Summary | |
---|---|
InMemoryFileSystem()
|
|
InMemoryFileSystem(URI uri,
Configuration conf)
|
Method Summary | |
---|---|
void |
close()
No more filesystem operations are needed. |
void |
completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Called when we're all done writing to the target. |
void |
copyFromLocalFile(Path src,
Path dst)
copy/move operations are not supported |
void |
copyToLocalFile(Path src,
Path dst,
boolean copyCrc)
The src file is under FS, and the dst is on the local disk. |
FSOutputStream |
createRaw(Path f,
boolean overwrite,
short replication,
long blockSize)
Opens an OutputStream at the indicated Path. |
FSOutputStream |
createRaw(Path f,
boolean overwrite,
short replication,
long blockSize,
Progressable progress)
Opens an OutputStream at the indicated Path with write-progress reporting. |
FSOutputStream |
createRaw(Path f,
org.apache.hadoop.fs.InMemoryFileSystem.FileAttributes fAttr)
|
boolean |
deleteRaw(Path f)
Deletes Path |
boolean |
exists(Path f)
Check if exists. |
long |
getBlockSize(Path f)
Get the size for a particular file. |
int |
getChecksumFileLength(int size)
|
long |
getDefaultBlockSize()
Return the number of bytes that large input files should be optimally be split into to minimize i/o time. |
short |
getDefaultReplication()
Get the default replication. |
String[][] |
getFileCacheHints(Path f,
long start,
long len)
Return 1x1 'inmemory' cell if the file exists. |
Path[] |
getFiles(PathFilter filter)
This API getClosedFiles could have been implemented over listPathsRaw but it is an overhead to maintain directory structures for this impl of the in-memory fs. |
int |
getFSSize()
|
long |
getLength(Path f)
The number of bytes in a file. |
String |
getName()
Deprecated. |
float |
getPercentUsed()
|
short |
getReplication(Path src)
Replication is not supported for the inmemory file system. |
URI |
getUri()
Returns a URI whose scheme and authority identify this FileSystem. |
Path |
getWorkingDirectory()
Get the current working directory for the given file system |
void |
initialize(URI uri,
Configuration conf)
Called after a new FileSystem instance is constructed. |
boolean |
isDirectory(Path f)
Directory operations are not supported |
Path[] |
listPathsRaw(Path f)
Directory operations are not supported |
void |
lock(Path f,
boolean shared)
lock operations are not supported |
boolean |
mkdirs(Path f)
Make the given file and all non-existent parents into directories. |
void |
moveFromLocalFile(Path src,
Path dst)
The src file is on the local disk. |
FSInputStream |
openRaw(Path f)
Opens an InputStream for the indicated Path, whether local or via DFS. |
void |
release(Path f)
Release the lock |
boolean |
renameRaw(Path src,
Path dst)
Renames Path src to Path dst. |
void |
reportChecksumFailure(Path p,
FSInputStream in,
long inPos,
FSInputStream sums,
long sumsPos)
Report a checksum error to the file system. |
boolean |
reserveSpaceWithCheckSum(Path f,
int size)
Register a path with its size. |
boolean |
setReplicationRaw(Path src,
short replication)
Set replication for an existing file. |
void |
setWorkingDirectory(Path new_dir)
Set the current working directory for the given file system. |
Path |
startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to. |
Methods inherited from class org.apache.hadoop.fs.FileSystem |
---|
checkPath, copyToLocalFile, create, create, create, create, create, create, create, create, createNewFile, delete, get, get, getChecksumFile, getChecksumFileLength, getContentLength, getLocal, getNamed, globPaths, globPaths, isChecksumFile, isFile, listPaths, listPaths, listPaths, listPaths, makeQualified, open, open, parseArgs, rename, setReplication |
Methods inherited from class org.apache.hadoop.conf.Configured |
---|
getConf, setConf |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public InMemoryFileSystem()
public InMemoryFileSystem(URI uri, Configuration conf)
Method Detail |
---|
public void initialize(URI uri, Configuration conf)
FileSystem
initialize
in class FileSystem
uri
- a uri whose authority section names the host, port, etc.
for this FileSystemconf
- the configurationpublic URI getUri()
FileSystem
getUri
in class FileSystem
public String getName()
getName
in class FileSystem
public String[][] getFileCacheHints(Path f, long start, long len) throws IOException
getFileCacheHints
in class FileSystem
IOException
public FSInputStream openRaw(Path f) throws IOException
FileSystem
openRaw
in class FileSystem
IOException
public FSOutputStream createRaw(Path f, boolean overwrite, short replication, long blockSize) throws IOException
FileSystem
createRaw
in class FileSystem
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.replication
- required block replication for the file.
IOException
public FSOutputStream createRaw(Path f, boolean overwrite, short replication, long blockSize, Progressable progress) throws IOException
FileSystem
createRaw
in class FileSystem
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.replication
- required block replication for the file.
IOException
public FSOutputStream createRaw(Path f, org.apache.hadoop.fs.InMemoryFileSystem.FileAttributes fAttr) throws IOException
IOException
public void close() throws IOException
FileSystem
close
in class FileSystem
IOException
public short getReplication(Path src) throws IOException
getReplication
in class FileSystem
src
- file name
IOException
public boolean setReplicationRaw(Path src, short replication) throws IOException
FileSystem
setReplicationRaw
in class FileSystem
src
- file namereplication
- new replication
IOException
public boolean renameRaw(Path src, Path dst) throws IOException
FileSystem
renameRaw
in class FileSystem
IOException
public boolean deleteRaw(Path f) throws IOException
FileSystem
deleteRaw
in class FileSystem
IOException
public boolean exists(Path f) throws IOException
FileSystem
exists
in class FileSystem
IOException
public boolean isDirectory(Path f) throws IOException
isDirectory
in class FileSystem
IOException
public long getLength(Path f) throws IOException
FileSystem
getLength
in class FileSystem
IOException
public Path[] listPathsRaw(Path f) throws IOException
listPathsRaw
in class FileSystem
IOException
public void setWorkingDirectory(Path new_dir)
FileSystem
setWorkingDirectory
in class FileSystem
public Path getWorkingDirectory()
FileSystem
getWorkingDirectory
in class FileSystem
public boolean mkdirs(Path f) throws IOException
FileSystem
mkdirs
in class FileSystem
IOException
public void lock(Path f, boolean shared) throws IOException
lock
in class FileSystem
IOException
public void release(Path f) throws IOException
FileSystem
release
in class FileSystem
IOException
public void copyFromLocalFile(Path src, Path dst) throws IOException
copyFromLocalFile
in class FileSystem
IOException
public void moveFromLocalFile(Path src, Path dst) throws IOException
FileSystem
moveFromLocalFile
in class FileSystem
IOException
public void copyToLocalFile(Path src, Path dst, boolean copyCrc) throws IOException
FileSystem
copyToLocalFile
in class FileSystem
IOException
public Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
FileSystem
startLocalOutput
in class FileSystem
IOException
public void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
FileSystem
completeLocalOutput
in class FileSystem
IOException
public void reportChecksumFailure(Path p, FSInputStream in, long inPos, FSInputStream sums, long sumsPos)
FileSystem
reportChecksumFailure
in class FileSystem
p
- the file name containing the errorin
- the stream open on the fileinPos
- the position of the beginning of the bad data in the filesums
- the stream open on the checksum filesumsPos
- the position of the beginning of the bad data in the checksum filepublic long getBlockSize(Path f) throws IOException
FileSystem
getBlockSize
in class FileSystem
f
- the filename
IOException
public long getDefaultBlockSize()
FileSystem
getDefaultBlockSize
in class FileSystem
public short getDefaultReplication()
FileSystem
getDefaultReplication
in class FileSystem
public boolean reserveSpaceWithCheckSum(Path f, int size)
public int getChecksumFileLength(int size)
public Path[] getFiles(PathFilter filter)
public int getFSSize()
public float getPercentUsed()
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |