|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hadoop.conf.Configured
org.apache.hadoop.fs.FileSystem
public abstract class FileSystem
An abstract base class for a fairly generic filesystem. It may be implemented as a distributed filesystem, or as a "local" one that reflects the locally-connected disk. The local version exists for small Hadopp instances and for testing.
All user code that may potentially use the Hadoop Distributed File System should be written to use a FileSystem object. The Hadoop DFS is a multi-machine system that appears as a single disk. It's useful because of its fault tolerance and potentially very large capacity.
The local implementation is LocalFileSystem
and distributed
implementation is DistributedFileSystem
.
Field Summary | |
---|---|
static org.apache.commons.logging.Log |
LOG
|
Constructor Summary | |
---|---|
protected |
FileSystem()
|
Method Summary | |
---|---|
protected void |
checkPath(Path path)
Check that a Path belongs to this FileSystem. |
void |
close()
No more filesystem operations are needed. |
static void |
closeAll()
Close all cached filesystems. |
void |
completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Called when we're all done writing to the target. |
void |
copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path src,
Path dst)
The src file is on the local disk. |
void |
copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is on the local disk. |
void |
copyFromLocalFile(Path src,
Path dst)
The src file is on the local disk. |
void |
copyToLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is under FS, and the dst is on the local disk. |
void |
copyToLocalFile(Path src,
Path dst)
The src file is under FS, and the dst is on the local disk. |
FSDataOutputStream |
create(Path f)
Opens an FSDataOutputStream at the indicated Path. |
FSDataOutputStream |
create(Path f,
boolean overwrite)
Opens an FSDataOutputStream at the indicated Path. |
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize)
Opens an FSDataOutputStream at the indicated Path. |
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress reporting. |
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Opens an FSDataOutputStream at the indicated Path. |
abstract FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress reporting. |
FSDataOutputStream |
create(Path f,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress reporting. |
FSDataOutputStream |
create(Path f,
short replication)
Opens an FSDataOutputStream at the indicated Path. |
FSDataOutputStream |
create(Path f,
short replication,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress reporting. |
boolean |
createNewFile(Path f)
Creates the given Path as a brand-new zero-length file. |
abstract boolean |
delete(Path f)
Delete a file |
abstract boolean |
exists(Path f)
Check if exists. |
static FileSystem |
get(Configuration conf)
Returns the configured filesystem implementation. |
static FileSystem |
get(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority. |
long |
getBlockSize(Path f)
Deprecated. Use getFileStatus() instead |
long |
getContentLength(Path f)
Return the number of bytes of the given path If f is a file, return the size of the file; If f is a directory, return the size of the directory tree |
long |
getDefaultBlockSize()
Return the number of bytes that large input files should be optimally be split into to minimize i/o time. |
short |
getDefaultReplication()
Get the default replication. |
String[][] |
getFileCacheHints(Path f,
long start,
long len)
Return a 2D array of size 1x1 or greater, containing hostnames where portions of the given file can be found. |
abstract FileStatus |
getFileStatus(Path f)
|
long |
getLength(Path f)
Deprecated. Use getFileStatus() instead |
static LocalFileSystem |
getLocal(Configuration conf)
Get the local file syste |
String |
getName()
Deprecated. call #getUri() instead. |
static FileSystem |
getNamed(String name,
Configuration conf)
Deprecated. call #get(URI,Configuration) instead. |
short |
getReplication(Path src)
Deprecated. Use getFileStatus() instead |
abstract URI |
getUri()
Returns a URI whose scheme and authority identify this FileSystem. |
long |
getUsed()
Return the total size of all files in the filesystem. |
abstract Path |
getWorkingDirectory()
Get the current working directory for the given file system |
Path[] |
globPaths(Path filePattern)
Return all the files that match filePattern and are not checksum files. |
Path[] |
globPaths(Path filePattern,
PathFilter filter)
glob all the file names that matches filePattern and is accepted by filter. |
abstract void |
initialize(URI name,
Configuration conf)
Called after a new FileSystem instance is constructed. |
boolean |
isDirectory(Path f)
Deprecated. Use getFileStatus() instead |
boolean |
isFile(Path f)
True iff the named path is a regular file. |
Path[] |
listPaths(Path f)
Deprecated. |
Path[] |
listPaths(Path[] files)
Filter files in the given pathes using the default checksum filter. |
Path[] |
listPaths(Path[] files,
PathFilter filter)
Filter files in a list directories using user-supplied path filter. |
Path[] |
listPaths(Path f,
PathFilter filter)
Filter files in a directory. |
abstract FileStatus[] |
listStatus(Path f)
|
Path |
makeQualified(Path path)
Make sure that a path specifies a FileSystem. |
abstract boolean |
mkdirs(Path f)
Make the given file and all non-existent parents into directories. |
void |
moveFromLocalFile(Path src,
Path dst)
The src file is on the local disk. |
void |
moveToLocalFile(Path src,
Path dst)
The src file is under FS, and the dst is on the local disk. |
FSDataInputStream |
open(Path f)
Opens an FSDataInputStream at the indicated Path. |
abstract FSDataInputStream |
open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path. |
static FileSystem |
parseArgs(String[] argv,
int i,
Configuration conf)
Parse the cmd-line args, starting at i. |
abstract boolean |
rename(Path src,
Path dst)
Renames Path src to Path dst. |
boolean |
setReplication(Path src,
short replication)
Set replication for an existing file. |
abstract void |
setWorkingDirectory(Path new_dir)
Set the current working directory for the given file system. |
Path |
startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to. |
Methods inherited from class org.apache.hadoop.conf.Configured |
---|
getConf, setConf |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final org.apache.commons.logging.Log LOG
Constructor Detail |
---|
protected FileSystem()
Method Detail |
---|
public static FileSystem parseArgs(String[] argv, int i, Configuration conf) throws IOException
IOException
public static FileSystem get(Configuration conf) throws IOException
IOException
public abstract void initialize(URI name, Configuration conf) throws IOException
name
- a uri whose authority section names the host, port, etc.
for this FileSystemconf
- the configuration
IOException
public abstract URI getUri()
public String getName()
public static FileSystem getNamed(String name, Configuration conf) throws IOException
IOException
public static LocalFileSystem getLocal(Configuration conf) throws IOException
conf
- the configuration to configure the file system with
IOException
public static FileSystem get(URI uri, Configuration conf) throws IOException
IOException
public static void closeAll() throws IOException
IOException
public Path makeQualified(Path path)
protected void checkPath(Path path)
public String[][] getFileCacheHints(Path f, long start, long len) throws IOException
IOException
public abstract FSDataInputStream open(Path f, int bufferSize) throws IOException
f
- the file name to openbufferSize
- the size of the buffer to be used.
IOException
public FSDataInputStream open(Path f) throws IOException
f
- the file to open
IOException
public FSDataOutputStream create(Path f) throws IOException
IOException
public FSDataOutputStream create(Path f, boolean overwrite) throws IOException
IOException
public FSDataOutputStream create(Path f, Progressable progress) throws IOException
IOException
public FSDataOutputStream create(Path f, short replication) throws IOException
IOException
public FSDataOutputStream create(Path f, short replication, Progressable progress) throws IOException
IOException
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize) throws IOException
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.
IOException
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, Progressable progress) throws IOException
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.
IOException
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize) throws IOException
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.
IOException
public abstract FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.
IOException
public boolean createNewFile(Path f) throws IOException
IOException
@Deprecated public short getReplication(Path src) throws IOException
src
- file name
IOException
public boolean setReplication(Path src, short replication) throws IOException
src
- file namereplication
- new replication
IOException
public abstract boolean rename(Path src, Path dst) throws IOException
IOException
public abstract boolean delete(Path f) throws IOException
IOException
public abstract boolean exists(Path f) throws IOException
f
- source file
IOException
@Deprecated public boolean isDirectory(Path f) throws IOException
IOException
public boolean isFile(Path f) throws IOException
IOException
@Deprecated public long getLength(Path f) throws IOException
IOException
public long getContentLength(Path f) throws IOException
IOException
@Deprecated public Path[] listPaths(Path f) throws IOException
IOException
public abstract FileStatus[] listStatus(Path f) throws IOException
IOException
public Path[] listPaths(Path[] files) throws IOException
files
- a list of paths
IOException
public Path[] listPaths(Path f, PathFilter filter) throws IOException
IOException
public Path[] listPaths(Path[] files, PathFilter filter) throws IOException
files
- a list of paths
IOException
public Path[] globPaths(Path filePattern) throws IOException
Return all the files that match filePattern and are not checksum files. Results are sorted by their names.
A filename pattern is composed of regular characters and special pattern matching characters, which are:
filePattern
- a regular expression specifying file pattern
IOException
public Path[] globPaths(Path filePattern, PathFilter filter) throws IOException
IOException
public abstract void setWorkingDirectory(Path new_dir)
new_dir
- public abstract Path getWorkingDirectory()
public abstract boolean mkdirs(Path f) throws IOException
IOException
public void copyFromLocalFile(Path src, Path dst) throws IOException
IOException
public void moveFromLocalFile(Path src, Path dst) throws IOException
IOException
public void copyFromLocalFile(boolean delSrc, Path src, Path dst) throws IOException
IOException
public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path src, Path dst) throws IOException
IOException
public void copyToLocalFile(Path src, Path dst) throws IOException
IOException
public void moveToLocalFile(Path src, Path dst) throws IOException
IOException
public void copyToLocalFile(boolean delSrc, Path src, Path dst) throws IOException
IOException
public Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
IOException
public void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
IOException
public void close() throws IOException
IOException
public long getUsed() throws IOException
IOException
@Deprecated public long getBlockSize(Path f) throws IOException
IOException
public long getDefaultBlockSize()
public short getDefaultReplication()
public abstract FileStatus getFileStatus(Path f) throws IOException
IOException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |