|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hadoop.filecache.DistributedCache
public class DistributedCache
The DistributedCache maintains all the caching information of cached archives and unarchives all the files as well and returns the path
Constructor Summary | |
---|---|
DistributedCache()
|
Method Summary | |
---|---|
static void |
addArchiveToClassPath(Path archive,
Configuration conf)
Add an archive path to the current set of classpath entries. |
static void |
addCacheArchive(URI uri,
Configuration conf)
Add a archives to be localized to the conf |
static void |
addCacheFile(URI uri,
Configuration conf)
Add a file to be localized to the conf |
static void |
addFileToClassPath(Path file,
Configuration conf)
Add an file path to the current set of classpath entries It adds the file to cache as well. |
static boolean |
checkURIs(URI[] uriFiles,
URI[] uriArchives)
This method checks if there is a conflict in the fragment names of the uris. |
static void |
createAllSymlink(Configuration conf,
File jobCacheDir,
File workDir)
This method create symlinks for all files in a given dir in another directory |
static void |
createSymlink(Configuration conf)
This method allows you to create symlinks in the current working directory of the task to all the cache files/archives |
static Path[] |
getArchiveClassPaths(Configuration conf)
Get the archive entries in classpath as an array of Path |
static String[] |
getArchiveTimestamps(Configuration conf)
Get the timestamps of the archives |
static URI[] |
getCacheArchives(Configuration conf)
Get cache archives set in the Configuration |
static URI[] |
getCacheFiles(Configuration conf)
Get cache files set in the Configuration |
static Path[] |
getFileClassPaths(Configuration conf)
Get the file entries in classpath as an array of Path |
static String[] |
getFileTimestamps(Configuration conf)
Get the timestamps of the files |
static Path |
getLocalCache(URI cache,
Configuration conf,
Path baseDir,
boolean isArchive,
long confFileStamp,
Path currentWorkDir)
Get the locally cached file or archive; it could either be previously cached (and valid) or copy it from the FileSystem now. |
static Path[] |
getLocalCacheArchives(Configuration conf)
Return the path array of the localized caches |
static Path[] |
getLocalCacheFiles(Configuration conf)
Return the path array of the localized files |
static boolean |
getSymlink(Configuration conf)
This method checks to see if symlinks are to be create for the localized cache files in the current working directory |
static long |
getTimestamp(Configuration conf,
URI cache)
Returns mtime of a given cache file on hdfs. |
static void |
purgeCache(Configuration conf)
Clear the entire contents of the cache and delete the backing files. |
static void |
releaseCache(URI cache,
Configuration conf)
This is the opposite of getlocalcache. |
static void |
setArchiveTimestamps(Configuration conf,
String timestamps)
This is to check the timestamp of the archives to be localized |
static void |
setCacheArchives(URI[] archives,
Configuration conf)
Set the configuration with the given set of archives |
static void |
setCacheFiles(URI[] files,
Configuration conf)
Set the configuration with the given set of files |
static void |
setFileTimestamps(Configuration conf,
String timestamps)
This is to check the timestamp of the files to be localized |
static void |
setLocalArchives(Configuration conf,
String str)
Set the conf to contain the location for localized archives |
static void |
setLocalFiles(Configuration conf,
String str)
Set the conf to contain the location for localized files |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public DistributedCache()
Method Detail |
---|
public static Path getLocalCache(URI cache, Configuration conf, Path baseDir, boolean isArchive, long confFileStamp, Path currentWorkDir) throws IOException
FileSystem
now.
cache
- the cache to be localized, this should be specified as
new URI(hdfs://hostname:port/absoulte_path_to_file#LINKNAME). If no schema
or hostname:port is provided the file is assumed to be in the filesystem
being used in the Configurationconf
- The Confguration file which contains the filesystembaseDir
- The base cache Dir where you wnat to localize the files/archivesisArchive
- if the cache is an archive or a file. In case it is an archive
with a .zip or .jar extension it will be unzipped/unjarred automatically
and the directory where the archive is unjarred is returned as the Path.
In case of a file, the path to the file is returnedconfFileStamp
- this is the hdfs file modification timestamp to verify that the
file to be cached hasn't changed since the job startedcurrentWorkDir
- this is the directory where you would want to create symlinks
for the locally cached files/archives
IOException
public static void releaseCache(URI cache, Configuration conf) throws IOException
cache
- The cache URI to be releasedconf
- configuration which contains the filesystem the cache
is contained in.
IOException
public static long getTimestamp(Configuration conf, URI cache) throws IOException
conf
- configurationcache
- cache file
IOException
public static void createAllSymlink(Configuration conf, File jobCacheDir, File workDir) throws IOException
conf
- the configurationjobCacheDir
- the target directory for creating symlinksworkDir
- the directory in which the symlinks are created
IOException
public static void setCacheArchives(URI[] archives, Configuration conf)
archives
- The list of archives that need to be localizedconf
- Configuration which will be changedpublic static void setCacheFiles(URI[] files, Configuration conf)
files
- The list of files that need to be localizedconf
- Configuration which will be changedpublic static URI[] getCacheArchives(Configuration conf) throws IOException
conf
- The configuration which contains the archives
IOException
public static URI[] getCacheFiles(Configuration conf) throws IOException
conf
- The configuration which contains the files
IOException
public static Path[] getLocalCacheArchives(Configuration conf) throws IOException
conf
- Configuration that contains the localized archives
IOException
public static Path[] getLocalCacheFiles(Configuration conf) throws IOException
conf
- Configuration that contains the localized files
IOException
public static String[] getArchiveTimestamps(Configuration conf)
conf
- The configuration which stored the timestamps
IOException
public static String[] getFileTimestamps(Configuration conf)
conf
- The configuration which stored the timestamps
IOException
public static void setArchiveTimestamps(Configuration conf, String timestamps)
conf
- Configuration which stores the timestamp'stimestamps
- comma separated list of timestamps of archives.
The order should be the same as the order in which the archives are added.public static void setFileTimestamps(Configuration conf, String timestamps)
conf
- Configuration which stores the timestamp'stimestamps
- comma separated list of timestamps of files.
The order should be the same as the order in which the files are added.public static void setLocalArchives(Configuration conf, String str)
conf
- The conf to modify to contain the localized cachesstr
- a comma separated list of local archivespublic static void setLocalFiles(Configuration conf, String str)
conf
- The conf to modify to contain the localized cachesstr
- a comma separated list of local filespublic static void addCacheArchive(URI uri, Configuration conf)
uri
- The uri of the cache to be localizedconf
- Configuration to add the cache topublic static void addCacheFile(URI uri, Configuration conf)
uri
- The uri of the cache to be localizedconf
- Configuration to add the cache topublic static void addFileToClassPath(Path file, Configuration conf) throws IOException
file
- Path of the file to be addedconf
- Configuration that contains the classpath setting
IOException
public static Path[] getFileClassPaths(Configuration conf)
conf
- Configuration that contains the classpath settingpublic static void addArchiveToClassPath(Path archive, Configuration conf) throws IOException
archive
- Path of the archive to be addedconf
- Configuration that contains the classpath setting
IOException
public static Path[] getArchiveClassPaths(Configuration conf)
conf
- Configuration that contains the classpath settingpublic static void createSymlink(Configuration conf)
conf
- the jobconfpublic static boolean getSymlink(Configuration conf)
conf
- the jobconf
public static boolean checkURIs(URI[] uriFiles, URI[] uriArchives)
uriFiles
- The uri array of urifilesuriArchives
- the uri array of uri archivespublic static void purgeCache(Configuration conf) throws IOException
IOException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |