org.apache.hadoop.dfs
Class NameNode

java.lang.Object
  extended byorg.apache.hadoop.dfs.NameNode
All Implemented Interfaces:
org.apache.hadoop.dfs.ClientProtocol, org.apache.hadoop.dfs.DatanodeProtocol, FSConstants

public class NameNode
extends Object
implements org.apache.hadoop.dfs.ClientProtocol, org.apache.hadoop.dfs.DatanodeProtocol, FSConstants

NameNode serves as both directory namespace manager and "inode table" for the Hadoop DFS. There is a single NameNode running in any DFS deployment. (Well, except when there is a second backup/failover NameNode.) The NameNode controls two critical tables: 1) filename->blocksequence (namespace) 2) block->machinelist ("inodes") The first table is stored on disk and is very precious. The second table is rebuilt every time the NameNode comes up. 'NameNode' refers to both this class as well as the 'NameNode server'. The 'FSNamesystem' class actually performs most of the filesystem management. The majority of the 'NameNode' class itself is concerned with exposing the IPC interface to the outside world, plus some configuration management. NameNode implements the ClientProtocol interface, which allows clients to ask for DFS services. ClientProtocol is not designed for direct use by authors of DFS client code. End-users should instead use the org.apache.nutch.hadoop.fs.FileSystem class. NameNode also implements the DatanodeProtocol interface, used by DataNode programs that actually store DFS data blocks. These methods are invoked repeatedly and automatically by all the DataNodes in a DFS deployment.

Author:
Mike Cafarella

Field Summary
static int DISK_ERROR
           
static int INVALID_BLOCK
           
static org.apache.commons.logging.Log LOG
           
static org.apache.commons.logging.Log stateChangeLog
           
 
Fields inherited from interface org.apache.hadoop.dfs.FSConstants
BLOCKREPORT_INTERVAL, BUFFER_SIZE, CHUNKED_ENCODING, COMPLETE_SUCCESS, DATANODE_STARTUP_PERIOD, DFS_CURRENT_VERSION, EXPIRE_INTERVAL, HEARTBEAT_INTERVAL, LEASE_PERIOD, MIN_BLOCKS_FOR_WRITE, OP_ACK, OP_BLOCKRECEIVED, OP_BLOCKREPORT, OP_CLIENT_ABANDONBLOCK, OP_CLIENT_ABANDONBLOCK_ACK, OP_CLIENT_ADDBLOCK, OP_CLIENT_ADDBLOCK_ACK, OP_CLIENT_COMPLETEFILE, OP_CLIENT_COMPLETEFILE_ACK, OP_CLIENT_DATANODE_HINTS, OP_CLIENT_DATANODE_HINTS_ACK, OP_CLIENT_DATANODEREPORT, OP_CLIENT_DATANODEREPORT_ACK, OP_CLIENT_DELETE, OP_CLIENT_DELETE_ACK, OP_CLIENT_EXISTS, OP_CLIENT_EXISTS_ACK, OP_CLIENT_ISDIR, OP_CLIENT_ISDIR_ACK, OP_CLIENT_LISTING, OP_CLIENT_LISTING_ACK, OP_CLIENT_MKDIRS, OP_CLIENT_MKDIRS_ACK, OP_CLIENT_OBTAINLOCK, OP_CLIENT_OBTAINLOCK_ACK, OP_CLIENT_OPEN, OP_CLIENT_OPEN_ACK, OP_CLIENT_RAWSTATS, OP_CLIENT_RAWSTATS_ACK, OP_CLIENT_RELEASELOCK, OP_CLIENT_RELEASELOCK_ACK, OP_CLIENT_RENAMETO, OP_CLIENT_RENAMETO_ACK, OP_CLIENT_RENEW_LEASE, OP_CLIENT_RENEW_LEASE_ACK, OP_CLIENT_STARTFILE, OP_CLIENT_STARTFILE_ACK, OP_CLIENT_TRYAGAIN, OP_ERROR, OP_FAILURE, OP_HEARTBEAT, OP_INVALIDATE_BLOCKS, OP_READ_BLOCK, OP_READSKIP_BLOCK, OP_TRANSFERBLOCKS, OP_TRANSFERDATA, OP_WRITE_BLOCK, OPERATION_FAILED, READ_TIMEOUT, RUNLENGTH_ENCODING, STILL_WAITING, WRITE_COMPLETE
 
Constructor Summary
NameNode(Configuration conf)
          Create a NameNode at the default location
NameNode(File dir, int port, Configuration conf)
          Create a NameNode at the specified location and start it.
 
Method Summary
 void abandonBlock(org.apache.hadoop.dfs.Block b, String src)
          The client needs to give up on the block.
 void abandonFileInProgress(String src, String holder)
          A client that wants to abandon writing to the current file should call abandonFileInProgress().
 org.apache.hadoop.dfs.LocatedBlock addBlock(String src, String clientName)
          A client that wants to write an additional block to the indicated filename (which must currently be open for writing) should call addBlock().
 void blockReceived(org.apache.hadoop.dfs.DatanodeRegistration nodeReg, org.apache.hadoop.dfs.Block[] blocks)
          blockReceived() allows the DataNode to tell the NameNode about recently-received block data.
 org.apache.hadoop.dfs.Block[] blockReport(org.apache.hadoop.dfs.DatanodeRegistration nodeReg, org.apache.hadoop.dfs.Block[] blocks)
          blockReport() tells the NameNode about all the locally-stored blocks.
 boolean complete(String src, String clientName)
          The client is done writing data to the given filename, and would like to complete it.
 org.apache.hadoop.dfs.LocatedBlock create(String src, String clientName, String clientMachine, boolean overwrite, short replication, long blockSize)
          Create a new file.
 boolean delete(String src)
          Remove the given filename from the filesystem
 void errorReport(org.apache.hadoop.dfs.DatanodeRegistration nodeReg, int errorCode, String msg)
          errorReport() tells the NameNode about something that has gone awry.
 boolean exists(String src)
          Check whether the given file exists
static void format(Configuration conf)
          Format a new filesystem.
 long getBlockSize(String filename)
          Get the block size for the given file.
 DatanodeInfo[] getDatanodeReport()
          Get a full report on the system's current datanodes.
 String[][] getHints(String src, long start, long len)
          getHints() returns a list of hostnames that store data for a specific file region.
 org.apache.hadoop.dfs.DFSFileInfo[] getListing(String src)
          Get a listing of the indicated directory
 long[] getStats()
          Get a set of statistics about the filesystem.
 boolean isDir(String src)
          Check whether the given filename is a directory or not.
 void join()
          Wait for service to finish.
static void main(String[] argv)
           
 boolean mkdirs(String src)
          Create a directory (or hierarchy of directories) with the given name.
 boolean obtainLock(String src, String clientName, boolean exclusive)
          obtainLock() is used for lock managemnet.
 org.apache.hadoop.dfs.LocatedBlock[] open(String src)
          Open an existing file, at the given name.
 org.apache.hadoop.dfs.DatanodeRegistration register(org.apache.hadoop.dfs.DatanodeRegistration nodeReg)
          Register Datanode.
 boolean releaseLock(String src, String clientName)
          releaseLock() is called if the client would like to release a held lock.
 boolean rename(String src, String dst)
          Rename an item in the fs namespace
 void renewLease(String clientName)
          Client programs can cause stateful changes in the NameNode that affect other clients.
 void reportWrittenBlock(org.apache.hadoop.dfs.LocatedBlock lb)
          The client can report in a set written blocks that it wrote.
 org.apache.hadoop.dfs.BlockCommand sendHeartbeat(org.apache.hadoop.dfs.DatanodeRegistration nodeReg, long capacity, long remaining, int xmitsInProgress)
          Data node notify the name node that it is alive Return a block-oriented command for the datanode to execute.
 boolean setReplication(String src, short replication)
          Set replication for an existing file.
 void verifyRequest(org.apache.hadoop.dfs.DatanodeRegistration nodeReg)
          Verify request.
 void verifyVersion(int version)
          Verify version.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

stateChangeLog

public static final org.apache.commons.logging.Log stateChangeLog

DISK_ERROR

public static final int DISK_ERROR
See Also:
Constant Field Values

INVALID_BLOCK

public static final int INVALID_BLOCK
See Also:
Constant Field Values
Constructor Detail

NameNode

public NameNode(Configuration conf)
         throws IOException
Create a NameNode at the default location


NameNode

public NameNode(File dir,
                int port,
                Configuration conf)
         throws IOException
Create a NameNode at the specified location and start it.

Method Detail

format

public static void format(Configuration conf)
                   throws IOException
Format a new filesystem. Destroys any filesystem that may already exist at this location.

Throws:
IOException

join

public void join()
Wait for service to finish. (Normally, it runs forever.)


open

public org.apache.hadoop.dfs.LocatedBlock[] open(String src)
                                          throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Open an existing file, at the given name. Returns block and DataNode info. The client will then have to contact each indicated DataNode to obtain the actual data. There is no need to call close() or any other function after calling open().

Specified by:
open in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

create

public org.apache.hadoop.dfs.LocatedBlock create(String src,
                                                 String clientName,
                                                 String clientMachine,
                                                 boolean overwrite,
                                                 short replication,
                                                 long blockSize)
                                          throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Create a new file. Get back block and datanode info, which describes where the first block should be written. Successfully calling this method prevents any other client from creating a file under the given name, but the caller must invoke complete() for the file to be added to the filesystem. Blocks have a maximum size. Clients that intend to create multi-block files must also use reportWrittenBlock() and addBlock().

Specified by:
create in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

setReplication

public boolean setReplication(String src,
                              short replication)
                       throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Set replication for an existing file. The NameNode sets replication to the new value and returns. The actual block replication is not expected to be performed during this method call. The blocks will be populated or removed in the background as the result of the routine block maintenance procedures.

Specified by:
setReplication in interface org.apache.hadoop.dfs.ClientProtocol
Parameters:
src - file name
replication - new replication
Returns:
true if successful; false if file does not exist or is a directory
Throws:
IOException

addBlock

public org.apache.hadoop.dfs.LocatedBlock addBlock(String src,
                                                   String clientName)
                                            throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
A client that wants to write an additional block to the indicated filename (which must currently be open for writing) should call addBlock(). addBlock() returns block and datanode info, just like the initial call to create(). A null response means the NameNode could not allocate a block, and that the caller should try again.

Specified by:
addBlock in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

reportWrittenBlock

public void reportWrittenBlock(org.apache.hadoop.dfs.LocatedBlock lb)
                        throws IOException
The client can report in a set written blocks that it wrote. These blocks are reported via the client instead of the datanode to prevent weird heartbeat race conditions.

Specified by:
reportWrittenBlock in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

abandonBlock

public void abandonBlock(org.apache.hadoop.dfs.Block b,
                         String src)
                  throws IOException
The client needs to give up on the block.

Specified by:
abandonBlock in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

abandonFileInProgress

public void abandonFileInProgress(String src,
                                  String holder)
                           throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
A client that wants to abandon writing to the current file should call abandonFileInProgress(). After this call, any client can call create() to obtain the filename. Any blocks that have been written for the file will be garbage-collected.

Specified by:
abandonFileInProgress in interface org.apache.hadoop.dfs.ClientProtocol
Parameters:
src - The filename
holder - The datanode holding the lease
Throws:
IOException

complete

public boolean complete(String src,
                        String clientName)
                 throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
The client is done writing data to the given filename, and would like to complete it. The function returns whether the file has been closed successfully. If the function returns false, the caller should try again. A call to complete() will not return true until all the file's blocks have been replicated the minimum number of times. Thus, DataNode failures may cause a client to call complete() several times before succeeding.

Specified by:
complete in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

getHints

public String[][] getHints(String src,
                           long start,
                           long len)
                    throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
getHints() returns a list of hostnames that store data for a specific file region. It returns a set of hostnames for every block within the indicated region. This function is very useful when writing code that considers data-placement when performing operations. For example, the MapReduce system tries to schedule tasks on the same machines as the data-block the task processes.

Specified by:
getHints in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

getBlockSize

public long getBlockSize(String filename)
                  throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Get the block size for the given file.

Specified by:
getBlockSize in interface org.apache.hadoop.dfs.ClientProtocol
Parameters:
filename - The name of the file
Returns:
The number of bytes in each block
Throws:
IOException

rename

public boolean rename(String src,
                      String dst)
               throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Rename an item in the fs namespace

Specified by:
rename in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

delete

public boolean delete(String src)
               throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Remove the given filename from the filesystem

Specified by:
delete in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

exists

public boolean exists(String src)
               throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Check whether the given file exists

Specified by:
exists in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

isDir

public boolean isDir(String src)
              throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Check whether the given filename is a directory or not.

Specified by:
isDir in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

mkdirs

public boolean mkdirs(String src)
               throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Create a directory (or hierarchy of directories) with the given name.

Specified by:
mkdirs in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

obtainLock

public boolean obtainLock(String src,
                          String clientName,
                          boolean exclusive)
                   throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
obtainLock() is used for lock managemnet. It returns true if the lock has been seized correctly. It returns false if the lock could not be obtained, and the client should try again. Locking is a part of most filesystems and is useful for a number of inter-process synchronization tasks.

Specified by:
obtainLock in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

releaseLock

public boolean releaseLock(String src,
                           String clientName)
                    throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
releaseLock() is called if the client would like to release a held lock. It returns true if the lock is correctly released. It returns false if the client should wait and try again.

Specified by:
releaseLock in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

renewLease

public void renewLease(String clientName)
                throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Client programs can cause stateful changes in the NameNode that affect other clients. A client may obtain a file and neither abandon nor complete it. A client might hold a series of locks that prevent other clients from proceeding. Clearly, it would be bad if a client held a bunch of locks that it never gave up. This can happen easily if the client dies unexpectedly. So, the NameNode will revoke the locks and live file-creates for clients that it thinks have died. A client tells the NameNode that it is still alive by periodically calling renewLease(). If a certain amount of time passes since the last call to renewLease(), the NameNode assumes the client has died.

Specified by:
renewLease in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

getListing

public org.apache.hadoop.dfs.DFSFileInfo[] getListing(String src)
                                               throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Get a listing of the indicated directory

Specified by:
getListing in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

getStats

public long[] getStats()
                throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Get a set of statistics about the filesystem. Right now, only two values are returned. [0] contains the total storage capacity of the system, in bytes. [1] contains the available storage of the system, in bytes.

Specified by:
getStats in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

getDatanodeReport

public DatanodeInfo[] getDatanodeReport()
                                 throws IOException
Description copied from interface: org.apache.hadoop.dfs.ClientProtocol
Get a full report on the system's current datanodes. One DatanodeInfo object is returned for each DataNode.

Specified by:
getDatanodeReport in interface org.apache.hadoop.dfs.ClientProtocol
Throws:
IOException

register

public org.apache.hadoop.dfs.DatanodeRegistration register(org.apache.hadoop.dfs.DatanodeRegistration nodeReg)
                                                    throws IOException
Description copied from interface: org.apache.hadoop.dfs.DatanodeProtocol
Register Datanode.

Specified by:
register in interface org.apache.hadoop.dfs.DatanodeProtocol
Returns:
updated DatanodeRegistration, which contains new storageID if the datanode did not have one and registration ID for further communication.
Throws:
IOException
See Also:
DataNode#register(), FSNamesystem#registerDatanode(DatanodeRegistration)

sendHeartbeat

public org.apache.hadoop.dfs.BlockCommand sendHeartbeat(org.apache.hadoop.dfs.DatanodeRegistration nodeReg,
                                                        long capacity,
                                                        long remaining,
                                                        int xmitsInProgress)
                                                 throws IOException
Data node notify the name node that it is alive Return a block-oriented command for the datanode to execute. This will be either a transfer or a delete operation.

Specified by:
sendHeartbeat in interface org.apache.hadoop.dfs.DatanodeProtocol
Throws:
IOException

blockReport

public org.apache.hadoop.dfs.Block[] blockReport(org.apache.hadoop.dfs.DatanodeRegistration nodeReg,
                                                 org.apache.hadoop.dfs.Block[] blocks)
                                          throws IOException
Description copied from interface: org.apache.hadoop.dfs.DatanodeProtocol
blockReport() tells the NameNode about all the locally-stored blocks. The NameNode returns an array of Blocks that have become obsolete and should be deleted. This function is meant to upload *all* the locally-stored blocks. It's invoked upon startup and then infrequently afterwards.

Specified by:
blockReport in interface org.apache.hadoop.dfs.DatanodeProtocol
Throws:
IOException

blockReceived

public void blockReceived(org.apache.hadoop.dfs.DatanodeRegistration nodeReg,
                          org.apache.hadoop.dfs.Block[] blocks)
                   throws IOException
Description copied from interface: org.apache.hadoop.dfs.DatanodeProtocol
blockReceived() allows the DataNode to tell the NameNode about recently-received block data. For example, whenever client code writes a new Block here, or another DataNode copies a Block to this DataNode, it will call blockReceived().

Specified by:
blockReceived in interface org.apache.hadoop.dfs.DatanodeProtocol
Throws:
IOException

errorReport

public void errorReport(org.apache.hadoop.dfs.DatanodeRegistration nodeReg,
                        int errorCode,
                        String msg)
                 throws IOException
Description copied from interface: org.apache.hadoop.dfs.DatanodeProtocol
errorReport() tells the NameNode about something that has gone awry. Useful for debugging.

Specified by:
errorReport in interface org.apache.hadoop.dfs.DatanodeProtocol
Throws:
IOException

verifyRequest

public void verifyRequest(org.apache.hadoop.dfs.DatanodeRegistration nodeReg)
                   throws IOException
Verify request. Verifies correctness of the datanode version and registration ID.

Parameters:
nodeReg - data node registration
Throws:
IOException

verifyVersion

public void verifyVersion(int version)
                   throws IOException
Verify version.

Parameters:
version -
Throws:
IOException

main

public static void main(String[] argv)
                 throws Exception
Throws:
Exception


Copyright © 2006 The Apache Software Foundation