org.apache.hadoop.hdfs.server.datanode
Class DataNode

java.lang.Object
  extended by org.apache.hadoop.conf.Configured
      extended by org.apache.hadoop.hdfs.server.datanode.DataNode
All Implemented Interfaces:
Runnable, org.apache.hadoop.conf.Configurable, ClientDatanodeProtocol, FSConstants, DataNodeMXBean, InterDatanodeProtocol, org.apache.hadoop.ipc.VersionedProtocol

@InterfaceAudience.Private
public class DataNode
extends org.apache.hadoop.conf.Configured
implements InterDatanodeProtocol, ClientDatanodeProtocol, FSConstants, Runnable, DataNodeMXBean

DataNode is a class (and program) that stores a set of blocks for a DFS deployment. A single deployment can have one or many DataNodes. Each DataNode communicates regularly with a single NameNode. It also communicates with client code and other DataNodes from time to time. DataNodes store a series of named blocks. The DataNode allows client code to read these blocks, or to write new block data. The DataNode may also, in response to instructions from its NameNode, delete blocks or copy blocks to/from other DataNodes. The DataNode maintains just one critical table: block-> stream of bytes (of BLOCK_SIZE or less) This info is stored on a local disk. The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards. DataNodes spend their lives in an endless loop of asking the NameNode for something to do. A NameNode cannot connect to a DataNode directly; a NameNode simply returns values from functions invoked by a DataNode. DataNodes maintain an open server socket so that client code or other DataNodes can read/write data. The host/port for this server is reported to the NameNode, which then sends that information to clients or other DataNodes that might be interested.


Nested Class Summary
 
Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.protocol.FSConstants
FSConstants.DatanodeReportType, FSConstants.SafeModeAction, FSConstants.UpgradeAction
 
Field Summary
 org.apache.hadoop.hdfs.server.datanode.DataBlockScanner blockScanner
           
 org.apache.hadoop.util.Daemon blockScannerThread
           
 FSDatasetInterface data
           
static String DN_CLIENTTRACE_FORMAT
           
 DatanodeRegistration dnRegistration
           
static String EMPTY_DEL_HINT
           
 org.apache.hadoop.ipc.Server ipcServer
           
static org.apache.commons.logging.Log LOG
           
 DatanodeProtocol namenode
           
 
Fields inherited from interface org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol
versionID
 
Fields inherited from interface org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol
versionID
 
Fields inherited from interface org.apache.hadoop.hdfs.protocol.FSConstants
BLOCK_INVALIDATE_CHUNK, BLOCKREPORT_INITIAL_DELAY, BLOCKREPORT_INTERVAL, BUFFER_SIZE, DEFAULT_BLOCK_SIZE, DEFAULT_BYTES_PER_CHECKSUM, DEFAULT_DATA_SOCKET_SIZE, DEFAULT_FILE_BUFFER_SIZE, DEFAULT_REPLICATION_FACTOR, DEFAULT_WRITE_PACKET_SIZE, HDFS_URI_SCHEME, HEARTBEAT_INTERVAL, LAYOUT_VERSION, LEASE_HARDLIMIT_PERIOD, LEASE_RECOVER_PERIOD, LEASE_SOFTLIMIT_PERIOD, MAX_PATH_DEPTH, MAX_PATH_LENGTH, MIN_BLOCKS_FOR_WRITE, QUOTA_DONT_SET, QUOTA_RESET, SIZE_OF_INTEGER, SMALL_BUFFER_SIZE
 
Method Summary
protected  void checkDiskError()
          Check if there is a disk failure and if so, handle the error
protected  void checkDiskError(Exception e)
          Check if there is no space in disk
static DataNode createDataNode(String[] args, org.apache.hadoop.conf.Configuration conf)
          Instantiate & Start a single datanode daemon and wait for it to finish.
static DataNode createDataNode(String[] args, org.apache.hadoop.conf.Configuration conf, SecureDataNodeStarter.SecureResources resources)
          Instantiate & Start a single datanode daemon and wait for it to finish.
static InterDatanodeProtocol createInterDataNodeProtocolProxy(DatanodeID datanodeid, org.apache.hadoop.conf.Configuration conf, int socketTimeout)
           
static InetSocketAddress createSocketAddr(String target)
          Deprecated. 
static DataNode getDataNode()
          Return the DataNode object
 DatanodeRegistration getDatanodeRegistration()
          Return DatanodeRegistration
 FSDatasetInterface getFSDataset()
          This method is used for testing.
 String getHttpPort()
          Gets the http port.
static InetSocketAddress getInfoAddr(org.apache.hadoop.conf.Configuration conf)
          Determine the http server's effective addr
 int getInfoPort()
           
 InetSocketAddress getNameNodeAddr()
           
 String getNamenodeAddress()
          Gets the namenode IP address.
 InetSocketAddress getNameNodeAddrForClient()
           
 long getProtocolVersion(String protocol, long clientVersion)
          
 long getReplicaVisibleLength(Block block)
          Return the visible length of a replica.
 String getRpcPort()
          Gets the rpc port.
 InetSocketAddress getSelfAddr()
           
static InetSocketAddress getStreamingAddr(org.apache.hadoop.conf.Configuration conf)
           
 String getVersion()
          Gets the version of Hadoop.
 String getVolumeInfo()
          Returned information is a JSON representation of a map with volume name as the key and value is a map of volume attribute keys to its values
 ReplicaRecoveryInfo initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock)
          Initialize a replica recovery.
static DataNode instantiateDataNode(String[] args, org.apache.hadoop.conf.Configuration conf)
          Instantiate a single datanode object.
static DataNode instantiateDataNode(String[] args, org.apache.hadoop.conf.Configuration conf, SecureDataNodeStarter.SecureResources resources)
          Instantiate a single datanode object, along with its secure resources.
static void main(String[] args)
           
protected  Socket newSocket()
          Creates either NIO or regular depending on socketWriteTimeout.
protected  void notifyNamenodeReceivedBlock(Block block, String delHint)
           
 void offerService()
          Main loop for the DataNode.
 org.apache.hadoop.util.Daemon recoverBlocks(Collection<BlockRecoveryCommand.RecoveringBlock> blocks)
           
 void run()
          No matter what kind of exception we get, keep retrying to offerService().
static void runDatanodeDaemon(DataNode dn)
          Start a single datanode daemon and wait for it to finish.
 void scheduleBlockReport(long delay)
          This methods arranges for the data node to send the block report at the next heartbeat.
static void secureMain(String[] args, SecureDataNodeStarter.SecureResources resources)
           
static void setNewStorageID(DatanodeRegistration dnReg)
           
 void shutdown()
          Shut down this instance of the datanode.
 String toString()
           
 Block updateReplicaUnderRecovery(Block oldBlock, long recoveryId, long newLength)
          Update replica with the new generation stamp and length.
 
Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

DN_CLIENTTRACE_FORMAT

public static final String DN_CLIENTTRACE_FORMAT
See Also:
Constant Field Values

namenode

public DatanodeProtocol namenode

data

public FSDatasetInterface data

dnRegistration

public DatanodeRegistration dnRegistration

EMPTY_DEL_HINT

public static final String EMPTY_DEL_HINT
See Also:
Constant Field Values

blockScanner

public org.apache.hadoop.hdfs.server.datanode.DataBlockScanner blockScanner

blockScannerThread

public org.apache.hadoop.util.Daemon blockScannerThread

ipcServer

public org.apache.hadoop.ipc.Server ipcServer
Method Detail

createSocketAddr

@Deprecated
public static InetSocketAddress createSocketAddr(String target)
                                          throws IOException
Deprecated. 

Use NetUtils.createSocketAddr(String) instead.

Throws:
IOException

getInfoAddr

public static InetSocketAddress getInfoAddr(org.apache.hadoop.conf.Configuration conf)
Determine the http server's effective addr


newSocket

protected Socket newSocket()
                    throws IOException
Creates either NIO or regular depending on socketWriteTimeout.

Throws:
IOException

getDataNode

public static DataNode getDataNode()
Return the DataNode object


createInterDataNodeProtocolProxy

public static InterDatanodeProtocol createInterDataNodeProtocolProxy(DatanodeID datanodeid,
                                                                     org.apache.hadoop.conf.Configuration conf,
                                                                     int socketTimeout)
                                                              throws IOException
Throws:
IOException

getNameNodeAddr

public InetSocketAddress getNameNodeAddr()

getNameNodeAddrForClient

public InetSocketAddress getNameNodeAddrForClient()

getSelfAddr

public InetSocketAddress getSelfAddr()

getDatanodeRegistration

public DatanodeRegistration getDatanodeRegistration()
Return DatanodeRegistration


setNewStorageID

public static void setNewStorageID(DatanodeRegistration dnReg)

shutdown

public void shutdown()
Shut down this instance of the datanode. Returns only after shutdown is complete. This method can only be called by the offerService thread. Otherwise, deadlock might occur.


checkDiskError

protected void checkDiskError(Exception e)
                       throws IOException
Check if there is no space in disk

Parameters:
e - that caused this checkDiskError call
Throws:
IOException

checkDiskError

protected void checkDiskError()
Check if there is a disk failure and if so, handle the error


offerService

public void offerService()
                  throws Exception
Main loop for the DataNode. Runs until shutdown, forever calling remote NameNode functions.

Throws:
Exception

notifyNamenodeReceivedBlock

protected void notifyNamenodeReceivedBlock(Block block,
                                           String delHint)

run

public void run()
No matter what kind of exception we get, keep retrying to offerService(). That's the loop that connects to the NameNode and provides basic DataNode functionality. Only stop when "shouldRun" is turned off (which can only happen at shutdown).

Specified by:
run in interface Runnable

runDatanodeDaemon

public static void runDatanodeDaemon(DataNode dn)
                              throws IOException
Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

Throws:
IOException

instantiateDataNode

public static DataNode instantiateDataNode(String[] args,
                                           org.apache.hadoop.conf.Configuration conf)
                                    throws IOException
Instantiate a single datanode object. This must be run by invoking runDatanodeDaemon(DataNode) subsequently.

Throws:
IOException

instantiateDataNode

public static DataNode instantiateDataNode(String[] args,
                                           org.apache.hadoop.conf.Configuration conf,
                                           SecureDataNodeStarter.SecureResources resources)
                                    throws IOException
Instantiate a single datanode object, along with its secure resources. This must be run by invokingrunDatanodeDaemon(DataNode) subsequently.

Throws:
IOException

createDataNode

public static DataNode createDataNode(String[] args,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException
Instantiate & Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

Throws:
IOException

createDataNode

@InterfaceAudience.Private
public static DataNode createDataNode(String[] args,
                                                                org.apache.hadoop.conf.Configuration conf,
                                                                SecureDataNodeStarter.SecureResources resources)
                               throws IOException
Instantiate & Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

Throws:
IOException

toString

public String toString()
Overrides:
toString in class Object

scheduleBlockReport

public void scheduleBlockReport(long delay)
This methods arranges for the data node to send the block report at the next heartbeat.


getFSDataset

public FSDatasetInterface getFSDataset()
This method is used for testing. Examples are adding and deleting blocks directly. The most common usage will be when the data node's storage is similated.

Returns:
the fsdataset that stores the blocks

secureMain

public static void secureMain(String[] args,
                              SecureDataNodeStarter.SecureResources resources)

main

public static void main(String[] args)

recoverBlocks

public org.apache.hadoop.util.Daemon recoverBlocks(Collection<BlockRecoveryCommand.RecoveringBlock> blocks)

initReplicaRecovery

public ReplicaRecoveryInfo initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock)
                                        throws IOException
Description copied from interface: InterDatanodeProtocol
Initialize a replica recovery.

Specified by:
initReplicaRecovery in interface InterDatanodeProtocol
Returns:
actual state of the replica on this data-node or null if data-node does not have the replica.
Throws:
IOException

updateReplicaUnderRecovery

public Block updateReplicaUnderRecovery(Block oldBlock,
                                        long recoveryId,
                                        long newLength)
                                 throws IOException
Update replica with the new generation stamp and length.

Specified by:
updateReplicaUnderRecovery in interface InterDatanodeProtocol
Throws:
IOException

getProtocolVersion

public long getProtocolVersion(String protocol,
                               long clientVersion)
                        throws IOException

Specified by:
getProtocolVersion in interface org.apache.hadoop.ipc.VersionedProtocol
Throws:
IOException

getReplicaVisibleLength

public long getReplicaVisibleLength(Block block)
                             throws IOException
Return the visible length of a replica.

Specified by:
getReplicaVisibleLength in interface ClientDatanodeProtocol
Throws:
IOException

getStreamingAddr

public static InetSocketAddress getStreamingAddr(org.apache.hadoop.conf.Configuration conf)

getVersion

public String getVersion()
Description copied from interface: DataNodeMXBean
Gets the version of Hadoop.

Specified by:
getVersion in interface DataNodeMXBean
Returns:
the version of Hadoop

getRpcPort

public String getRpcPort()
Description copied from interface: DataNodeMXBean
Gets the rpc port.

Specified by:
getRpcPort in interface DataNodeMXBean
Returns:
the rpc port

getHttpPort

public String getHttpPort()
Description copied from interface: DataNodeMXBean
Gets the http port.

Specified by:
getHttpPort in interface DataNodeMXBean
Returns:
the http port

getInfoPort

public int getInfoPort()

getNamenodeAddress

public String getNamenodeAddress()
Description copied from interface: DataNodeMXBean
Gets the namenode IP address.

Specified by:
getNamenodeAddress in interface DataNodeMXBean
Returns:
the namenode IP address

getVolumeInfo

public String getVolumeInfo()
Returned information is a JSON representation of a map with volume name as the key and value is a map of volume attribute keys to its values

Specified by:
getVolumeInfo in interface DataNodeMXBean
Returns:
the volume info


Copyright © 2009 The Apache Software Foundation