org.apache.hadoop.dfs
Class DataNode

java.lang.Object
  extended byorg.apache.hadoop.dfs.DataNode
All Implemented Interfaces:
org.apache.hadoop.dfs.FSConstants, Runnable

public class DataNode
extends Object
implements org.apache.hadoop.dfs.FSConstants, Runnable

DataNode is a class (and program) that stores a set of blocks for a DFS deployment. A single deployment can have one or many DataNodes. Each DataNode communicates regularly with a single NameNode. It also communicates with client code and other DataNodes from time to time. DataNodes store a series of named blocks. The DataNode allows client code to read these blocks, or to write new block data. The DataNode may also, in response to instructions from its NameNode, delete blocks or copy blocks to/from other DataNodes. The DataNode maintains just one critical table: block-> stream of bytes (of BLOCK_SIZE or less) This info is stored on a local disk. The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards. DataNodes spend their lives in an endless loop of asking the NameNode for something to do. A NameNode cannot connect to a DataNode directly; a NameNode simply returns values from functions invoked by a DataNode. DataNodes maintain an open server socket so that client code or other DataNodes can read/write data. The host/port for this server is reported to the NameNode, which then sends that information to clients or other DataNodes that might be interested.

Author:
Mike Cafarella

Field Summary
static int BLOCK_SIZE
           
static long BLOCKREPORT_INTERVAL
           
static int BUFFER_SIZE
           
static byte CHUNKED_ENCODING
           
static int COMPLETE_SUCCESS
           
static long DATANODE_STARTUP_PERIOD
           
static long EXPIRE_INTERVAL
           
static long HEARTBEAT_INTERVAL
           
static long LEASE_PERIOD
           
static Logger LOG
           
static int MIN_BLOCKS_FOR_WRITE
           
static byte OP_ACK
           
static byte OP_BLOCKRECEIVED
           
static byte OP_BLOCKREPORT
           
static byte OP_CLIENT_ABANDONBLOCK
           
static byte OP_CLIENT_ABANDONBLOCK_ACK
           
static byte OP_CLIENT_ADDBLOCK
           
static byte OP_CLIENT_ADDBLOCK_ACK
           
static byte OP_CLIENT_COMPLETEFILE
           
static byte OP_CLIENT_COMPLETEFILE_ACK
           
static byte OP_CLIENT_DATANODE_HINTS
           
static byte OP_CLIENT_DATANODE_HINTS_ACK
           
static byte OP_CLIENT_DATANODEREPORT
           
static byte OP_CLIENT_DATANODEREPORT_ACK
           
static byte OP_CLIENT_DELETE
           
static byte OP_CLIENT_DELETE_ACK
           
static byte OP_CLIENT_EXISTS
           
static byte OP_CLIENT_EXISTS_ACK
           
static byte OP_CLIENT_ISDIR
           
static byte OP_CLIENT_ISDIR_ACK
           
static byte OP_CLIENT_LISTING
           
static byte OP_CLIENT_LISTING_ACK
           
static byte OP_CLIENT_MKDIRS
           
static byte OP_CLIENT_MKDIRS_ACK
           
static byte OP_CLIENT_OBTAINLOCK
           
static byte OP_CLIENT_OBTAINLOCK_ACK
           
static byte OP_CLIENT_OPEN
           
static byte OP_CLIENT_OPEN_ACK
           
static byte OP_CLIENT_RAWSTATS
           
static byte OP_CLIENT_RAWSTATS_ACK
           
static byte OP_CLIENT_RELEASELOCK
           
static byte OP_CLIENT_RELEASELOCK_ACK
           
static byte OP_CLIENT_RENAMETO
           
static byte OP_CLIENT_RENAMETO_ACK
           
static byte OP_CLIENT_RENEW_LEASE
           
static byte OP_CLIENT_RENEW_LEASE_ACK
           
static byte OP_CLIENT_STARTFILE
           
static byte OP_CLIENT_STARTFILE_ACK
           
static byte OP_CLIENT_TRYAGAIN
           
static byte OP_ERROR
           
static byte OP_FAILURE
           
static byte OP_HEARTBEAT
           
static byte OP_INVALIDATE_BLOCKS
           
static byte OP_READ_BLOCK
           
static byte OP_READSKIP_BLOCK
           
static byte OP_TRANSFERBLOCKS
           
static byte OP_TRANSFERDATA
           
static byte OP_WRITE_BLOCK
           
static int OPERATION_FAILED
           
static int READ_TIMEOUT
           
static byte RUNLENGTH_ENCODING
           
static int STILL_WAITING
           
static long WRITE_COMPLETE
           
 
Constructor Summary
DataNode(Configuration conf, String datadir)
          Create the DataNode given a configuration and a dataDir.
DataNode(String machineName, File datadir, InetSocketAddress nameNodeAddr, Configuration conf)
          A DataNode can also be created with configuration information explicitly given.
 
Method Summary
static InetSocketAddress createSocketAddr(String s)
          Util method to build socket addr from string
 String getNamenode()
          Return the namenode's identifier
static void main(String[] args)
           
 void offerService()
          Main loop for the DataNode.
 void run()
          No matter what kind of exception we get, keep retrying to offerService().
static void run(Configuration conf)
          Start datanode daemons.
 String toString()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

LOG

public static final Logger LOG

BLOCK_SIZE

public static final int BLOCK_SIZE
See Also:
Constant Field Values

MIN_BLOCKS_FOR_WRITE

public static final int MIN_BLOCKS_FOR_WRITE
See Also:
Constant Field Values

WRITE_COMPLETE

public static final long WRITE_COMPLETE
See Also:
Constant Field Values

OP_ERROR

public static final byte OP_ERROR
See Also:
Constant Field Values

OP_HEARTBEAT

public static final byte OP_HEARTBEAT
See Also:
Constant Field Values

OP_BLOCKRECEIVED

public static final byte OP_BLOCKRECEIVED
See Also:
Constant Field Values

OP_BLOCKREPORT

public static final byte OP_BLOCKREPORT
See Also:
Constant Field Values

OP_TRANSFERDATA

public static final byte OP_TRANSFERDATA
See Also:
Constant Field Values

OP_CLIENT_OPEN

public static final byte OP_CLIENT_OPEN
See Also:
Constant Field Values

OP_CLIENT_STARTFILE

public static final byte OP_CLIENT_STARTFILE
See Also:
Constant Field Values

OP_CLIENT_ADDBLOCK

public static final byte OP_CLIENT_ADDBLOCK
See Also:
Constant Field Values

OP_CLIENT_RENAMETO

public static final byte OP_CLIENT_RENAMETO
See Also:
Constant Field Values

OP_CLIENT_DELETE

public static final byte OP_CLIENT_DELETE
See Also:
Constant Field Values

OP_CLIENT_COMPLETEFILE

public static final byte OP_CLIENT_COMPLETEFILE
See Also:
Constant Field Values

OP_CLIENT_LISTING

public static final byte OP_CLIENT_LISTING
See Also:
Constant Field Values

OP_CLIENT_OBTAINLOCK

public static final byte OP_CLIENT_OBTAINLOCK
See Also:
Constant Field Values

OP_CLIENT_RELEASELOCK

public static final byte OP_CLIENT_RELEASELOCK
See Also:
Constant Field Values

OP_CLIENT_EXISTS

public static final byte OP_CLIENT_EXISTS
See Also:
Constant Field Values

OP_CLIENT_ISDIR

public static final byte OP_CLIENT_ISDIR
See Also:
Constant Field Values

OP_CLIENT_MKDIRS

public static final byte OP_CLIENT_MKDIRS
See Also:
Constant Field Values

OP_CLIENT_RENEW_LEASE

public static final byte OP_CLIENT_RENEW_LEASE
See Also:
Constant Field Values

OP_CLIENT_ABANDONBLOCK

public static final byte OP_CLIENT_ABANDONBLOCK
See Also:
Constant Field Values

OP_CLIENT_RAWSTATS

public static final byte OP_CLIENT_RAWSTATS
See Also:
Constant Field Values

OP_CLIENT_DATANODEREPORT

public static final byte OP_CLIENT_DATANODEREPORT
See Also:
Constant Field Values

OP_CLIENT_DATANODE_HINTS

public static final byte OP_CLIENT_DATANODE_HINTS
See Also:
Constant Field Values

OP_ACK

public static final byte OP_ACK
See Also:
Constant Field Values

OP_TRANSFERBLOCKS

public static final byte OP_TRANSFERBLOCKS
See Also:
Constant Field Values

OP_INVALIDATE_BLOCKS

public static final byte OP_INVALIDATE_BLOCKS
See Also:
Constant Field Values

OP_FAILURE

public static final byte OP_FAILURE
See Also:
Constant Field Values

OP_CLIENT_OPEN_ACK

public static final byte OP_CLIENT_OPEN_ACK
See Also:
Constant Field Values

OP_CLIENT_STARTFILE_ACK

public static final byte OP_CLIENT_STARTFILE_ACK
See Also:
Constant Field Values

OP_CLIENT_ADDBLOCK_ACK

public static final byte OP_CLIENT_ADDBLOCK_ACK
See Also:
Constant Field Values

OP_CLIENT_RENAMETO_ACK

public static final byte OP_CLIENT_RENAMETO_ACK
See Also:
Constant Field Values

OP_CLIENT_DELETE_ACK

public static final byte OP_CLIENT_DELETE_ACK
See Also:
Constant Field Values

OP_CLIENT_COMPLETEFILE_ACK

public static final byte OP_CLIENT_COMPLETEFILE_ACK
See Also:
Constant Field Values

OP_CLIENT_TRYAGAIN

public static final byte OP_CLIENT_TRYAGAIN
See Also:
Constant Field Values

OP_CLIENT_LISTING_ACK

public static final byte OP_CLIENT_LISTING_ACK
See Also:
Constant Field Values

OP_CLIENT_OBTAINLOCK_ACK

public static final byte OP_CLIENT_OBTAINLOCK_ACK
See Also:
Constant Field Values

OP_CLIENT_RELEASELOCK_ACK

public static final byte OP_CLIENT_RELEASELOCK_ACK
See Also:
Constant Field Values

OP_CLIENT_EXISTS_ACK

public static final byte OP_CLIENT_EXISTS_ACK
See Also:
Constant Field Values

OP_CLIENT_ISDIR_ACK

public static final byte OP_CLIENT_ISDIR_ACK
See Also:
Constant Field Values

OP_CLIENT_MKDIRS_ACK

public static final byte OP_CLIENT_MKDIRS_ACK
See Also:
Constant Field Values

OP_CLIENT_RENEW_LEASE_ACK

public static final byte OP_CLIENT_RENEW_LEASE_ACK
See Also:
Constant Field Values

OP_CLIENT_ABANDONBLOCK_ACK

public static final byte OP_CLIENT_ABANDONBLOCK_ACK
See Also:
Constant Field Values

OP_CLIENT_RAWSTATS_ACK

public static final byte OP_CLIENT_RAWSTATS_ACK
See Also:
Constant Field Values

OP_CLIENT_DATANODEREPORT_ACK

public static final byte OP_CLIENT_DATANODEREPORT_ACK
See Also:
Constant Field Values

OP_CLIENT_DATANODE_HINTS_ACK

public static final byte OP_CLIENT_DATANODE_HINTS_ACK
See Also:
Constant Field Values

OP_WRITE_BLOCK

public static final byte OP_WRITE_BLOCK
See Also:
Constant Field Values

OP_READ_BLOCK

public static final byte OP_READ_BLOCK
See Also:
Constant Field Values

OP_READSKIP_BLOCK

public static final byte OP_READSKIP_BLOCK
See Also:
Constant Field Values

RUNLENGTH_ENCODING

public static final byte RUNLENGTH_ENCODING
See Also:
Constant Field Values

CHUNKED_ENCODING

public static final byte CHUNKED_ENCODING
See Also:
Constant Field Values

OPERATION_FAILED

public static final int OPERATION_FAILED
See Also:
Constant Field Values

STILL_WAITING

public static final int STILL_WAITING
See Also:
Constant Field Values

COMPLETE_SUCCESS

public static final int COMPLETE_SUCCESS
See Also:
Constant Field Values

HEARTBEAT_INTERVAL

public static final long HEARTBEAT_INTERVAL
See Also:
Constant Field Values

EXPIRE_INTERVAL

public static final long EXPIRE_INTERVAL
See Also:
Constant Field Values

BLOCKREPORT_INTERVAL

public static final long BLOCKREPORT_INTERVAL
See Also:
Constant Field Values

DATANODE_STARTUP_PERIOD

public static final long DATANODE_STARTUP_PERIOD
See Also:
Constant Field Values

LEASE_PERIOD

public static final long LEASE_PERIOD
See Also:
Constant Field Values

READ_TIMEOUT

public static final int READ_TIMEOUT
See Also:
Constant Field Values

BUFFER_SIZE

public static final int BUFFER_SIZE
Constructor Detail

DataNode

public DataNode(Configuration conf,
                String datadir)
         throws IOException
Create the DataNode given a configuration and a dataDir. 'dataDir' is where the blocks are stored.


DataNode

public DataNode(String machineName,
                File datadir,
                InetSocketAddress nameNodeAddr,
                Configuration conf)
         throws IOException
A DataNode can also be created with configuration information explicitly given.

Method Detail

createSocketAddr

public static InetSocketAddress createSocketAddr(String s)
                                          throws IOException
Util method to build socket addr from string

Throws:
IOException

getNamenode

public String getNamenode()
Return the namenode's identifier


offerService

public void offerService()
                  throws Exception
Main loop for the DataNode. Runs until shutdown, forever calling remote NameNode functions.

Throws:
Exception

run

public void run()
No matter what kind of exception we get, keep retrying to offerService(). That's the loop that connects to the NameNode and provides basic DataNode functionality. Only stop when "shouldRun" is turned off (which can only happen at shutdown).

Specified by:
run in interface Runnable

run

public static void run(Configuration conf)
                throws IOException
Start datanode daemons. Start a datanode daemon for each comma separated data directory specified in property dfs.data.dir

Throws:
IOException

toString

public String toString()

main

public static void main(String[] args)
                 throws IOException
Throws:
IOException


Copyright © 2006 The Apache Software Foundation