org.apache.hadoop.hdfs
Class BlockReader

java.lang.Object
  extended by java.io.InputStream
      extended by org.apache.hadoop.fs.FSInputStream
          extended by org.apache.hadoop.fs.FSInputChecker
              extended by org.apache.hadoop.hdfs.BlockReader
All Implemented Interfaces:
Closeable, org.apache.hadoop.fs.PositionedReadable, org.apache.hadoop.fs.Seekable

@InterfaceAudience.Private
public class BlockReader
extends org.apache.hadoop.fs.FSInputChecker

This is a wrapper around connection to datanode and understands checksum, offset etc. Terminology:

block
The hdfs block, typically large (~64MB).
chunk
A block is divided into chunks, each comes with a checksum. We want transfers to be chunk-aligned, to be able to verify checksums.
packet
A grouping of chunks used for transport. It contains a header, followed by checksum data, followed by real data.
Please see DataNode for the RPC specification.


Field Summary
 
Fields inherited from class org.apache.hadoop.fs.FSInputChecker
CHECKSUM_SIZE, file, LOG
 
Method Summary
 void close()
           
protected  long getChunkPosition(long pos)
           
static String getFileName(InetSocketAddress s, long blockId)
           
 boolean hasSentStatusCode()
          Whether the BlockReader has reached the end of its input stream and successfully sent a status code back to the datanode.
static BlockReader newBlockReader(Socket sock, String file, Block block, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, long startOffset, long len, int bufferSize)
           
static BlockReader newBlockReader(Socket sock, String file, Block block, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, long startOffset, long len, int bufferSize, boolean verifyChecksum)
          Java Doc required
static BlockReader newBlockReader(Socket sock, String file, Block block, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, long startOffset, long len, int bufferSize, boolean verifyChecksum, String clientName)
          Create a new BlockReader specifically to satisfy a read.
 int read()
           
 int read(byte[] buf, int off, int len)
           
 int readAll(byte[] buf, int offset, int len)
          kind of like readFully().
protected  int readChunk(long pos, byte[] buf, int offset, int len, byte[] checksumBuf)
           
 void seek(long pos)
           
 boolean seekToNewSource(long targetPos)
           
 long skip(long n)
           
 Socket takeSocket()
          Take the socket used to talk to the DN.
 
Methods inherited from class org.apache.hadoop.fs.FSInputChecker
available, checksum2long, getPos, mark, markSupported, needChecksum, readFully, reset, set
 
Methods inherited from class org.apache.hadoop.fs.FSInputStream
read, readFully, readFully
 
Methods inherited from class java.io.InputStream
read
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Method Detail

read

public int read(byte[] buf,
                int off,
                int len)
         throws IOException
Overrides:
read in class org.apache.hadoop.fs.FSInputChecker
Throws:
IOException

skip

public long skip(long n)
          throws IOException
Overrides:
skip in class org.apache.hadoop.fs.FSInputChecker
Throws:
IOException

read

public int read()
         throws IOException
Overrides:
read in class org.apache.hadoop.fs.FSInputChecker
Throws:
IOException

seekToNewSource

public boolean seekToNewSource(long targetPos)
                        throws IOException
Specified by:
seekToNewSource in interface org.apache.hadoop.fs.Seekable
Specified by:
seekToNewSource in class org.apache.hadoop.fs.FSInputStream
Throws:
IOException

seek

public void seek(long pos)
          throws IOException
Specified by:
seek in interface org.apache.hadoop.fs.Seekable
Overrides:
seek in class org.apache.hadoop.fs.FSInputChecker
Throws:
IOException

getChunkPosition

protected long getChunkPosition(long pos)
Specified by:
getChunkPosition in class org.apache.hadoop.fs.FSInputChecker

readChunk

protected int readChunk(long pos,
                        byte[] buf,
                        int offset,
                        int len,
                        byte[] checksumBuf)
                 throws IOException
Specified by:
readChunk in class org.apache.hadoop.fs.FSInputChecker
Throws:
IOException

newBlockReader

public static BlockReader newBlockReader(Socket sock,
                                         String file,
                                         Block block,
                                         org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
                                         long startOffset,
                                         long len,
                                         int bufferSize)
                                  throws IOException
Throws:
IOException

newBlockReader

public static BlockReader newBlockReader(Socket sock,
                                         String file,
                                         Block block,
                                         org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
                                         long startOffset,
                                         long len,
                                         int bufferSize,
                                         boolean verifyChecksum)
                                  throws IOException
Java Doc required

Throws:
IOException

newBlockReader

public static BlockReader newBlockReader(Socket sock,
                                         String file,
                                         Block block,
                                         org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
                                         long startOffset,
                                         long len,
                                         int bufferSize,
                                         boolean verifyChecksum,
                                         String clientName)
                                  throws IOException
Create a new BlockReader specifically to satisfy a read. This method also sends the OP_READ_BLOCK request.

Parameters:
sock - An established Socket to the DN. The BlockReader will not close it normally
file - File location
block - The block object
blockToken - The block token for security
startOffset - The read offset, relative to block head
len - The number of bytes to read
bufferSize - The IO buffer size (not the client buffer size)
verifyChecksum - Whether to verify checksum
clientName - Client name
Returns:
New BlockReader instance, or null on error.
Throws:
IOException

close

public void close()
           throws IOException
Specified by:
close in interface Closeable
Overrides:
close in class InputStream
Throws:
IOException

readAll

public int readAll(byte[] buf,
                   int offset,
                   int len)
            throws IOException
kind of like readFully(). Only reads as much as possible. And allows use of protected readFully().

Throws:
IOException

takeSocket

public Socket takeSocket()
Take the socket used to talk to the DN.


hasSentStatusCode

public boolean hasSentStatusCode()
Whether the BlockReader has reached the end of its input stream and successfully sent a status code back to the datanode.


getFileName

public static String getFileName(InetSocketAddress s,
                                 long blockId)


Copyright © 2009 The Apache Software Foundation