Hadoop HDFS 2.0.3-alpha Release Notes

• HDFS-4468. Minor bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE
  Fix TestHDFSCLI and TestQuota for HADOOP-9252
  

• HDFS-4462. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (namenode)
  2NN will fail to checkpoint after an HDFS upgrade from a pre-federation version of HDFS
  

• HDFS-4458. Major bug reported by wenwupeng and fixed by Binglin Chang (balancer)
  start balancer failed with "Failed to create file [/system/balancer.id]" if configure IP on fs.defaultFS
  

• HDFS-4456. Major new feature reported by Tsz Wo (Nicholas), SZE and fixed by Plamen Jeliazkov (webhdfs)
  Add concat to HttpFS and WebHDFS REST API docs
  

• HDFS-4452. Critical bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (namenode)
  getAdditionalBlock() can create multiple blocks if the client times out and retries.
  

• HDFS-4451. Major bug reported by Joshua Blatt and fixed by (balancer)
  hdfs balancer command returns exit code 1 on success instead of 0
  

  This is an incompatible change from release 2.0.2-alpha and prior releases. Balancer tool exited with exit code 1 on success. It is changed to exit with exit code 0 on success. Non 0 exit code indicates failure.

• HDFS-4445. Blocker sub-task reported by Vinay and fixed by Vinay
  All BKJM ledgers are not checked while tailing, So failover will fail.
  

• HDFS-4444. Trivial bug reported by Stephen Chu and fixed by Stephen Chu
  Add space between total transaction time and number of transactions in FSEditLog#printStatistics
  

• HDFS-4443. Trivial bug reported by Christian Rohling and fixed by Christian Rohling (namenode)
  Remove trailing '`' character from HDFS nodelist jsp
  

• HDFS-4428. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  FsDatasetImpl should disclose what the error is when a rename fails
  

• HDFS-4426. Blocker bug reported by Jason Lowe and fixed by Arpit Agarwal (namenode)
  Secondary namenode shuts down immediately after startup
  

• HDFS-4415. Major bug reported by Robert Kanter and fixed by Robert Kanter
  HostnameFilter should handle hostname resolution failures and continue processing
  

• HDFS-4404. Critical bug reported by liaowenrui and fixed by Todd Lipcon (ha , hdfs-client)
  Create file failure when the machine of first attempted NameNode is down
  

• HDFS-4403. Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (hdfs-client)
  DFSClient can infer checksum type when not provided by reading first byte
  

  The HDFS implementation of getFileChecksum() can now operate correctly against earlier-version datanodes which do not include the checksum type information in their checksum response. The checksum type is automatically inferred by issuing a read of the first byte of each block.

• HDFS-4393. Minor improvement reported by Brandon Li and fixed by Brandon Li
  Empty request and responses in protocol translators can be static final members
  

• HDFS-4392. Trivial improvement reported by Andrew Purtell and fixed by Andrew Purtell (test)
  Use NetUtils#getFreeSocketPort in MiniDFSCluster
  

• HDFS-4385. Critical bug reported by Thomas Graves and fixed by Thomas Graves (build)
  Maven RAT plugin is not checking all source files
  

• HDFS-4384. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  test_libhdfs_threaded gets SEGV if JNIEnv cannot be initialized
  

• HDFS-4381. Major improvement reported by Jing Zhao and fixed by Jing Zhao (namenode)
  Document fsimage format details in FSImageFormat class javadoc
  

• HDFS-4377. Trivial bug reported by Eli Collins and fixed by Eli Collins
  Some trivial DN comment cleanup
  

• HDFS-4375. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode , security)
  Use token request messages defined in hadoop common
  

• HDFS-4369. Blocker bug reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
  GetBlockKeysResponseProto does not handle null response
  

  Protobuf message GetBlockKeysResponseProto member keys is made optional from required so that null values can be passed over the wire. This is an incompatible wire protocol change and does not affect the API backward compatibility.

• HDFS-4367. Blocker bug reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
  GetDataEncryptionKeyResponseProto does not handle null response
  

  Member dataEncryptionKey of the protobuf message GetDataEncryptionKeyResponseProto is made optional instead of required. This is incompatible change is not likely to affect the existing users (that are using HDFS FileSystem and other public APIs).

• HDFS-4364. Blocker bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  GetLinkTargetResponseProto does not handle null path
  

  Protobuf message GetLinkTargetResponseProto member targetPath is made optional from required so that null values can be passed over the wire. This is an incompatible wire protocol change and does not affect the API backward compatibility.

• HDFS-4363. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  Combine PBHelper and HdfsProtoUtil and remove redundant methods
  

• HDFS-4362. Critical bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  GetDelegationTokenResponseProto does not handle null token
  

• HDFS-4359. Major bug reported by Liang Xie and fixed by Liang Xie (datanode)
  remove an unnecessary synchronized keyword in BPOfferService.java
  

• HDFS-4351. Major bug reported by Andrew Wang and fixed by Andrew Wang (namenode)
  Fix BlockPlacementPolicyDefault#chooseTarget when avoiding stale nodes
  

• HDFS-4350. Major bug reported by Andrew Wang and fixed by Andrew Wang
  Make enabling of stale marking on read and write paths independent
  

  This patch makes an incompatible configuration change, as described below: In releases 1.1.0 and other point releases 1.1.x, the configuration parameter "dfs.namenode.check.stale.datanode" could be used to turn on checking for the stale nodes. This configuration is no longer supported in release 1.2.0 onwards and is renamed as "dfs.namenode.avoid.read.stale.datanode". How feature works and configuring this feature: As described in HDFS-3703 release notes, datanode stale period can be configured using parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds). NameNode can be configured to use this staleness information for reads using configuration "dfs.namenode.avoid.read.stale.datanode". When this parameter is set to true, namenode picks a stale datanode as the last target to read from when returning block locations for reads. Using staleness information for writes is as described in the releases notes of HDFS-3912.

• HDFS-4349. Major test reported by Konstantin Shvachko and fixed by Konstantin Shvachko (namenode , test)
  Test reading files from BackupNode
  

• HDFS-4347. Major bug reported by Konstantin Shvachko and fixed by Plamen Jeliazkov (namenode , test)
  TestBackupNode can go into infinite loop "Waiting checkpoint to complete."
  

• HDFS-4344. Major bug reported by tamtam180 and fixed by Andy Isaacson (namenode)
  dfshealth.jsp throws NumberFormatException when dfs.hosts/dfs.hosts.exclude includes port number
  

• HDFS-4326. Major task reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  bump up Tomcat version for HttpFS to 6.0.36
  

• HDFS-4315. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (datanode)
  DNs with multiple BPs can have BPOfferServices fail to start due to unsynchronized map access
  

• HDFS-4308. Major bug reported by Konstantin Shvachko and fixed by Plamen Jeliazkov (namenode)
  addBlock() should persist file blocks once
  

• HDFS-4307. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  SocketCache should use monotonic time
  

• HDFS-4306. Major bug reported by Binglin Chang and fixed by Binglin Chang
  PBHelper.convertLocatedBlock miss convert BlockToken
  

• HDFS-4302. Major bug reported by Eugene Koontz and fixed by Eugene Koontz (ha , namenode)
  Precondition in EditLogFileInputStream's length() method is checked too early in NameNode startup, causing fatal exception
  

• HDFS-4295. Major bug reported by Stephen Chu and fixed by Stephen Chu (security)
  Using port 1023 should be valid when starting Secure DataNode
  

• HDFS-4294. Major bug reported by Robert Parker and fixed by Robert Parker
  Backwards compatibility is not maintained for TestVolumeId
  

• HDFS-4292. Minor bug reported by Binglin Chang and fixed by Binglin Chang
  Sanity check not correct in RemoteBlockReader2.newBlockReader
  

• HDFS-4291. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  edit log unit tests leave stray test_edit_log_file around
  

• HDFS-4288. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  NN accepts incremental BR as IBR in safemode
  

• HDFS-4282. Major bug reported by Junping Du and fixed by Todd Lipcon (namenode , test)
  TestEditLog.testFuzzSequences FAILED in all pre-commit test
  

• HDFS-4279. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (namenode)
  NameNode does not initialize generic conf keys when started with -recover
  

• HDFS-4274. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (datanode)
  BlockPoolSliceScanner does not close verification log during shutdown
  

• HDFS-4270. Minor bug reported by Derek Dagit and fixed by Derek Dagit (namenode)
  Replications of the highest priority should be allowed to choose a source datanode that has reached its max replication limit
  

• HDFS-4268. Major bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (namenode)
  Remove redundant enum NNHAStatusHeartbeat.State
  

• HDFS-4259. Minor improvement reported by Harsh J and fixed by Harsh J (hdfs-client)
  Improve pipeline DN replacement failure message
  

• HDFS-4247. Blocker sub-task reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  saveNamespace should be tolerant of dangling lease
  

• HDFS-4242. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Map.Entry is incorrectly used in LeaseManager
  

• HDFS-4238. Major bug reported by Vinay and fixed by Todd Lipcon (ha)
  [HA] Standby namenode should not do purging of shared storage edits.
  

• HDFS-4236. Blocker bug reported by Allen Wittenauer and fixed by Alejandro Abdelnur
  Regression: HDFS-4171 puts artificial limit on username length
  

• HDFS-4232. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  NN fails to write a fsimage with stale leases
  

• HDFS-4231. Major improvement reported by Konstantin Shvachko and fixed by Konstantin Shvachko (ha , namenode)
  Introduce HAState for BackupNode
  

• HDFS-4216. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Adding symlink should not ignore QuotaExceededException
  

• HDFS-4214. Trivial improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (tools)
  OfflineEditsViewer should print out the offset at which it encountered an error
  

• HDFS-4213. Major new feature reported by Jing Zhao and fixed by Jing Zhao (hdfs-client , namenode)
  When the client calls hsync, allows the client to update the file length in the NameNode
  

• HDFS-4199. Minor test reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
  Provide test for HdfsVolumeId
  

• HDFS-4186. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  logSync() is called with the write lock held while releasing lease
  

• HDFS-4182. Critical bug reported by Todd Lipcon and fixed by Robert Joseph Evans (namenode)
  SecondaryNameNode leaks NameCache entries
  

• HDFS-4181. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  LeaseManager tries to double remove and prints extra messages
  

• HDFS-4179. Major bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (namenode)
  BackupNode: allow reads, fix checkpointing, safeMode
  

• HDFS-4178. Major bug reported by Andy Isaacson and fixed by Andy Isaacson (scripts)
  shell scripts should not close stderr
  

• HDFS-4172. Minor bug reported by Derek Dagit and fixed by Derek Dagit (namenode)
  namenode does not URI-encode parameters when building URI for datanode request
  

• HDFS-4171. Major bug reported by Harsh J and fixed by Alejandro Abdelnur
  WebHDFS and HttpFs should accept only valid Unix user names
  

• HDFS-4164. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
  fuse_dfs: add -lrt to the compiler command line on Linux
  

• HDFS-4162. Minor bug reported by Derek Dagit and fixed by Derek Dagit (datanode)
  Some malformed and unquoted HTML strings are returned from datanode web ui
  

• HDFS-4156. Major bug reported by Eli Collins and fixed by Eli Reisman
  Seeking to a negative position should throw an IOE
  

• HDFS-4155. Major improvement reported by Liang Xie and fixed by Liang Xie (libhdfs)
  libhdfs implementation of hsync API
  

• HDFS-4153. Major improvement reported by Liang Xie and fixed by Liang Xie (journal-node)
  Add START_MSG/SHUTDOWN_MSG for JournalNode
  

• HDFS-4143. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Change INodeFile.blocks to private
  

• HDFS-4140. Major bug reported by Andy Isaacson and fixed by Colin Patrick McCabe (fuse-dfs)
  fuse-dfs handles open(O_TRUNC) poorly
  

• HDFS-4139. Major bug reported by Andy Isaacson and fixed by Colin Patrick McCabe (fuse-dfs)
  fuse-dfs RO mode still allows file truncation
  

• HDFS-4132. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  when libwebhdfs is not enabled, nativeMiniDfsClient frees uninitialized memory
  

• HDFS-4130. Major sub-task reported by Han Xiao and fixed by Han Xiao (ha , performance)
  BKJM: The reading for editlog at NN starting using bkjm is not efficient
  

• HDFS-4127. Minor bug reported by Junping Du and fixed by Junping Du (namenode)
  Log message is not correct in case of short of replica
  

• HDFS-4122. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas (datanode , hdfs-client , namenode)
  Cleanup HDFS logs and reduce the size of logged messages
  

  The change from this jira changes the content of some of the log messages. No log message are removed. Only the content of the log messages is changed to reduce the size. If you have a tool that depends on the exact content of the log, please look at the patch and make appropriate updates to the tool.

• HDFS-4121. Minor improvement reported by Binglin Chang and fixed by Binglin Chang
  Add namespace declarations in hdfs .proto files for languages other than java
  

• HDFS-4112. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  A few improvements on INodeDirectory
  

• HDFS-4110. Trivial improvement reported by Liang Xie and fixed by Liang Xie (journal-node)
  Refine JNStorage log
  

• HDFS-4107. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Add utility methods to cast INode to INodeFile and INodeFileUnderConstruction
  

• HDFS-4106. Minor bug reported by Jing Zhao and fixed by Jing Zhao (namenode , test)
  BPServiceActor#lastHeartbeat, lastBlockReport and lastDeletedReport should be declared as volatile
  

• HDFS-4105. Major bug reported by Arpit Gupta and fixed by Arpit Gupta
  the SPNEGO user for secondary namenode should use the web keytab
  

• HDFS-4104. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
  dfs -test -d prints inappropriate error on nonexistent directory
  

• HDFS-4100. Major sub-task reported by Liang Xie and fixed by Liang Xie (datanode , journal-node , security)
  Fix all findbug security warings
  

• HDFS-4099. Minor bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Clean up replication code and add more javadoc
  

• HDFS-4090. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (hdfs-client)
  getFileChecksum() result incompatible when called against zero-byte files.
  

• HDFS-4088. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Remove "throws QuotaExceededException" from an INodeDirectoryWithQuota constructor
  

• HDFS-4080. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  Add a separate logger for block state change logs to enable turning off those logs
  

• HDFS-4075. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  Reduce recommissioning overhead
  

• HDFS-4074. Trivial improvement reported by Brandon Li and fixed by Brandon Li (namenode)
  Remove empty constructors for INode
  

• HDFS-4073. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Jing Zhao (namenode)
  Two minor improvements to FSDirectory
  

• HDFS-4072. Minor bug reported by Jing Zhao and fixed by Jing Zhao (namenode)
  On file deletion remove corresponding blocks pending replication
  

• HDFS-4068. Minor improvement reported by Eli Collins and fixed by Eli Collins (datanode)
  DatanodeID and DatanodeInfo member should be private
  

• HDFS-4061. Major bug reported by Eli Collins and fixed by Eli Collins
  TestBalancer and TestUnderReplicatedBlocks need timeouts
  

• HDFS-4059. Minor sub-task reported by Jing Zhao and fixed by Jing Zhao (datanode , namenode)
  Add number of stale DataNodes to metrics
  

  This jira adds a new metric with name "StaleDataNodes" under metrics context "dfs" of type Gauge. This tracks the number of DataNodes marked as stale. A DataNode is marked stale when the heartbeat message from the DataNode is not received within the configured time ""dfs.namenode.stale.datanode.interval". Please see hdfs-default.xml documentation corresponding to ""dfs.namenode.stale.datanode.interval" for more details on how to configure this feature. When this feature is not configured, this metrics would return zero.

• HDFS-4058. Major improvement reported by Eli Collins and fixed by Eli Collins (datanode)
  DirectoryScanner may fail with IOOB if the directory scanning threads return out of volume order
  

• HDFS-4055. Major bug reported by Binglin Chang and fixed by Binglin Chang
  TestAuditLogs is flaky
  

• HDFS-4049. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (datanode , performance)
  hflush performance regression due to nagling delays
  

• HDFS-4048. Major improvement reported by Stephen Chu and fixed by Stephen Chu
  Use ERROR instead of INFO for volume failure logs
  

• HDFS-4046. Minor bug reported by Binglin Chang and fixed by Binglin Chang (datanode , namenode)
  ChecksumTypeProto use NULL as enum value which is illegal in C/C++
  

• HDFS-4044. Major bug reported by Binglin Chang and fixed by Binglin Chang (datanode)
  Duplicate ChecksumType definition in HDFS .proto files
  

• HDFS-4041. Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (build)
  Hadoop HDFS Maven protoc calls must not depend on external sh script
  

• HDFS-4038. Minor sub-task reported by Vinay and fixed by Vinay (ha)
  Override toString() for BookKeeperEditLogInputStream
  

• HDFS-4037. Major improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Rename the getReplication() method in BlockCollection to getBlockReplication()
  

• HDFS-4036. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Jing Zhao (namenode)
  FSDirectory.unprotectedAddFile(..) should not throw UnresolvedLinkException
  

• HDFS-4035. Major sub-task reported by Eli Collins and fixed by Eli Collins
  LightWeightGSet and LightWeightHashSet increment a volatile without synchronization
  

• HDFS-4034. Major sub-task reported by Eli Collins and fixed by Eli Collins
  Remove redundant null checks
  

• HDFS-4033. Major sub-task reported by Eli Collins and fixed by Eli Collins
  Miscellaneous findbugs 2 fixes
  

• HDFS-4032. Major sub-task reported by Eli Collins and fixed by Eli Collins
  Specify the charset explicitly rather than rely on the default
  

• HDFS-4031. Major sub-task reported by Eli Collins and fixed by Eli Collins (namenode)
  Update findbugsExcludeFile.xml to include findbugs 2 exclusions
  

• HDFS-4030. Major sub-task reported by Eli Collins and fixed by Eli Collins (namenode)
  BlockManager excessBlocksCount and postponedMisreplicatedBlocksCount should be AtomicLongs
  

• HDFS-4029. Major sub-task reported by Eli Collins and fixed by Eli Collins (namenode)
  GenerationStamp should use an AtomicLong
  

• HDFS-4022. Blocker bug reported by suja s and fixed by Vinay
  Replication not happening for appended block
  

• HDFS-4021. Minor bug reported by Colin Patrick McCabe and fixed by Christopher Conner (namenode)
  Misleading error message when resources are low on the NameNode
  

• HDFS-4020. Major bug reported by Eli Collins and fixed by Eli Collins
  TestRBWBlockInvalidation may time out
  

• HDFS-4018. Minor bug reported by Eli Collins and fixed by Eli Collins
  TestDataNodeMultipleRegistrations#testMiniDFSClusterWithMultipleNN is missing some cluster cleanup
  

• HDFS-4008. Minor improvement reported by Eli Collins and fixed by Eli Collins (test)
  TestBalancerWithEncryptedTransfer needs a timeout
  

• HDFS-4007. Minor test reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (test)
  Rehabilitate bit-rotted unit tests under hadoop-hdfs-project/hadoop-hdfs/src/test/unit/
  

• HDFS-4006. Major bug reported by Eli Collins and fixed by Todd Lipcon (namenode)
  TestCheckpoint#testSecondaryHasVeryOutOfDateImage occasionally fails due to unexpected exit
  

• HDFS-4000. Major improvement reported by Eli Collins and fixed by Colin Patrick McCabe
  TestParallelLocalRead fails with "input ByteBuffers must be direct buffers"
  

• HDFS-3999. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  HttpFS OPEN operation expects len parameter, it should be length
  

• HDFS-3997. Trivial bug reported by Mithun Radhakrishnan and fixed by Mithun Radhakrishnan (namenode)
  OfflineImageViewer incorrectly passes value of imageVersion when visiting IS_COMPRESSED element
  

• HDFS-3996. Minor bug reported by Eli Collins and fixed by Eli Collins
  Add debug log removed in HDFS-3873 back
  

• HDFS-3992. Minor bug reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
  Method org.apache.hadoop.hdfs.TestHftpFileSystem.tearDown() sometimes throws NPEs
  

• HDFS-3990. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  NN's health report has severe performance problems
  

• HDFS-3985. Major bug reported by Eli Collins and fixed by (test)
  Add timeouts to TestMulitipleNNDataBlockScanner
  

• HDFS-3979. Major bug reported by Lars Hofhansl and fixed by Lars Hofhansl (datanode)
  Fix hsync semantics
  

• HDFS-3970. Major bug reported by Vinay and fixed by Andrew Wang (datanode)
  BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
  

• HDFS-3964. Minor bug reported by Eli Collins and fixed by Eli Collins (namenode)
  Make NN log of fs.defaultFS debug rather than info
  

• HDFS-3957. Minor improvement reported by Andrew Wang and fixed by Andrew Wang
  Change MutableQuantiles to use a shared thread for rolling over metrics
  

• HDFS-3951. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (datanode)
  datanode web ui does not work over HTTPS when datanode is started in secure mode
  

• HDFS-3949. Minor bug reported by Eli Collins and fixed by Eli Collins (namenode)
  NameNodeRpcServer#join should join on both client and server RPC servers
  

• HDFS-3948. Minor bug reported by Eli Collins and fixed by Jing Zhao (test)
  TestWebHDFS#testNamenodeRestart occasionally fails
  

• HDFS-3944. Major task reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  Httpfs resolveAuthority() is not resolving host correctly
  

• HDFS-3939. Minor improvement reported by Eli Collins and fixed by Eli Collins (namenode)
  NN RPC address cleanup
  

• HDFS-3938. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (documentation)
  remove current limitations from HttpFS docs
  

• HDFS-3936. Major bug reported by Eli Collins and fixed by Eli Collins
  MiniDFSCluster shutdown races with BlocksMap usage
  

• HDFS-3935. Major sub-task reported by Eli Collins and fixed by Andy Isaacson
  QJM: Add JournalNode to the start / stop scripts
  

• HDFS-3932. Major bug reported by Eli Collins and fixed by Eli Collins
  NameNode Web UI broken if the rpc-address is set to the wildcard
  

• HDFS-3931. Minor bug reported by Eli Collins and fixed by Andy Isaacson (test)
  TestDatanodeBlockScanner#testBlockCorruptionPolicy2 is broken
  

• HDFS-3925. Minor improvement reported by Andrew Wang and fixed by Andrew Wang
  Prettify PipelineAck#toString() for printing to a log
  

• HDFS-3924. Major bug reported by Andrew Wang and fixed by Andrew Wang (hdfs-client)
  Multi-byte id in HdfsVolumeId
  

• HDFS-3923. Major sub-task reported by Jing Zhao and fixed by Jing Zhao
  libwebhdfs testing code cleanup
  

• HDFS-3921. Major bug reported by Stephen Chu and fixed by Aaron T. Myers
  NN will prematurely consider blocks missing when entering active state while still in safe mode
  

• HDFS-3920. Major sub-task reported by Jing Zhao and fixed by Jing Zhao
  libwebdhfs code cleanup: string processing and using strerror consistently to handle all errors
  

• HDFS-3919. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson (test)
  MiniDFSCluster:waitClusterUp can hang forever
  

• HDFS-3916. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (webhdfs)
  libwebhdfs (C client) code cleanups
  

• HDFS-3912. Major sub-task reported by Jing Zhao and fixed by Jing Zhao
  Detecting and avoiding stale datanodes for writing
  

• HDFS-3910. Minor improvement reported by Eli Collins and fixed by Eli Collins (test)
  DFSTestUtil#waitReplication should timeout
  

• HDFS-3896. Minor improvement reported by Jeff Lord and fixed by Jeff Lord
  Add descriptions for dfs.namenode.rpc-address and dfs.namenode.servicerpc-address to hdfs-default.xml
  

• HDFS-3831. Critical bug reported by Jason Lowe and fixed by Jason Lowe (security)
  Failure to renew tokens due to test-sources left in classpath
  

• HDFS-3829. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestHftpURLTimeouts fails intermittently with JDK7
  

• HDFS-3824. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestHftpDelegationToken fails intermittently with JDK7
  

• HDFS-3813. Major improvement reported by Stephen Chu and fixed by Stephen Chu (security , webhdfs)
  Log error message if security and WebHDFS are enabled but principal/keytab are not configured
  

• HDFS-3810. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly
  Implement format() for BKJM
  

• HDFS-3809. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly (namenode)
  Make BKJM use protobufs for all serialization with ZK
  

• HDFS-3804. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestHftpFileSystem fails intermittently with JDK7
  

• HDFS-3789. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly (ha , namenode)
  JournalManager#format() should be able to throw IOException
  

• HDFS-3753. Major bug reported by Eli Collins and fixed by Colin Patrick McCabe (build , test)
  Tests don't run with native libraries
  

• HDFS-3703. Major improvement reported by nkeywal and fixed by Jing Zhao (datanode , namenode)
  Decrease the datanode failure detection time
  

  This jira adds a new DataNode state called "stale" at the NameNode. DataNodes are marked as stale if it does not send heartbeat message to NameNode within the timeout configured using the configuration parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds). NameNode picks a stale datanode as the last target to read from when returning block locations for reads. This feature is by default turned * off *. To turn on the feature, set the HDFS configuration "dfs.namenode.check.stale.datanode" to true.

• HDFS-3695. Major sub-task reported by Todd Lipcon and fixed by Todd Lipcon (ha , namenode)
  Genericize format() to non-file JournalManagers
  

• HDFS-3682. Minor improvement reported by Eli Collins and fixed by Todd Lipcon (test)
  MiniDFSCluster#init should provide more info when it fails
  

• HDFS-3680. Minor improvement reported by Marcelo Vanzin and fixed by Marcelo Vanzin (namenode)
  Allow customized audit logging in HDFS FSNamesystem
  

• HDFS-3678. Critical bug reported by Todd Lipcon and fixed by Aaron T. Myers (namenode)
  Edit log files are never being purged from 2NN
  

• HDFS-3626. Blocker bug reported by Todd Lipcon and fixed by Todd Lipcon (namenode)
  Creating file with invalid path can corrupt edit log
  

• HDFS-3623. Major sub-task reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (namenode)
  BKJM: zkLatchWaitTimeout hard coded to 6000. Make use of ZKSessionTimeout instead.
  

• HDFS-3616. Major bug reported by Uma Maheswara Rao G and fixed by Jing Zhao (datanode)
  TestWebHdfsWithMultipleNameNodes fails with ConcurrentModificationException in DN shutdown
  

• HDFS-3598. Major new feature reported by Tsz Wo (Nicholas), SZE and fixed by Plamen Jeliazkov (webhdfs)
  WebHDFS: support file concat
  

• HDFS-3573. Minor sub-task reported by Todd Lipcon and fixed by Todd Lipcon (namenode)
  Supply NamespaceInfo when instantiating JournalManagers
  

• HDFS-3571. Major sub-task reported by Todd Lipcon and fixed by Todd Lipcon (ha , namenode)
  Allow EditLogFileInputStream to read from a remote URL
  

• HDFS-3553. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp
  Hftp proxy tokens are broken
  

• HDFS-3510. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Improve FSEditLog pre-allocation
  

• HDFS-3507. Critical bug reported by Vinay and fixed by Vinay (ha)
  DFS#isInSafeMode needs to execute only on Active NameNode
  

• HDFS-3483. Major improvement reported by Stephen Chu and fixed by Stephen Fritz
  Better error message when hdfs fsck is run against a ViewFS config
  

• HDFS-3429. Major bug reported by Todd Lipcon and fixed by Todd Lipcon (datanode , performance)
  DataNode reads checksums even if client does not need them
  

• HDFS-3373. Major bug reported by Todd Lipcon and fixed by John George (hdfs-client)
  FileContext HDFS implementation can leak socket caches
  

• HDFS-3224. Minor bug reported by Eli Collins and fixed by Jason Lowe
  Bug in check for DN re-registration with different storage ID
  

• HDFS-3077. Major new feature reported by Todd Lipcon and fixed by Todd Lipcon (ha , namenode)
  Quorum-based protocol for reading and writing edit logs
  

• HDFS-3049. Minor new feature reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (namenode)
  During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt
  

• HDFS-2946. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , namenode)
  HA: Put a cap on the number of completed edits files retained by the NN
  

• HDFS-2908. Minor sub-task reported by Suresh Srinivas and fixed by Brandon Li
  Add apache license header for StorageReport.java
  

• HDFS-2656. Major improvement reported by Zhanwei.Wang and fixed by Jing Zhao (webhdfs)
  Implement a pure c client based on webhdfs
  

• HDFS-2264. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (namenode)
  NamenodeProtocol has the wrong value for clientPrincipal in KerberosInfo annotation
  

• HDFS-1331. Minor bug reported by Allen Wittenauer and fixed by Andy Isaacson (tools)
  dfs -test should work like /bin/test
  

  "test" will not print a warning for non-existent paths when testing for existence

• HDFS-1322. Major bug reported by Ravi Gummadi and fixed by Colin Patrick McCabe
  Document umask in DistributedFileSystem#mkdirs javadocs
  

• HDFS-1245. Major new feature reported by Dmytro Molkov and fixed by Konstantin Shvachko (namenode)
  Plugable block id generation