Hadoop HDFS 2.1.0-beta Release Notes
These release notes include new developer and user-facing incompatibilities, features, and major improvements.
Changes since Hadoop 2.0.5-alpha
- HDFS-5083.
Blocker bug reported by Kihwal Lee and fixed by Kihwal Lee
Update the HDFS compatibility version range
- HDFS-5027.
Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (datanode)
On startup, DN should scan volumes in parallel
- HDFS-5025.
Major sub-task reported by Jing Zhao and fixed by Jing Zhao (ha , namenode)
Record ClientId and CallId in EditLog to enable rebuilding retry cache in case of HA failover
- HDFS-5024.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
Make DatanodeProtocol#commitBlockSynchronization idempotent
- HDFS-5020.
Major improvement reported by Jing Zhao and fixed by Jing Zhao (namenode)
Make DatanodeProtocol#blockReceivedAndDeleted idempotent
- HDFS-5018.
Minor bug reported by Ted Yu and fixed by Ted Yu
Misspelled DFSConfigKeys#DFS_NAMENODE_STALE_DATANODE_INTERVAL_DEFAULT in javadoc of DatanodeInfo#isStale()
- HDFS-5016.
Blocker bug reported by Devaraj Das and fixed by Suresh Srinivas
Deadlock in pipeline recovery causes Datanode to be marked dead
- HDFS-5010.
Major improvement reported by Kihwal Lee and fixed by Kihwal Lee (namenode , performance)
Reduce the frequency of getCurrentUser() calls from namenode
- HDFS-5008.
Major improvement reported by Suresh Srinivas and fixed by Jing Zhao (namenode)
Make ClientProtocol#abandonBlock() idempotent
- HDFS-5007.
Minor improvement reported by Kousuke Saruta and fixed by Kousuke Saruta
Replace hard-coded property keys with DFSConfigKeys fields
- HDFS-5005.
Major bug reported by Jing Zhao and fixed by Jing Zhao
Move SnapshotException and SnapshotAccessControlException to o.a.h.hdfs.protocol
- HDFS-5003.
Minor bug reported by Xi Fang and fixed by Xi Fang (test)
TestNNThroughputBenchmark failed caused by existing directories
- HDFS-4999.
Major bug reported by Kihwal Lee and fixed by Colin Patrick McCabe
fix TestShortCircuitLocalRead on branch-2
- HDFS-4998.
Major bug reported by Kihwal Lee and fixed by Kihwal Lee (test)
TestUnderReplicatedBlocks fails intermittently
- HDFS-4996.
Minor improvement reported by Chris Nauroth and fixed by Chris Nauroth (namenode)
ClientProtocol#metaSave can be made idempotent by overwriting the output file instead of appending to it
The dfsadmin -metasave command has been changed to overwrite the output file. Previously, this command would append to the output file if it already existed.
- HDFS-4992.
Major improvement reported by Max Lapan and fixed by Max Lapan (balancer)
Make balancer's thread count configurable
- HDFS-4982.
Major bug reported by Todd Lipcon and fixed by Todd Lipcon (journal-node , security)
JournalNode should relogin from keytab before fetching logs from other JNs
- HDFS-4980.
Major bug reported by Mark Grover and fixed by Mark Grover (build)
Incorrect logging.properties file for hadoop-httpfs
- HDFS-4979.
Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
Implement retry cache on the namenode
- HDFS-4978.
Major improvement reported by Jing Zhao and fixed by Jing Zhao
Make disallowSnapshot idempotent
- HDFS-4974.
Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (ha , namenode)
Analyze and add annotations to Namenode protocol methods and enable retry
- HDFS-4969.
Blocker bug reported by Robert Kanter and fixed by Robert Kanter (test , webhdfs)
WebhdfsFileSystem expects non-standard WEBHDFS Json element
- HDFS-4954.
Major bug reported by Brandon Li and fixed by Brandon Li (nfs)
compile failure in branch-2: getFlushedOffset should catch or rethrow IOException
- HDFS-4951.
Major bug reported by Robert Kanter and fixed by Robert Kanter (security)
FsShell commands using secure httpfs throw exceptions due to missing TokenRenewer
- HDFS-4948.
Major bug reported by Robert Joseph Evans and fixed by Brandon Li
mvn site for hadoop-hdfs-nfs fails
- HDFS-4944.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (webhdfs)
WebHDFS cannot create a file path containing characters that must be URI-encoded, such as space.
- HDFS-4943.
Minor bug reported by Jerry He and fixed by Jerry He (webhdfs)
WebHdfsFileSystem does not work when original file path has encoded chars
- HDFS-4932.
Minor improvement reported by Fengdong Yu and fixed by Fengdong Yu (ha , namenode)
Avoid a wide line on the name node webUI if we have more Journal nodes
- HDFS-4927.
Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
CreateEditsLog creates inodes with an invalid inode ID, which then cannot be loaded by a namenode.
- HDFS-4917.
Major bug reported by Fengdong Yu and fixed by Fengdong Yu (datanode , namenode)
Start-dfs.sh cannot pass the parameters correctly
- HDFS-4914.
Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (hdfs-client)
When possible, Use DFSClient.Conf instead of Configuration
- HDFS-4912.
Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
Cleanup FSNamesystem#startFileInternal
- HDFS-4910.
Major bug reported by Chuan Liu and fixed by Chuan Liu
TestPermission failed in branch-2
- HDFS-4908.
Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode , snapshots)
Reduce snapshot inode memory usage
- HDFS-4906.
Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs-client)
HDFS Output streams should not accept writes after being closed
- HDFS-4903.
Minor improvement reported by Suresh Srinivas and fixed by Arpit Agarwal (namenode)
Print trash configuration and trash emptier state in namenode log
- HDFS-4902.
Major bug reported by Binglin Chang and fixed by Binglin Chang (snapshots)
DFSClient.getSnapshotDiffReport should use string path rather than o.a.h.fs.Path
- HDFS-4888.
Major bug reported by Ravi Prakash and fixed by Ravi Prakash
Refactor and fix FSNamesystem.getTurnOffTip to sanity
- HDFS-4887.
Blocker bug reported by Kihwal Lee and fixed by Kihwal Lee (benchmarks , test)
TestNNThroughputBenchmark exits abruptly
- HDFS-4883.
Major bug reported by Konstantin Shvachko and fixed by Tao Luo (namenode)
complete() should verify fileId
- HDFS-4880.
Major bug reported by Arpit Agarwal and fixed by Suresh Srinivas (namenode)
Diagnostic logging while loading name/edits files
- HDFS-4878.
Major bug reported by Tao Luo and fixed by Tao Luo (namenode)
On Remove Block, Block is not Removed from neededReplications queue
- HDFS-4877.
Blocker bug reported by Jing Zhao and fixed by Jing Zhao (snapshots)
Snapshot: fix the scenario where a directory is renamed under its prior descendant
- HDFS-4876.
Minor sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (snapshots)
The javadoc of FileWithSnapshot is incorrect
- HDFS-4875.
Minor sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Arpit Agarwal (snapshots , test)
Add a test for testing snapshot file length
- HDFS-4873.
Major bug reported by Hari Mankude and fixed by Jing Zhao (snapshots)
callGetBlockLocations returns incorrect number of blocks for snapshotted files
- HDFS-4867.
Major bug reported by Kihwal Lee and fixed by Plamen Jeliazkov (namenode)
metaSave NPEs when there are invalid blocks in repl queue.
- HDFS-4866.
Blocker bug reported by Ralph Castain and fixed by Arpit Agarwal (namenode)
Protocol buffer support cannot compile under C
The Protocol Buffers definition of the inter-namenode protocol required a change for compatibility with compiled C clients. This is a backwards-incompatible change. A namenode prior to this change will not be able to communicate with a namenode after this change.
- HDFS-4865.
Major bug reported by Wei Yan and fixed by Wei Yan
Remove sub resource warning from httpfs log at startup time
- HDFS-4863.
Major bug reported by Jing Zhao and fixed by Jing Zhao (snapshots)
The root directory should be added to the snapshottable directory list while loading fsimage
- HDFS-4862.
Major bug reported by Ravi Prakash and fixed by Ravi Prakash
SafeModeInfo.isManual() returns true when resources are low even if it wasn't entered into manually
- HDFS-4857.
Major bug reported by Jing Zhao and fixed by Jing Zhao (snapshots)
Snapshot.Root and AbstractINodeDiff#snapshotINode should not be put into INodeMap when loading FSImage
- HDFS-4850.
Major bug reported by Stephen Chu and fixed by Jing Zhao (tools)
fix OfflineImageViewer to work on fsimages with empty files or snapshots
- HDFS-4848.
Minor improvement reported by Stephen Chu and fixed by Jing Zhao (snapshots)
copyFromLocal and renaming a file to ".snapshot" should output that ".snapshot" is a reserved name
- HDFS-4846.
Minor bug reported by Stephen Chu and fixed by Jing Zhao (snapshots)
Clean up snapshot CLI commands output stacktrace for invalid arguments
- HDFS-4845.
Critical bug reported by Kihwal Lee and fixed by Arpit Agarwal (namenode)
FSEditLogLoader gets NPE while accessing INodeMap in TestEditLogRace
- HDFS-4842.
Major sub-task reported by Jing Zhao and fixed by Jing Zhao (snapshots)
Snapshot: identify the correct prior snapshot when deleting a snapshot under a renamed subtree
- HDFS-4841.
Major bug reported by Stephen Chu and fixed by Robert Kanter (security , webhdfs)
FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
- HDFS-4840.
Major bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
ReplicationMonitor gets NPE during shutdown
- HDFS-4832.
Critical bug reported by Ravi Prakash and fixed by Ravi Prakash
Namenode doesn't change the number of missing blocks in safemode when DNs rejoin or leave
This change makes name node keep its internal replication queues and data node state updated in manual safe mode. This allows metrics and UI to present up-to-date information while in safe mode. The behavior during start-up safe mode is unchanged.
- HDFS-4830.
Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers
Typo in config settings for AvailableSpaceVolumeChoosingPolicy in hdfs-default.xml
- HDFS-4827.
Major bug reported by Devaraj Das and fixed by Devaraj Das
Slight update to the implementation of API for handling favored nodes in DFSClient
- HDFS-4826.
Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
TestNestedSnapshots times out due to repeated slow edit log flushes when running on virtualized disk
- HDFS-4825.
Major bug reported by Andrew Wang and fixed by Andrew Wang (webhdfs)
webhdfs / httpfs tests broken because of min block size change
- HDFS-4824.
Major bug reported by Henry Robinson and fixed by Colin Patrick McCabe (hdfs-client)
FileInputStreamCache.close leaves dangling reference to FileInputStreamCache.cacheCleaner
- HDFS-4819.
Minor sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (documentation)
Update Snapshot doc for HDFS-4758
- HDFS-4818.
Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (namenode , test)
several HDFS tests that attempt to make directories unusable do not work correctly on Windows
- HDFS-4815.
Major bug reported by Tian Hong Wang and fixed by Tian Hong Wang (datanode , test)
TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN: Double call countReplicas() to fetch corruptReplicas and liveReplicas is not needed
- HDFS-4813.
Minor bug reported by Tsz Wo (Nicholas), SZE and fixed by Jing Zhao (namenode)
BlocksMap may throw NullPointerException during shutdown
- HDFS-4810.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
several HDFS HA tests have timeouts that are too short
- HDFS-4807.
Major bug reported by Kihwal Lee and fixed by Cristina L. Abad
DFSOutputStream.createSocketForPipeline() should not include timeout extension on connect
- HDFS-4805.
Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (webhdfs)
Webhdfs client is fragile to token renewal errors
- HDFS-4804.
Minor improvement reported by Stephen Chu and fixed by Stephen Chu
WARN when users set the block balanced preference percent below 0.5 or above 1.0
- HDFS-4799.
Blocker bug reported by Todd Lipcon and fixed by Todd Lipcon (namenode)
Corrupt replica can be prematurely removed from corruptReplicas map
- HDFS-4797.
Minor bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (datanode)
BlockScanInfo does not override equals(..) and hashCode() consistently
- HDFS-4787.
Major improvement reported by Tian Hong Wang and fixed by Tian Hong Wang
Create a new HdfsConfiguration before each TestDFSClientRetries testcases
- HDFS-4785.
Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
Concat operation does not remove concatenated files from InodeMap
- HDFS-4784.
Major sub-task reported by Brandon Li and fixed by Brandon Li (namenode)
NPE in FSDirectory.resolvePath()
- HDFS-4783.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
TestDelegationTokensWithHA#testHAUtilClonesDelegationTokens fails on Windows
- HDFS-4780.
Minor bug reported by Kihwal Lee and fixed by Robert Parker (namenode)
Use the correct relogin method for services
- HDFS-4778.
Major bug reported by Devaraj Das and fixed by Devaraj Das (namenode)
Invoke getPipeline in the chooseTarget implementation that has favoredNodes
- HDFS-4772.
Minor improvement reported by Brandon Li and fixed by Brandon Li (namenode)
Add number of children in HdfsFileStatus
- HDFS-4768.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (datanode)
File handle leak in datanode when a block pool is removed
- HDFS-4765.
Major bug reported by Andrew Wang and fixed by Andrew Wang (namenode)
Permission check of symlink deletion incorrectly throws UnresolvedLinkException
- HDFS-4763.
Major sub-task reported by Brandon Li and fixed by Brandon Li (nfs)
Add script changes/utility for starting NFS gateway
- HDFS-4762.
Major sub-task reported by Brandon Li and fixed by Brandon Li (nfs)
Provide HDFS based NFSv3 and Mountd implementation
- HDFS-4751.
Minor bug reported by Andrew Wang and fixed by Andrew Wang (test)
TestLeaseRenewer#testThreadName flakes
- HDFS-4748.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (qjm , test)
MiniJournalCluster#restartJournalNode leaks resources, which causes sporadic test failures
- HDFS-4745.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
TestDataTransferKeepalive#testSlowReader has race condition that causes sporadic failure
- HDFS-4743.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
TestNNStorageRetentionManager fails on Windows
- HDFS-4741.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
TestStorageRestore#testStorageRestoreFailure fails on Windows
- HDFS-4740.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
Fixes for a few test failures on Windows
- HDFS-4739.
Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (namenode)
NN can miscalculate the number of extra edit log segments to retain
- HDFS-4737.
Major bug reported by Sean Mackrory and fixed by Sean Mackrory
JVM path embedded in fuse binaries
- HDFS-4734.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal
HDFS Tests that use ShellCommandFencer are broken on Windows
- HDFS-4733.
Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
Make HttpFS username pattern configurable
- HDFS-4732.
Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
TestDFSUpgradeFromImage fails on Windows due to failure to unpack old image tarball that contains hard links
- HDFS-4725.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (namenode , test , tools)
fix HDFS file handle leaks
- HDFS-4722.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic (test)
TestGetConf#testFederation times out on Windows
- HDFS-4721.
Major improvement reported by Varun Sharma and fixed by Varun Sharma (namenode)
Speed up lease/block recovery when DN fails and a block goes into recovery
- HDFS-4714.
Major bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
Log short messages in Namenode RPC server for exceptions meant for clients
- HDFS-4705.
Minor bug reported by Ivan Mitic and fixed by Ivan Mitic
Address HDFS test failures on Windows because of invalid dfs.namenode.name.dir
- HDFS-4699.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
- HDFS-4698.
Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
provide client-side metrics for remote reads, local reads, and short-circuit reads
- HDFS-4695.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic (test)
TestEditLog leaks open file handles between tests
- HDFS-4693.
Minor bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
Some test cases in TestCheckpoint do not clean up after themselves
- HDFS-4687.
Minor bug reported by Andrew Wang and fixed by Andrew Wang (test)
TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with JDK7
- HDFS-4679.
Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
Namenode operation checks should be done in a consistent manner
- HDFS-4677.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic
Editlog should support synchronous writes
- HDFS-4676.
Minor bug reported by Suresh Srinivas and fixed by Suresh Srinivas (test)
TestHDFSFileSystemContract should set MiniDFSCluster variable to null to free up memory
- HDFS-4674.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
TestBPOfferService fails on Windows due to failure parsing datanode data directory as URI
- HDFS-4669.
Major bug reported by Tian Hong Wang and fixed by Tian Hong Wang (test)
TestBlockPoolManager fails using IBM java
- HDFS-4659.
Major bug reported by Brandon Li and fixed by Brandon Li (namenode)
Support setting execution bit for regular files
- HDFS-4658.
Trivial bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , namenode)
Standby NN will log that it has received a block report "after becoming active"
- HDFS-4655.
Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (datanode)
DNA_FINALIZE is logged as being an unknown command by the DN when received from the standby NN
- HDFS-4646.
Minor bug reported by Jagane Sundar and fixed by (namenode)
createNNProxyWithClientProtocol ignores configured timeout value
- HDFS-4645.
Major improvement reported by Suresh Srinivas and fixed by Arpit Agarwal (namenode)
Move from randomly generated block ID to sequentially generated block ID
- HDFS-4643.
Trivial bug reported by Todd Lipcon and fixed by Todd Lipcon (qjm , test)
Fix flakiness in TestQuorumJournalManager
- HDFS-4639.
Major bug reported by Konstantin Shvachko and fixed by Plamen Jeliazkov (namenode)
startFileInternal() should not increment generation stamp
- HDFS-4635.
Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
Move BlockManager#computeCapacity to LightWeightGSet
- HDFS-4625.
Minor bug reported by Arpit Agarwal and fixed by Ivan Mitic (test)
Make TestNNWithQJM#testNewNamenodeTakesOverWriter work on Windows
- HDFS-4621.
Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (ha , qjm)
additional logging to help diagnose slow QJM logSync
- HDFS-4620.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza (documentation)
Documentation for dfs.namenode.rpc-address specifies wrong format
- HDFS-4618.
Major bug reported by Todd Lipcon and fixed by Todd Lipcon (namenode)
default for checkpoint txn interval is too low
- HDFS-4615.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
Fix TestDFSShell failures on Windows
- HDFS-4614.
Trivial bug reported by Aaron T. Myers and fixed by Aaron T. Myers (namenode)
FSNamesystem#getContentSummary should use getPermissionChecker helper method
- HDFS-4610.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic
Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
- HDFS-4609.
Minor bug reported by Ivan Mitic and fixed by Ivan Mitic (test)
TestAuditLogs should release log handles between tests
- HDFS-4607.
Minor bug reported by Ivan Mitic and fixed by Ivan Mitic (test)
TestGetConf#testGetSpecificKey fails on Windows
- HDFS-4604.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic
TestJournalNode fails on Windows
- HDFS-4603.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic
TestMiniDFSCluster fails on Windows
- HDFS-4602.
Major sub-task reported by Suresh Srinivas and fixed by Uma Maheswara Rao G
TestBookKeeperHACheckpoints fails
- HDFS-4598.
Minor bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (webhdfs)
WebHDFS concat: the default value of sources in the code does not match the doc
- HDFS-4596.
Major bug reported by Andrew Wang and fixed by Andrew Wang (namenode)
Shutting down namenode during checkpointing can lead to md5sum error
- HDFS-4595.
Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas (hdfs-client)
When short circuit read is fails, DFSClient does not fallback to regular reads
- HDFS-4593.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal
TestSaveNamespace fails on Windows
- HDFS-4592.
Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (namenode)
Default values for access time precision are out of sync between hdfs-default.xml and the code
- HDFS-4591.
Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , namenode)
HA clients can fail to fail over while Standby NN is performing long checkpoint
- HDFS-4586.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic
TestDataDirs.testGetDataDirsFromURIs fails with all directories in dfs.datanode.data.dir are invalid
- HDFS-4583.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic
TestNodeCount fails
- HDFS-4582.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic
TestHostsFiles fails on Windows
- HDFS-4573.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
Fix TestINodeFile on Windows
- HDFS-4572.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode , test)
Fix TestJournal failures on Windows
- HDFS-4569.
Trivial improvement reported by Andrew Wang and fixed by Andrew Wang
Small image transfer related cleanups.
- HDFS-4565.
Minor improvement reported by Arpit Gupta and fixed by Arpit Gupta (security)
use DFSUtil.getSpnegoKeytabKey() to get the spnego keytab key in secondary namenode and namenode http server
- HDFS-4544.
Major bug reported by Amareshwari Sriramadasu and fixed by Arpit Agarwal
Error in deleting blocks should not do check disk, for all types of errors
- HDFS-4542.
Blocker sub-task reported by Daryn Sharp and fixed by Daryn Sharp (webhdfs)
Webhdfs doesn't support secure proxy users
- HDFS-4541.
Major bug reported by Arpit Gupta and fixed by Arpit Gupta (datanode , security)
set hadoop.log.dir and hadoop.id.str when starting secure datanode so it writes the logs to the correct dir by default
- HDFS-4540.
Major bug reported by Arpit Gupta and fixed by Arpit Gupta (security)
namenode http server should use the web authentication keytab for spnego principal
- HDFS-4533.
Major bug reported by Fengdong Yu and fixed by Fengdong Yu (datanode , namenode)
start-dfs.sh ignored additional parameters besides -upgrade
- HDFS-4532.
Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
RPC call queue may fill due to current user lookup
- HDFS-4525.
Major sub-task reported by Uma Maheswara Rao G and fixed by SreeHari (namenode)
Provide an API for knowing that whether file is closed or not.
- HDFS-4522.
Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
LightWeightGSet expects incrementing a volatile to be atomic
- HDFS-4521.
Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
invalid network topologies should not be cached
- HDFS-4519.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (datanode , scripts)
Support override of jsvc binary and log file locations when launching secure datanode.
With this improvement the following options are available in release 1.2.0 and later on 1.x release stream:
1. jsvc location can be overridden by setting environment variable JSVC_HOME. Defaults to jsvc binary packaged within the Hadoop distro.
2. jsvc log output is directed to the file defined by JSVC_OUTFILE. Defaults to $HADOOP_LOG_DIR/jsvc.out.
3. jsvc error output is directed to the file defined by JSVC_ERRFILE file. Defaults to $HADOOP_LOG_DIR/jsvc.err.
With this improvement the following options are available in release 2.0.4 and later on 2.x release stream:
1. jsvc log output is directed to the file defined by JSVC_OUTFILE. Defaults to $HADOOP_LOG_DIR/jsvc.out.
2. jsvc error output is directed to the file defined by JSVC_ERRFILE file. Defaults to $HADOOP_LOG_DIR/jsvc.err.
For overriding jsvc location on 2.x releases, here is the release notes from HDFS-2303:
To run secure Datanodes users must install jsvc for their platform and set JSVC_HOME to point to the location of jsvc in their environment.
- HDFS-4518.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal
Finer grained metrics for HDFS capacity
- HDFS-4502.
Blocker sub-task reported by Alejandro Abdelnur and fixed by Brandon Li (webhdfs)
WebHdfsFileSystem handling of fileld breaks compatibility
- HDFS-4495.
Major bug reported by Kihwal Lee and fixed by Kihwal Lee (hdfs-client)
Allow client-side lease renewal to be retried beyond soft-limit
- HDFS-4484.
Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
libwebhdfs compilation broken with gcc 4.6.2
- HDFS-4477.
Critical bug reported by Kihwal Lee and fixed by Daryn Sharp (security)
Secondary namenode may retain old tokens
- HDFS-4471.
Major bug reported by Andrew Wang and fixed by Andrew Wang (namenode)
Namenode WebUI file browsing does not work with wildcard addresses configured
- HDFS-4470.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth
several HDFS tests attempt file operations on invalid HDFS paths when running on Windows
- HDFS-4465.
Major improvement reported by Suresh Srinivas and fixed by Aaron T. Myers (datanode)
Optimize datanode ReplicasMap and ReplicaInfo
- HDFS-4461.
Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
DirectoryScanner: volume path prefix takes up memory for every block that is scanned
- HDFS-4434.
Major sub-task reported by Brandon Li and fixed by Suresh Srinivas (namenode)
Provide a mapping from INodeId to INode
This change adds support for referencing files and directories based on fileID/inodeID using a path /.reserved/.inodes/<inodeid>. With this change creating a file or directory /.reserved is not longer allowed. Before upgrading to a release with this change, files /.reserved needs to be renamed to another name.
- HDFS-4382.
Major bug reported by Ted Yu and fixed by Ted Yu
Fix typo MAX_NOT_CHANGED_INTERATIONS
- HDFS-4374.
Major sub-task reported by Chris Nauroth and fixed by Chris Nauroth (namenode)
Display NameNode startup progress in UI
- HDFS-4373.
Major sub-task reported by Chris Nauroth and fixed by Chris Nauroth (namenode)
Add HTTP API for querying NameNode startup progress
- HDFS-4372.
Major sub-task reported by Chris Nauroth and fixed by Chris Nauroth (namenode)
Track NameNode startup progress
- HDFS-4346.
Minor sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
Refactor INodeId and GenerationStamp
- HDFS-4342.
Major bug reported by Mark Yang and fixed by Arpit Agarwal (namenode)
Edits dir in dfs.namenode.edits.dir.required will be silently ignored if it is not in dfs.namenode.edits.dir
- HDFS-4340.
Major sub-task reported by Brandon Li and fixed by Brandon Li (hdfs-client , namenode)
Update addBlock() to inculde inode id as additional argument
- HDFS-4339.
Major sub-task reported by Brandon Li and fixed by Brandon Li (namenode)
Persist inode id in fsimage and editlog
- HDFS-4334.
Major sub-task reported by Brandon Li and fixed by Brandon Li (namenode)
Add a unique id to each INode
- HDFS-4305.
Minor bug reported by Todd Lipcon and fixed by Andrew Wang (namenode)
Add a configurable limit on number of blocks per file, and min block size
This change introduces a maximum number of blocks per file, by default one million, and a minimum block size, by default 1MB. These can optionally be changed via the configuration settings "dfs.namenode.fs-limits.max-blocks-per-file" and "dfs.namenode.fs-limits.min-block-size", respectively.
- HDFS-4304.
Major improvement reported by Todd Lipcon and fixed by Colin Patrick McCabe (namenode)
Make FSEditLogOp.MAX_OP_SIZE configurable
- HDFS-4300.
Critical bug reported by Todd Lipcon and fixed by Andrew Wang
TransferFsImage.downloadEditsToStorage should use a tmp file for destination
- HDFS-4298.
Major bug reported by Todd Lipcon and fixed by Aaron T. Myers (namenode)
StorageRetentionManager spews warnings when used with QJM
- HDFS-4296.
Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
Add layout version for HDFS-4256 for release 1.2.0
- HDFS-4287.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (webhdfs)
HTTPFS tests fail on Windows
- HDFS-4261.
Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Junping Du (balancer)
TestBalancerWithNodeGroup times out
- HDFS-4249.
Major new feature reported by Suresh Srinivas and fixed by Chris Nauroth (namenode)
Add status NameNode startup to webUI
- HDFS-4246.
Minor improvement reported by Harsh J and fixed by Harsh J (hdfs-client)
The exclude node list should be more forgiving, for each output stream
- HDFS-4240.
Major bug reported by Junping Du and fixed by Junping Du (namenode)
In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
- HDFS-4235.
Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
when outputting XML, OfflineEditsViewer can't handle some edits containing non-ASCII strings
- HDFS-4234.
Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (balancer)
Use the generic code for choosing datanode in Balancer
- HDFS-4222.
Minor bug reported by Xiaobo Peng and fixed by Xiaobo Peng (namenode)
NN is unresponsive and loses heartbeats of DNs when Hadoop is configured to use LDAP and LDAP has issues
- HDFS-4215.
Major improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
Improvements on INode and image loading
- HDFS-4209.
Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
Clean up the addNode/addChild/addChildNoQuotaCheck methods in FSDirectory
- HDFS-4206.
Major improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
Change the fields in INode and its subclasses to private
- HDFS-4205.
Major bug reported by Andy Isaacson and fixed by Jason Lowe (hdfs-client)
fsck fails with symlinks
- HDFS-4152.
Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Jing Zhao (namenode)
Add a new class for the parameter in INode.collectSubtreeBlocksAndClear(..)
- HDFS-4151.
Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
Passing INodesInPath instead of INode[] in FSDirectory
- HDFS-4129.
Minor test reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
Add utility methods to dump NameNode in memory tree for testing
- HDFS-4128.
Major bug reported by Todd Lipcon and fixed by Kihwal Lee (namenode)
2NN gets stuck in inconsistent state if edit log replay fails in the middle
- HDFS-4124.
Minor new feature reported by Jing Zhao and fixed by Jing Zhao
Refactor INodeDirectory#getExistingPathINodes() to enable returning more than INode array
- HDFS-4053.
Major improvement reported by Eli Collins and fixed by Eli Collins
Increase the default block size
The default blocks size prior to this change was 64MB. This jira changes the default block size to 128MB. To go back to previous behavior, please configure the in hdfs-site.xml, the configuration parameter "dfs.blocksize" to 67108864.
- HDFS-4013.
Trivial bug reported by Chao Shi and fixed by Chao Shi (hdfs-client)
TestHftpURLTimeouts throws NPE
- HDFS-3940.
Minor improvement reported by Eli Collins and fixed by Suresh Srinivas
Add Gset#clear method and clear the block map when namenode is shutdown
- HDFS-3934.
Minor bug reported by Andy Isaacson and fixed by Colin Patrick McCabe
duplicative dfs_hosts entries handled wrong
- HDFS-3880.
Minor improvement reported by Brandon Li and fixed by Brandon Li (datanode , ha , namenode , security)
Use Builder to get RPC server in HDFS
- HDFS-3875.
Critical bug reported by Todd Lipcon and fixed by Kihwal Lee (datanode , hdfs-client)
Issue handling checksum errors in write pipeline
- HDFS-3817.
Major improvement reported by Brandon Li and fixed by Brandon Li (namenode)
avoid printing stack information for SafeModeException
- HDFS-3792.
Trivial bug reported by Todd Lipcon and fixed by Todd Lipcon (build , namenode)
Fix two findbugs introduced by HDFS-3695
- HDFS-3769.
Critical sub-task reported by liaowenrui and fixed by (ha)
standby namenode become active fails because starting log segment fail on shared storage
- HDFS-3601.
Major new feature reported by Junping Du and fixed by Junping Du (namenode)
Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
- HDFS-3499.
Major bug reported by Junping Du and fixed by Junping Du (datanode)
Make NetworkTopology support user specified topology class
- HDFS-3498.
Major improvement reported by Junping Du and fixed by Junping Du (namenode)
Make Replica Removal Policy pluggable and ReplicaPlacementPolicyDefault extensible for reusing code in subclass
- HDFS-3495.
Major new feature reported by Junping Du and fixed by Junping Du (balancer)
Update Balancer to support new NetworkTopology with NodeGroup
- HDFS-3277.
Major bug reported by Colin Patrick McCabe and fixed by Andrew Wang
fail over to loading a different FSImage if the first one we try to load is corrupt
- HDFS-3180.
Major bug reported by Daryn Sharp and fixed by Chris Nauroth (webhdfs)
Add socket timeouts to webhdfs
- HDFS-3163.
Trivial improvement reported by Brandon Li and fixed by Brandon Li (test)
TestHDFSCLI.testAll fails if the user name is not all lowercase
- HDFS-3009.
Trivial bug reported by Hari Mankude and fixed by Hari Mankude (hdfs-client)
DFSClient islocaladdress() can use similar routine in netutils
- HDFS-2857.
Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
Cleanup BlockInfo class
- HDFS-2576.
Major new feature reported by Pritam Damania and fixed by Devaraj Das (hdfs-client , namenode)
Namenode should have a favored nodes hint to enable clients to have control over block placement.
- HDFS-2572.
Trivial improvement reported by Harsh J and fixed by Harsh J (datanode)
Unnecessary double-check in DN#getHostName
- HDFS-2042.
Minor improvement reported by Eli Collins and fixed by (libhdfs)
Require c99 when building libhdfs
- HDFS-1804.
Minor new feature reported by Harsh J and fixed by Aaron T. Myers (datanode)
Add a new block-volume device choosing policy that looks at free space
There is now a new option to have the DN take into account available disk space on each volume when choosing where to place a replica when performing an HDFS write. This can be enabled by setting the config "dfs.datanode.fsdataset.volume.choosing.policy" to the value "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".
- HDFS-347.
Major improvement reported by George Porter and fixed by Colin Patrick McCabe (datanode , hdfs-client , performance)
DFS read performance suboptimal when client co-located on nodes with data