Hadoop HADOOP 2.0.3-alpha Release Notes
These release notes include new developer and user-facing incompatibilities, features, and major improvements.
Changes since Hadoop 2.0.2
- HADOOP-9289.
Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
FsShell rm -f fails for non-matching globs
- HADOOP-9278.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (fs)
HarFileSystem may leak file handle
- HADOOP-9276.
Minor improvement reported by Arun C Murthy and fixed by Arun C Murthy
Allow BoundedByteArrayOutputStream to be resettable
- HADOOP-9260.
Critical bug reported by Jerry Chen and fixed by Chris Nauroth
Hadoop version may be not correct when starting name node or data node
- HADOOP-9255.
Critical bug reported by Thomas Graves and fixed by Thomas Graves (scripts)
relnotes.py missing last jira
- HADOOP-9252.
Minor bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (util)
StringUtils.humanReadableInt(..) has a race condition
- HADOOP-9247.
Minor improvement reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
parametrize Clover "generateXxx" properties to make them re-definable via -D in mvn calls
- HADOOP-9231.
Major bug reported by Konstantin Boudnik and fixed by Konstantin Boudnik (build)
Parametrize staging URL for the uniformity of distributionManagement
- HADOOP-9221.
Major bug reported by Andy Isaacson and fixed by Andy Isaacson
Convert remaining xdocs to APT
- HADOOP-9217.
Major test reported by Andrey Klochkov and fixed by Andrey Klochkov (test)
Print thread dumps when hadoop-common tests fail
- HADOOP-9216.
Major improvement reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (io)
CompressionCodecFactory#getCodecClasses should trim the result of parsing by Configuration.
- HADOOP-9215.
Blocker bug reported by Thomas Graves and fixed by Colin Patrick McCabe
when using cmake-2.6, libhadoop.so doesn't get created (only libhadoop.so.1.0.0)
- HADOOP-9212.
Major bug reported by Tom White and fixed by Tom White (fs)
Potential deadlock in FileSystem.Cache/IPC/UGI
- HADOOP-9203.
Trivial bug reported by Andrew Purtell and fixed by Andrew Purtell (ipc , test)
RPCCallBenchmark should find a random available port
- HADOOP-9193.
Minor bug reported by Jason Lowe and fixed by Andy Isaacson (scripts)
hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script
- HADOOP-9192.
Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (security)
Move token related request/response messages to common
- HADOOP-9190.
Major bug reported by Thomas Graves and fixed by Andy Isaacson (documentation)
packaging docs is broken
- HADOOP-9183.
Major bug reported by Tom White and fixed by Tom White (ha)
Potential deadlock in ActiveStandbyElector
- HADOOP-9181.
Major bug reported by Liang Xie and fixed by Liang Xie
Set daemon flag for HttpServer's QueuedThreadPool
- HADOOP-9178.
Minor bug reported by Sandy Ryza and fixed by Sandy Ryza
src/main/conf is missing hadoop-policy.xml
- HADOOP-9173.
Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas
Add security token protobuf definition to common and use it in hdfs
- HADOOP-9162.
Minor improvement reported by Binglin Chang and fixed by Binglin Chang (native)
Add utility to check native library availability
- HADOOP-9155.
Minor bug reported by Binglin Chang and fixed by Binglin Chang
FsPermission should have different default value, 777 for directory and 666 for file
- HADOOP-9153.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (viewfs)
Support createNonRecursive in ViewFileSystem
- HADOOP-9152.
Minor bug reported by Brock Noland and fixed by Brock Noland (fs)
HDFS can report negative DFS Used on clusters with very small amounts of data
- HADOOP-9147.
Trivial improvement reported by Jonathan Allen and fixed by Jonathan Allen
Add missing fields to FIleStatus.toString
Update FileStatus.toString to include missing fields
- HADOOP-9135.
Trivial bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (security)
JniBasedUnixGroupsMappingWithFallback should log at debug rather than info during fallback
- HADOOP-9127.
Major improvement reported by Daisuke Kobayashi and fixed by Daisuke Kobayashi (documentation)
Update documentation for ZooKeeper Failover Controller
- HADOOP-9119.
Minor test reported by Steve Loughran and fixed by Steve Loughran (fs , test)
Add test to FileSystemContractBaseTest to verify integrity of overwritten files
Patches adds more tests to verify overwritten and more complex operations -write-delete-overwrite. By using differently sized datasets and different data inside, these tests verify that the overwrite really did take place. While HDFS meets all these requirements directly, eventually consistent object stores may not -hence these tests.
- HADOOP-9118.
Trivial improvement reported by Steve Loughran and fixed by (test)
FileSystemContractBaseTest test data for read/write isn't rigorous enough
Resolved as part of HADOOP-9119 -it's test data generator creates more bits in every test byte
- HADOOP-9113.
Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (security , test)
o.a.h.fs.TestDelegationTokenRenewer is failing intermittently
- HADOOP-9106.
Major improvement reported by Todd Lipcon and fixed by Robert Parker (ipc)
Allow configuration of IPC connect timeout
This jira introduces a new configuration parameter "ipc.client.connect.timeout". This configuration defines the Hadoop RPC connection timeout in milliseconds for a client to connect to a server. For details see the description associated with this configuration in core-default.xml.
- HADOOP-9105.
Major bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
FsShell -moveFromLocal erroneously fails
- HADOOP-9103.
Major bug reported by yixiaohua and fixed by Todd Lipcon (io)
UTF8 class does not properly decode Unicode characters outside the basic multilingual plane
- HADOOP-9097.
Critical bug reported by Tom White and fixed by Thomas Graves (build)
Maven RAT plugin is not checking all source files
- HADOOP-9093.
Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas
Move all the Exception in PathExceptions to o.a.h.fs package
- HADOOP-9090.
Minor new feature reported by Mostafa Elhemali and fixed by Mostafa Elhemali (metrics)
Support on-demand publish of metrics
- HADOOP-9072.
Major bug reported by Robert Parker and fixed by Robert Parker
Hadoop-Common-0.23-Build Fails to build in Jenkins
- HADOOP-9070.
Blocker sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
Kerberos SASL server cannot find kerberos key
- HADOOP-9067.
Minor test reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
provide test for method org.apache.hadoop.fs.LocalFileSystem.reportChecksumFailure(Path, FSDataInputStream, long, FSDataInputStream, long)
- HADOOP-9064.
Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (security)
Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction
- HADOOP-9054.
Major new feature reported by Robert Kanter and fixed by Robert Kanter (security)
Add AuthenticationHandler that uses Kerberos but allows for an alternate form of authentication for browsers
- HADOOP-9049.
Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (security)
DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from.
- HADOOP-9042.
Minor test reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
Add a test for umask in FileSystemContractBaseTest
- HADOOP-9041.
Critical bug reported by Radim Kolar and fixed by Radim Kolar (fs)
FileSystem initialization can go into infinite loop
- HADOOP-9038.
Minor test reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator
- HADOOP-9035.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (security)
Generalize setup of LoginContext
- HADOOP-9025.
Major bug reported by Robert Joseph Evans and fixed by Jonathan Eagles
org.apache.hadoop.tools.TestCopyListing failing
- HADOOP-9022.
Major bug reported by Haiyang Jiang and fixed by Jonathan Eagles
Hadoop distcp tool fails to copy file if -m 0 specified
- HADOOP-9021.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc , security)
Enforce configured SASL method on the server
- HADOOP-9020.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc , security)
Add a SASL PLAIN server
- HADOOP-9015.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
Standardize creation of SaslRpcServers
- HADOOP-9014.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
Standardize creation of SaslRpcClients
- HADOOP-9013.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (fs , security)
UGI should not hardcode loginUser's authenticationType
- HADOOP-9012.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
IPC Client sends wrong connection context
- HADOOP-9010.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (fs , security)
Map UGI authenticationMethod to RPC authMethod
- HADOOP-9009.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (fs , security)
Add SecurityUtil methods to get/set authentication method
- HADOOP-9004.
Major improvement reported by Stephen Chu and fixed by Stephen Chu (security , test)
Allow security unit tests to use external KDC
- HADOOP-8999.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
SASL negotiation is flawed
The RPC SASL negotiation now always ends with final response. If the SASL mechanism does not have a final response (GSSAPI, PLAIN), then an empty success response is sent to the client. The client will now always expect a final response to definitively know if negotiation is complete/successful.
- HADOOP-8998.
Minor improvement reported by Andy Isaacson and fixed by Alejandro Abdelnur
set Cache-Control no-cache header on all dynamic content
- HADOOP-8994.
Minor bug reported by Andy Isaacson and fixed by Andy Isaacson (test)
TestDFSShell creates file named "noFileHere", making further tests hard to understand
- HADOOP-8992.
Minor improvement reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
Enhance unit-test coverage of class HarFileSystem
- HADOOP-8986.
Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (ipc)
Server$Call object is never released after it is sent
- HADOOP-8985.
Minor improvement reported by Binglin Chang and fixed by Binglin Chang (ha , ipc)
Add namespace declarations in .proto files for languages other than java
- HADOOP-8981.
Major bug reported by Chris Nauroth and fixed by Xuan Gong (metrics)
TestMetricsSystemImpl fails on Windows
- HADOOP-8962.
Critical bug reported by Jason Lowe and fixed by Jason Lowe (fs)
RawLocalFileSystem.listStatus fails when a child filename contains a colon
- HADOOP-8951.
Minor improvement reported by Steve Loughran and fixed by Steve Loughran (util)
RunJar to fail with user-comprehensible error message if jar missing
- HADOOP-8948.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
TestFileUtil.testGetDU fails on Windows due to incorrect assumption of line separator
- HADOOP-8932.
Major improvement reported by Kihwal Lee and fixed by Kihwal Lee (security)
JNI-based user-group mapping modules can be too chatty on lookup failures
- HADOOP-8931.
Trivial improvement reported by Eli Collins and fixed by Eli Collins
Add Java version to startup message
- HADOOP-8930.
Major improvement reported by Andrey Klochkov and fixed by Andrey Klochkov (test)
Cumulative code coverage calculation
- HADOOP-8929.
Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (metrics)
Add toString, other improvements for SampleQuantiles
- HADOOP-8926.
Major improvement reported by Gopal V and fixed by Gopal V (util)
hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data
Speed up Crc32 by improving the cache hit-ratio of hadoop.util.PureJavaCrc32
- HADOOP-8925.
Minor improvement reported by Eli Collins and fixed by Eli Collins (build)
Remove the packaging
- HADOOP-8922.
Trivial improvement reported by Damien Hardy and fixed by Damien Hardy (metrics)
Provide alternate JSONP output for JMXJsonServlet to allow javascript in browser dashboard
Add a JSONP alternative outpout for /jmx HTTP interface to provide a Javascript polling ability in browsers.
- HADOOP-8913.
Minor bug reported by Sandy Ryza and fixed by Sandy Ryza (metrics)
hadoop-metrics2.properties should give units in comment for sampling period
- HADOOP-8912.
Major bug reported by Raja Aluri and fixed by Raja Aluri (build)
adding .gitattributes file to prevent CRLF and LF mismatches for source and text files
- HADOOP-8911.
Major bug reported by Raja Aluri and fixed by Raja Aluri (build)
CRLF characters in source and text files
- HADOOP-8909.
Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (build)
Hadoop Common Maven protoc calls must not depend on external sh script
- HADOOP-8906.
Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
paths with multiple globs are unreliable
- HADOOP-8901.
Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
GZip and Snappy support may not work without unversioned libraries
- HADOOP-8900.
Major bug reported by Slavik Krassovsky and fixed by Andy Isaacson
BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match decompressed size
- HADOOP-8894.
Major improvement reported by Todd Lipcon and fixed by Todd Lipcon
GenericTestUtils.waitFor should dump thread stacks on timeout
- HADOOP-8889.
Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (build , test)
Upgrade to Surefire 2.12.3
- HADOOP-8883.
Major bug reported by Robert Kanter and fixed by Robert Kanter
Anonymous fallback in KerberosAuthenticator is broken
- HADOOP-8881.
Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
FileBasedKeyStoresFactory initialization logging should be debug not info
- HADOOP-8878.
Major bug reported by Arpit Gupta and fixed by Arpit Gupta
uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on
- HADOOP-8866.
Minor improvement reported by Andrew Wang and fixed by Andrew Wang
SampleQuantiles#query is O(N^2) instead of O(N)
- HADOOP-8860.
Major task reported by Tom White and fixed by Tom White (documentation)
Split MapReduce and YARN sections in documentation navigation
- HADOOP-8855.
Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (security)
SSL-based image transfer does not work when Kerberos is disabled
- HADOOP-8851.
Minor improvement reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky (test)
Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests
- HADOOP-8849.
Minor improvement reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
FileUtil#fullyDelete should grant the target directories +rwx permissions before trying to delete them
- HADOOP-8843.
Critical bug reported by Robert Joseph Evans and fixed by Jason Lowe
Old trash directories are never deleted on upgrade from 1.x
- HADOOP-8833.
Major bug reported by Harsh J and fixed by Harsh J (fs)
fs -text should make sure to call inputstream.seek(0) before using input stream
- HADOOP-8822.
Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
relnotes.py was deleted post mavenization
- HADOOP-8819.
Major bug reported by Brandon Li and fixed by Brandon Li (fs)
Should use && instead of & in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs
- HADOOP-8816.
Major bug reported by Moritz Moeller and fixed by Moritz Moeller (net)
HTTP Error 413 full HEAD if using kerberos authentication
- HADOOP-8812.
Minor improvement reported by Eli Collins and fixed by Eli Collins
ExitUtil#terminate should print Exception#toString
- HADOOP-8811.
Critical bug reported by Radim Kolar and fixed by Radim Kolar (native)
Compile hadoop native library in FreeBSD
- HADOOP-8806.
Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (build)
libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
- HADOOP-8804.
Minor improvement reported by Eli Collins and fixed by Senthil V Kumar
Improve Web UIs when the wildcard address is used
- HADOOP-8795.
Minor bug reported by Sean Mackrory and fixed by Sean Mackrory (scripts)
BASH tab completion doesn't look in PATH, assumes path to executable is specified
- HADOOP-8791.
Major bug reported by Bertrand Dechoux and fixed by Jing Zhao (documentation)
rm "Only deletes non empty directory and files."
- HADOOP-8789.
Minor improvement reported by Andy Isaacson and fixed by Andy Isaacson (test)
Tests setLevel(Level.OFF) should be Level.ERROR
- HADOOP-8786.
Major bug reported by Todd Lipcon and fixed by Todd Lipcon
HttpServer continues to start even if AuthenticationFilter fails to init
- HADOOP-8784.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc , security)
Improve IPC.Client's token use
- HADOOP-8783.
Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc , security)
Improve RPC.Server's digest auth
- HADOOP-8780.
Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan
Update DeprecatedProperties apt file
- HADOOP-8756.
Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
- HADOOP-8755.
Major improvement reported by Andrey Klochkov and fixed by Andrey Klochkov (test)
Print thread dump when tests fail due to timeout
- HADOOP-8736.
Major improvement reported by Brandon Li and fixed by Brandon Li (ipc)
Add Builder for building an RPC server
- HADOOP-8713.
Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
TestRPCCompatibility fails intermittently with JDK7
- HADOOP-8712.
Minor improvement reported by Robert Parker and fixed by Robert Parker (security)
Change default hadoop.security.group.mapping
The default group mapping policy has been changed to JniBasedUnixGroupsNetgroupMappingWithFallback. This should maintain the same semantics as the prior default for most users.
- HADOOP-8684.
Minor bug reported by Hiroshi Ikeda and fixed by Jing Zhao (io)
Deadlock between WritableComparator and WritableComparable
- HADOOP-8616.
Major bug reported by Eli Collins and fixed by Sandy Ryza (viewfs)
ViewFS configuration requires a trailing slash
- HADOOP-8597.
Major new feature reported by Harsh J and fixed by Ivan Vladimirov Ivanov (fs)
FsShell's Text command should be able to read avro data files
- HADOOP-8589.
Major bug reported by Andrey Klochkov and fixed by Sanjay Radia (fs , test)
ViewFs tests fail when tests and home dirs are nested
- HADOOP-8561.
Major improvement reported by Luke Lu and fixed by Yu Gao (security)
Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes
- HADOOP-8427.
Major task reported by Eli Collins and fixed by Andy Isaacson (documentation)
Convert Forrest docs to APT, incremental
- HADOOP-8418.
Major bug reported by Luke Lu and fixed by Yu Gao (security)
Fix UGI for IBM JDK running on Windows
- HADOOP-7886.
Minor improvement reported by Jakob Homan and fixed by SreeHari
Add toString to FileStatus
- HADOOP-7688.
Major improvement reported by Tsz Wo (Nicholas), SZE and fixed by Uma Maheswara Rao G
When a servlet filter throws an exception in init(..), the Jetty server failed silently.
- HADOOP-7115.
Major bug reported by Arun C Murthy and fixed by Alejandro Abdelnur
Add a cache for getpwuid_r and getpwgid_r calls
- HADOOP-6762.
Critical bug reported by sam rash and fixed by sam rash
exception while doing RPC I/O closes channel
- HADOOP-6607.
Minor bug reported by Steve Loughran and fixed by Alejandro Abdelnur (io)
Add different variants of non caching HTTP headers