Hadoop MAPREDUCE 2.1.0-beta Release Notes
These release notes include new developer and user-facing incompatibilities, features, and major improvements.
Changes since Hadoop 2.0.5-alpha
- MAPREDUCE-5421.
Blocker bug reported by Junping Du and fixed by Junping Du (test)
TestNonExistentJob is failed due to recent changes in YARN
- MAPREDUCE-5419.
Major bug reported by Robert Parker and fixed by Robert Parker (mrv2)
TestSlive is getting FileNotFound Exception
- MAPREDUCE-5412.
Major bug reported by Jian He and fixed by Jian He
Change MR to use multiple containers API of ContainerManager after YARN-926
- MAPREDUCE-5399.
Blocker bug reported by Stanislav Barton and fixed by Stanislav Barton (mrv1 , mrv2)
Unnecessary Configuration instantiation in IFileInputStream slows down merge
Fixes blank Configuration object creation overhead by reusing the Job configuration in InMemoryReader.
- MAPREDUCE-5398.
Major improvement reported by Bikas Saha and fixed by Jian He
MR changes for YARN-513
- MAPREDUCE-5366.
Minor bug reported by Chuan Liu and fixed by Chuan Liu (test)
TestMRAsyncDiskService fails on Windows
- MAPREDUCE-5360.
Minor bug reported by Chuan Liu and fixed by Chuan Liu (test)
TestMRJobClient fails on Windows due to path format
- MAPREDUCE-5359.
Minor bug reported by Chuan Liu and fixed by Chuan Liu
JobHistory should not use File.separator to match timestamp in path
- MAPREDUCE-5357.
Minor bug reported by Chuan Liu and fixed by Chuan Liu
Job staging directory owner checking could fail on Windows
- MAPREDUCE-5355.
Minor bug reported by Chuan Liu and fixed by Chuan Liu
MiniMRYarnCluster with localFs does not work on Windows
- MAPREDUCE-5352.
Major improvement reported by Siddharth Seth and fixed by Siddharth Seth
Optimize node local splits generated by CombineFileInputFormat
- MAPREDUCE-5349.
Minor bug reported by Chuan Liu and fixed by Chuan Liu
TestClusterMapReduceTestCase and TestJobName fail on Windows in branch-2
- MAPREDUCE-5334.
Blocker bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
TestContainerLauncherImpl is failing
- MAPREDUCE-5333.
Major test reported by Alejandro Abdelnur and fixed by Wei Yan (mr-am)
Add test that verifies MRAM works correctly when sending requests with non-normalized capabilities
- MAPREDUCE-5328.
Major bug reported by Omkar Vinit Joshi and fixed by Omkar Vinit Joshi
ClientToken should not be set in the environment
- MAPREDUCE-5326.
Blocker bug reported by Arun C Murthy and fixed by Zhijie Shen
Add version to shuffle header
- MAPREDUCE-5325.
Major bug reported by Xuan Gong and fixed by Xuan Gong
ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter---MR changes
- MAPREDUCE-5319.
Major bug reported by yeshavora and fixed by Xuan Gong
Job.xml file does not has 'user.name' property for Hadoop2
- MAPREDUCE-5315.
Critical bug reported by Mithun Radhakrishnan and fixed by Mithun Radhakrishnan (distcp)
DistCp reports success even on failure.
- MAPREDUCE-5312.
Major bug reported by Alejandro Abdelnur and fixed by Sandy Ryza
TestRMNMInfo is failing
- MAPREDUCE-5310.
Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (applicationmaster)
MRAM should not normalize allocation request capabilities
- MAPREDUCE-5308.
Major bug reported by Nathan Roberts and fixed by Nathan Roberts
Shuffling to memory can get out-of-sync when fetching multiple compressed map outputs
- MAPREDUCE-5304.
Blocker sub-task reported by Alejandro Abdelnur and fixed by Karthik Kambatla
mapreduce.Job killTask/failTask/getTaskCompletionEvents methods have incompatible signature changes
- MAPREDUCE-5303.
Major bug reported by Jian He and fixed by Jian He
Changes on MR after moving ProtoBase to package impl.pb on YARN-724
- MAPREDUCE-5301.
Major bug reported by Siddharth Seth and fixed by Siddharth Seth
Update MR code to work with YARN-635 changes
- MAPREDUCE-5300.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
Two function signature changes in filecache.DistributedCache
- MAPREDUCE-5299.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent
- MAPREDUCE-5298.
Major new feature reported by Steve Loughran and fixed by Steve Loughran (applicationmaster)
Move MapReduce services to YARN-117 stricter lifecycle
- MAPREDUCE-5297.
Major bug reported by Jian He and fixed by Jian He
Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748
- MAPREDUCE-5296.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
Mapred API: Function signature change in JobControl
- MAPREDUCE-5291.
Major bug reported by Siddharth Seth and fixed by Zhijie Shen
Change MR App to use update property names in container-log4j.properties
- MAPREDUCE-5289.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Jian He
Update MR App to use Token directly after YARN-717
- MAPREDUCE-5286.
Major task reported by Siddharth Seth and fixed by Vinod Kumar Vavilapalli
startContainer call should use the ContainerToken instead of Container [YARN-684]
- MAPREDUCE-5285.
Major bug reported by Jian He and fixed by
Update MR App to use immutable ApplicationAttemptID, ContainerID, NodeID after YARN-735
- MAPREDUCE-5283.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (applicationmaster , test)
Over 10 different tests have near identical implementations of AppContext
- MAPREDUCE-5282.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Siddharth Seth
Update MR App to use immutable ApplicationID after YARN-716
- MAPREDUCE-5280.
Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
Mapreduce API: ClusterMetrics incompatibility issues with MR1
- MAPREDUCE-5275.
Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
Mapreduce API: TokenCache incompatibility issues with MR1
- MAPREDUCE-5274.
Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
Mapreduce API: String toHex(byte[]) is removed from SecureShuffleUtils
- MAPREDUCE-5273.
Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
Protected variables are removed from CombineFileRecordReader in both mapred and mapreduce
- MAPREDUCE-5270.
Major bug reported by Jian He and fixed by Jian He
Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
- MAPREDUCE-5268.
Major improvement reported by Jason Lowe and fixed by Karthik Kambatla (jobhistoryserver)
Improve history server startup performance
- MAPREDUCE-5263.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
filecache.DistributedCache incompatiblity issues with MR1
- MAPREDUCE-5259.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic (test)
TestTaskLog fails on Windows because of path separators missmatch
- MAPREDUCE-5257.
Major bug reported by Jason Lowe and fixed by Omkar Vinit Joshi (mr-am , mrv2)
TestContainerLauncherImpl fails
- MAPREDUCE-5246.
Major improvement reported by Mayank Bansal and fixed by Mayank Bansal
Adding application type to submission context
- MAPREDUCE-5245.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
A number of public static variables are removed from JobConf
- MAPREDUCE-5244.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
Two functions changed their visibility in JobStatus
- MAPREDUCE-5240.
Blocker bug reported by Roman Shaposhnik and fixed by Vinod Kumar Vavilapalli (mrv2)
inside of FileOutputCommitter the initialized Credentials cache appears to be empty
- MAPREDUCE-5239.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Siddharth Seth
Update MR App to reflect YarnRemoteException changes after YARN-634
- MAPREDUCE-5237.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
ClusterStatus incompatiblity issues with MR1
- MAPREDUCE-5235.
Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
mapred.Counters incompatiblity issues with MR1
- MAPREDUCE-5234.
Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
Signature changes for getTaskId of TaskReport in mapred
- MAPREDUCE-5233.
Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
Functions are changed or removed from Job in jobcontrol
- MAPREDUCE-5231.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
Constructor of DBInputFormat.DBRecordReader in mapred is changed
- MAPREDUCE-5230.
Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
createFileSplit is removed from NLineInputFormat of mapred
- MAPREDUCE-5229.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce
- MAPREDUCE-5228.
Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
- MAPREDUCE-5226.
Major bug reported by Xuan Gong and fixed by Xuan Gong
Handle exception related changes in YARN's AMRMProtocol api after YARN-630
- MAPREDUCE-5222.
Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla
Fix JobClient incompatibilities with MR1
- MAPREDUCE-5220.
Major sub-task reported by Sandy Ryza and fixed by Zhijie Shen (client)
Mapred API: TaskCompletionEvent incompatibility issues with MR1
- MAPREDUCE-5213.
Minor bug reported by Karthik Kambatla and fixed by Karthik Kambatla
Re-assess TokenCache methods marked @Private
- MAPREDUCE-5212.
Major bug reported by Xuan Gong and fixed by Xuan Gong
Handle exception related changes in YARN's ClientRMProtocol api after YARN-631
- MAPREDUCE-5209.
Minor bug reported by Radim Kolar and fixed by Tsuyoshi OZAWA (mrv2)
ShuffleScheduler log message incorrect
- MAPREDUCE-5208.
Major bug reported by Omkar Vinit Joshi and fixed by Omkar Vinit Joshi
SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
- MAPREDUCE-5205.
Blocker bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
Apps fail in secure cluster setup
- MAPREDUCE-5204.
Major bug reported by Xuan Gong and fixed by Xuan Gong
Handle YarnRemoteException separately from IOException in MR api
- MAPREDUCE-5199.
Blocker sub-task reported by Vinod Kumar Vavilapalli and fixed by Daryn Sharp (security)
AppTokens file can/should be removed
- MAPREDUCE-5194.
Minor task reported by Chris Douglas and fixed by Chris Douglas (task)
Heed interrupts during Fetcher shutdown
- MAPREDUCE-5193.
Major bug reported by Aaron T. Myers and fixed by Andrew Wang (test)
A few MR tests use block sizes which are smaller than the default minimum block size
- MAPREDUCE-5192.
Minor task reported by Chris Douglas and fixed by Chris Douglas (task)
Separate TCE resolution from fetch
- MAPREDUCE-5191.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic
TestQueue#testQueue fails with timeout on Windows
- MAPREDUCE-5187.
Major bug reported by Chuan Liu and fixed by Chuan Liu (mrv2)
Create mapreduce command scripts on Windows
- MAPREDUCE-5184.
Major sub-task reported by Arun C Murthy and fixed by Zhijie Shen (documentation)
Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2
Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2 for end-users.
- MAPREDUCE-5181.
Major bug reported by Siddharth Seth and fixed by Vinod Kumar Vavilapalli (applicationmaster)
RMCommunicator should not use AMToken from the env
- MAPREDUCE-5179.
Major bug reported by Hitesh Shah and fixed by Hitesh Shah
Change TestHSWebServices to do string equal check on hadoop build version similar to YARN-605
- MAPREDUCE-5178.
Major bug reported by Hitesh Shah and fixed by Hitesh Shah
Fix use of BuilderUtils#newApplicationReport as a result of YARN-577.
- MAPREDUCE-5177.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic
Move to common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
- MAPREDUCE-5176.
Major improvement reported by Carlo Curino and fixed by Carlo Curino (mrv2)
Preemptable annotations (to support preemption in MR)
- MAPREDUCE-5175.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Xuan Gong
Update MR App to not set envs that will be set by NMs anyways after YARN-561
- MAPREDUCE-5171.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (applicationmaster)
Expose blacklisted nodes from the MR AM REST API
- MAPREDUCE-5167.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Jian He
Update MR App after YARN-562
- MAPREDUCE-5166.
Blocker bug reported by Gunther Hagleitner and fixed by Sandy Ryza
ConcurrentModificationException in LocalJobRunner
- MAPREDUCE-5163.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Xuan Gong
Update MR App after YARN-441
- MAPREDUCE-5159.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate
- MAPREDUCE-5157.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
- MAPREDUCE-5156.
Blocker sub-task reported by Zhijie Shen and fixed by Zhijie Shen
Hadoop-examples-1.x.x.jar cannot run on Yarn
- MAPREDUCE-5152.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
MR App is not using Container from RM
- MAPREDUCE-5151.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Sandy Ryza
Update MR App after YARN-444
- MAPREDUCE-5147.
Major bug reported by Robert Parker and fixed by Robert Parker (mrv2)
Maven build should create hadoop-mapreduce-client-app-VERSION.jar directly
- MAPREDUCE-5146.
Minor bug reported by Sangjin Lee and fixed by Sangjin Lee (task)
application classloader may be used too early to load classes
- MAPREDUCE-5145.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen
Change default max-attempts to be more than one for MR jobs as well
- MAPREDUCE-5140.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen
MR part of YARN-514
- MAPREDUCE-5139.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Xuan Gong
Update MR App after YARN-486
- MAPREDUCE-5138.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Omkar Vinit Joshi
Fix LocalDistributedCacheManager after YARN-112
- MAPREDUCE-5137.
Major bug reported by Thomas Graves and fixed by Thomas Graves (applicationmaster)
AM web UI: clicking on Map Task results in 500 error
- MAPREDUCE-5136.
Major bug reported by Amir Sanjar and fixed by Amir Sanjar
TestJobImpl->testJobNoTasks fails with IBM JAVA
- MAPREDUCE-5129.
Minor new feature reported by Billie Rinaldi and fixed by Billie Rinaldi
Add tag info to JH files
- MAPREDUCE-5128.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (documentation , jobhistoryserver)
mapred-default.xml is missing a bunch of history server configs
- MAPREDUCE-5113.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza
Streaming input/output types are ignored with java mapper/reducer
- MAPREDUCE-5098.
Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (contrib/gridmix)
Fix findbugs warnings in gridmix
- MAPREDUCE-5086.
Major bug reported by Jian He and fixed by Jian He
MR app master deletes staging dir when sent a reboot command from the RM
- MAPREDUCE-5079.
Critical improvement reported by Jason Lowe and fixed by Jason Lowe (mr-am)
Recovery should restore task state from job history info directly
- MAPREDUCE-5078.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (client)
TestMRAppMaster fails on Windows due to mismatched path separators
- MAPREDUCE-5077.
Minor bug reported by Karthik Kambatla and fixed by Karthik Kambatla (mrv2)
Cleanup: mapreduce.util.ResourceCalculatorPlugin and related code should be removed
- MAPREDUCE-5075.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (distcp)
DistCp leaks input file handles
- MAPREDUCE-5069.
Minor improvement reported by Sangjin Lee and fixed by (mrv1 , mrv2)
add concrete common implementations of CombineFileInputFormat
- MAPREDUCE-5066.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic
JobTracker should set a timeout when calling into job.end.notification.url
- MAPREDUCE-5065.
Major bug reported by Mithun Radhakrishnan and fixed by Mithun Radhakrishnan (distcp)
DistCp should skip checksum comparisons if block-sizes are different on source/target.
- MAPREDUCE-5062.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen
MR AM should read max-retries information from the RM
- MAPREDUCE-5060.
Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
Fetch failures that time out only count against the first map task
- MAPREDUCE-5059.
Major bug reported by Jason Lowe and fixed by Omkar Vinit Joshi (jobhistoryserver , webapps)
Job overview shows average merge time larger than for any reduce attempt
- MAPREDUCE-5043.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)
Fetch failure processing can cause AM event queue to backup and eventually OOM
- MAPREDUCE-5042.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe (mr-am , security)
Reducer unable to fetch for a map task that was recovered
- MAPREDUCE-5033.
Minor improvement reported by Andrew Wang and fixed by Andrew Wang
mapred shell script should respect usage flags (--help -help -h)
- MAPREDUCE-5027.
Major bug reported by Jason Lowe and fixed by Robert Parker
Shuffle does not limit number of outstanding connections
- MAPREDUCE-5015.
Major test reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov
Coverage fix for org.apache.hadoop.mapreduce.tools.CLI
- MAPREDUCE-5013.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza (client)
mapred.JobStatus compatibility: MR2 missing constructors from MR1
- MAPREDUCE-5009.
Critical bug reported by Robert Parker and fixed by Robert Parker (mrv1)
Killing the Task Attempt slated for commit does not clear the value from the Task commitAttempt member
- MAPREDUCE-5008.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza
Merger progress miscounts with respect to EOF_MARKER
- MAPREDUCE-5007.
Major test reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov
fix coverage org.apache.hadoop.mapreduce.v2.hs
- MAPREDUCE-5000.
Critical bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)
TaskImpl.getCounters() can return the counters for the wrong task attempt when task is speculating
- MAPREDUCE-4994.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza (client)
-jt generic command line option does not work
- MAPREDUCE-4992.
Critical bug reported by Robert Parker and fixed by Robert Parker (mr-am)
AM hangs in RecoveryService when recovering tasks with speculative attempts
- MAPREDUCE-4991.
Major test reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov
coverage for gridmix
- MAPREDUCE-4990.
Trivial improvement reported by Karthik Kambatla and fixed by Karthik Kambatla
Construct debug strings conditionally in ShuffleHandler.Shuffle#sendMapOutput()
- MAPREDUCE-4989.
Major improvement reported by Ravi Prakash and fixed by Ravi Prakash (jobhistoryserver , mr-am)
JSONify DataTables input data for Attempts page
- MAPREDUCE-4987.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (distributed-cache , nodemanager)
TestMRJobs#testDistributedCache fails on Windows due to classpath problems and unexpected behavior of symlinks
- MAPREDUCE-4985.
Trivial bug reported by Plamen Jeliazkov and fixed by Plamen Jeliazkov
TestDFSIO supports compression but usages doesn't reflect
- MAPREDUCE-4981.
Minor bug reported by Plamen Jeliazkov and fixed by Plamen Jeliazkov
WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
- MAPREDUCE-4974.
Major improvement reported by Arun A K and fixed by Gelesh (mrv1 , mrv2 , performance)
Optimising the LineRecordReader initialize() method
- MAPREDUCE-4972.
Major test reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov
Coverage fixing for org.apache.hadoop.mapreduce.jobhistory
- MAPREDUCE-4951.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza (applicationmaster , mr-am , mrv2)
Container preemption interpreted as task failure
- MAPREDUCE-4942.
Major sub-task reported by Robert Kanter and fixed by Robert Kanter (mrv2)
mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
- MAPREDUCE-4932.
Major bug reported by Robert Kanter and fixed by Robert Kanter (mrv2)
mapreduce.job#getTaskCompletionEvents incompatible with Hadoop 1
- MAPREDUCE-4927.
Major bug reported by Jason Lowe and fixed by Ashwin Shankar (jobhistoryserver)
Historyserver 500 error due to NPE when accessing specific counters page for failed job
- MAPREDUCE-4898.
Major bug reported by Robert Kanter and fixed by Robert Kanter (mrv2)
FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
- MAPREDUCE-4896.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza (client , scheduler)
"mapred queue -info" spits out ugly exception when queue does not exist
- MAPREDUCE-4892.
Major bug reported by Bikas Saha and fixed by Bikas Saha
CombineFileInputFormat node input split can be skewed on small clusters
- MAPREDUCE-4885.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (contrib/streaming , test)
Streaming tests have multiple failures on Windows
- MAPREDUCE-4875.
Major test reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov (test)
coverage fixing for org.apache.hadoop.mapred
- MAPREDUCE-4871.
Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize
- MAPREDUCE-4846.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (client)
Some JobQueueInfo methods are public in MR1 but protected in MR2
- MAPREDUCE-4794.
Major bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster)
DefaultSpeculator generates error messages on normal shutdown
- MAPREDUCE-4737.
Major bug reported by Daniel Dai and fixed by Arun C Murthy
Hadoop does not close output file / does not call Mapper.cleanup if exception in map
Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour.
- MAPREDUCE-4716.
Major bug reported by Thomas Graves and fixed by Thomas Graves (jobhistoryserver)
TestHsWebServicesJobsQuery.testJobsQueryStateInvalid fails with jdk7
- MAPREDUCE-4693.
Major bug reported by Jason Lowe and fixed by Xuan Gong (jobhistoryserver , mrv2)
Historyserver should provide counters for failed tasks
- MAPREDUCE-4671.
Major bug reported by Bikas Saha and fixed by Bikas Saha
AM does not tell the RM about container requests that are no longer needed
- MAPREDUCE-4571.
Major bug reported by Thomas Graves and fixed by Thomas Graves (webapps)
TestHsWebServicesJobs fails on jdk7
- MAPREDUCE-4374.
Minor bug reported by Chuan Liu and fixed by Chuan Liu (mrv2)
Fix child task environment variable config and add support for Windows
- MAPREDUCE-4356.
Major bug reported by Ravi Gummadi and fixed by Ravi Gummadi (tools/rumen)
Provide access to ParsedTask.obtainTaskAttempts()
Made the method ParsedTask.obtainTaskAttempts() public.
- MAPREDUCE-4149.
Major bug reported by Ravi Gummadi and fixed by Ravi Gummadi (tools/rumen)
Rumen fails to parse certain counter strings
Fixes Rumen to parse counter strings containing the special characters "{" and "}".
- MAPREDUCE-4100.
Minor bug reported by Karam Singh and fixed by Amar Kamat (contrib/gridmix)
Sometimes gridmix emulates data larger much larger then acutal counter for map only jobs
Bug fixed in compression emulation feature for map only jobs.
- MAPREDUCE-4087.
Major bug reported by Ravi Gummadi and fixed by Ravi Gummadi
[Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
Fixes the issue of GenerateDistCacheData job slowness.
- MAPREDUCE-4083.
Major bug reported by Karam Singh and fixed by Amar Kamat (contrib/gridmix)
GridMix emulated job tasks.resource-usage emulator for CPU usage throws NPE when Trace contains cumulativeCpuUsage value of 0 at attempt level
Fixes NPE in cpu emulation in Gridmix
- MAPREDUCE-4067.
Critical bug reported by Jitendra Nath Pandey and fixed by Xuan Gong
Replace YarnRemoteException with IOException in MRv2 APIs
- MAPREDUCE-4019.
Minor bug reported by B Anil Kumar and fixed by Ashwin Shankar (client)
-list-attempt-ids is not working
- MAPREDUCE-3953.
Major bug reported by Ravi Gummadi and fixed by Ravi Gummadi
Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
Fixes NPE and makes Gridmix simulate succeeded-jobs-with-failed-tasks. All tasks of such simulated jobs(including the failed ones of original job) will succeed.
- MAPREDUCE-3872.
Major bug reported by Patrick Hunt and fixed by Robert Kanter (client , mrv2)
event handling races in ContainerLauncherImpl and TestContainerLauncher
- MAPREDUCE-3829.
Major bug reported by Ravi Gummadi and fixed by Ravi Gummadi (contrib/gridmix)
[Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given
Makes Gridmix emit out correct error message when the input data directory already exists and -generate option is used. Makes Gridmix exit with proper exit codes when Gridmix fails in args-processing, startup/setup.
- MAPREDUCE-3787.
Major improvement reported by Amar Kamat and fixed by Amar Kamat (contrib/gridmix)
[Gridmix] Improve STRESS mode
JobMonitor can now deploy multiple threads for faster job-status polling. Use 'gridmix.job-monitor.thread-count' to set the number of threads. Stress mode now relies on the updates from the job monitor instead of polling for job status. Failures in job submission now get reported to the statistics module and ultimately reported to the user via summary.
- MAPREDUCE-3757.
Major bug reported by Ravi Gummadi and fixed by Ravi Gummadi (tools/rumen)
Rumen Folder is not adjusting the shuffleFinished and sortFinished times of reduce task attempts
Fixed the sortFinishTime and shuffleFinishTime adjustments in Rumen Folder.
- MAPREDUCE-3685.
Critical bug reported by anty.rao and fixed by anty (mrv2)
There are some bugs in implementation of MergeManager
- MAPREDUCE-3533.
Minor improvement reported by Steve Loughran and fixed by (mrv2)
have the service interface extend Closeable and use close() as its shutdown operation
- MAPREDUCE-3502.
Major task reported by Steve Loughran and fixed by Steve Loughran (mrv2)
Review all Service.stop() operations and make sure that they work before a service is started
- MAPREDUCE-3008.
Major sub-task reported by Amar Kamat and fixed by Amar Kamat (contrib/gridmix)
[Gridmix] Improve cumulative CPU usage emulation for short running tasks
Improves cumulative CPU emulation for short running tasks.
- MAPREDUCE-2722.
Major bug reported by Ravi Gummadi and fixed by Ravi Gummadi (contrib/gridmix)
Gridmix simulated job's map's hdfsBytesRead counter is wrong when compressed input is used
Makes Gridmix use the uncompressed input data size while simulating map tasks in the case where compressed input data was used in original job.