Please read this section to the end. Up front we wade through the weeds of Hadoop versions. Later we talk of what you must do in HBase to make it work w/ a particular Hadoop version.
HBase will lose data unless it is running on an HDFS that has a durable
sync
implementation. Hadoop 0.20.2, Hadoop 0.20.203.0, and Hadoop 0.20.204.0
DO NOT have this attribute.
Currently only Hadoop versions 0.20.205.x or any release in excess of this
version -- this includes hadoop 1.0.0 -- have a working, durable sync
[5]. Sync has to be explicitly enabled by setting
dfs.support.append
equal
to true on both the client side -- in hbase-site.xml
-- and on the serverside in hdfs-site.xml
(The sync
facility HBase needs is a subset of the append code path).
<property> <name>dfs.support.append</name> <value>true</value> </property>
You will have to restart your cluster after making this edit. Ignore the chicken-little
comment you'll find in the hdfs-default.xml
in the
description for the dfs.support.append
configuration; it says it is not enabled because there
are “... bugs in the 'append code' and is not supported in any production
cluster.”. This comment is stale, from another era, and while I'm sure there
are bugs, the sync/append code has been running
in production at large scale deploys and is on
by default in the offerings of hadoop by commercial vendors
[6]
[7][8].
Or use the Cloudera or MapR distributions. Cloudera' CDH3 is Apache Hadoop 0.20.x plus patches including all of the branch-0.20-append additions needed to add a durable sync. Use the released, most recent version of CDH3.
MapR includes a commercial, reimplementation of HDFS. It has a durable sync as well as some other interesting features that are not yet in Apache Hadoop. Their M3 product is free to use and unlimited.
Because HBase depends on Hadoop, it bundles an instance of the
Hadoop jar under its lib
directory. The bundled jar is ONLY for use in standalone mode.
In distributed mode, it is critical that the version of Hadoop that is out
on your cluster match what is under HBase. Replace the hadoop jar found in the HBase
lib
directory with the hadoop jar you are running on
your cluster to avoid version mismatch issues. Make sure you
replace the jar in HBase everywhere on your cluster. Hadoop version
mismatch issues have various manifestations but often all looks like
its hung up.
HBase will run on any Hadoop 0.20.x that incorporates Hadoop security features -- e.g. Y! 0.20S or CDH3B3 -- as long as you do as suggested above and replace the Hadoop jar that ships with HBase with the secure version.
An Hadoop HDFS datanode has an upper bound on the number of
files that it will serve at any one time. The upper bound parameter is
called xcievers
(yes, this is misspelled). Again,
before doing any loading, make sure you have configured Hadoop's
conf/hdfs-site.xml
setting the
xceivers
value to at least the following:
<property> <name>dfs.datanode.max.xcievers</name> <value>4096</value> </property>
Be sure to restart your HDFS after making the above configuration.
Not having this configuration in place makes for strange looking
failures. Eventually you'll see a complain in the datanode logs
complaining about the xcievers exceeded, but on the run up to this one
manifestation is complaint about missing blocks. For example:
10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block
blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node:
java.io.IOException: No live nodes contain current block. Will get new
block locations from namenode and retry...
[9]
[5] The Cloudera blog post An update on Apache Hadoop 1.0 by Charles Zedlweski has a nice exposition on how all the Hadoop versions relate. Its worth checking out if you are having trouble making sense of the Hadoop version morass.
[6] Until recently only the branch-0.20-append branch had a working sync but no official release was ever made from this branch. You had to build it yourself. Michael Noll wrote a detailed blog, Building an Hadoop 0.20.x version for HBase 0.90.2, on how to build an Hadoop from branch-0.20-append. Recommended.
[7] Praveen Kumar has written a complimentary article, Building Hadoop and HBase for HBase Maven application development.
[8] dfs.support.append
[9] See Hadoop HDFS: Deceived by Xciever for an informative rant on xceivering.