Chapter 13. Performance Tuning

Table of Contents

13.1. Java
13.1.1. The Garbage Collector and HBase
13.2. Configurations
13.2.1. Number of Regions
13.2.2. Managing Compactions
13.2.3. Compression
13.2.4. hbase.regionserver.handler.count
13.2.5. hfile.block.cache.size
13.2.6. hbase.regionserver.global.memstore.upperLimit
13.2.7. hbase.regionserver.global.memstore.lowerLimit
13.2.8. hbase.hstore.blockingStoreFiles
13.2.9. hbase.hregion.memstore.block.multiplier
13.3. Number of Column Families
13.4. Data Clumping
13.5. Batch Loading
13.5.1. Table Creation: Pre-Creating Regions
13.6. HBase Client
13.6.1. AutoFlush
13.6.2. Scan Caching
13.6.3. Scan Attribute Selection
13.6.4. Close ResultScanners
13.6.5. Block Cache
13.6.6. Optimal Loading of Row Keys

Start with the wiki Performance Tuning page. It has a general discussion of the main factors involved; RAM, compression, JVM settings, etc. Afterward, come back here for more pointers.

Enabling RPC-level logging

Enabling the RPC-level logging on a RegionServer can often given insight on timings at the server. Once enabled, the amount of log spewed is voluminous. It is not recommended that you leave this logging on for more than short bursts of time. To enable RPC-level logging, browse to the RegionServer UI and click on Log Level. Set the log level to DEBUG for the package org.apache.hadoop.ipc (Thats right, for hadoop.ipc, NOT, hbase.ipc). Then tail the RegionServers log. Analyze.

To disable, set the logging level back to INFO level.

13.1. Java

13.1.1. The Garbage Collector and HBase

13.1.1.1. Long GC pauses

In his presentation, Avoiding Full GCs with MemStore-Local Allocation Buffers, Todd Lipcon describes two cases of stop-the-world garbage collections common in HBase, especially during loading; CMS failure modes and old generation heap fragmentation brought. To address the first, start the CMS earlier than default by adding -XX:CMSInitiatingOccupancyFraction and setting it down from defaults. Start at 60 or 70 percent (The lower you bring down the threshold, the more GCing is done, the more CPU used). To address the second fragmentation issue, Todd added an experimental facility that must be explicitly enabled in HBase 0.90.x (Its defaulted to be on in 0.92.x HBase). See hbase.hregion.memstore.mslab.enabled to true in your Configuration. See the cited slides for background and detail.