Apache Accumulo Documentation : Administration

Starting accumulo for the first time

For the most part, accumulo is ready to go out of the box. To start it, first you must distribute and install the accumulo software to each machine in the cloud that you wish to run on. The software should be installed in the same directory on each machine and configured identically (or at least similarly... see the configuration sections for more details). Select one machine to be your bootstrap machine, the one that you will start accumulo with. Note that you must have passphrase-less ssh access to each machine from your bootstrap machine. On this machine, create a conf/masters and conf/slaves file. In the masters file, type the hostname of the machine you wish to run the master on (probably localhost). In the slaves file, type the hostnames, separated by newlines of each machine you wish to participate in accumulo as a tablet server. If you neglect to create these files, the startup scripts will assume you are trying to run on localhost only, and will instantiate a single-node instance only. It is probably a good idea to back up these files, or distribute them to the other nodes as well, so that you can easily boot up accumulo from another machine, if necessary. You can also make create a conf/accumulo-env.sh file if you want to configure any custom environment variables.

Once properly configured, you can initialize or prepare an instance of accumulo by running: bin/accumulo init
Follow the prompts and you are ready to go. This step only prepares accumulo to run, it does not start up accumulo.

Starting accumulo

Once you have configured accumulo to your liking, and distributed the appropriate configuration to each machine, you can start accumulo with bin/start-all.sh. If at any time, you wish to bring accumulo servers online after one or more have been shutdown, you can run bin/start-all.sh again. This step will only start services that are not already running. Be aware that if you run this command on more than one machine, you may unintentionally start an extra copy of the garbage collector service and the monitoring service, since each of these will run on the server on which you run this script.

Stopping accumulo

Similar to the start-all.sh script, we provide a bin/stop-all.sh script to shut down accumulo. This will prompt for the root password so that it can ask the master to shut down the tablet servers gracefully. If the tablet servers do not respond, or the master takes too long, you can force a shutdown by hitting Ctrl-C at the password prompt, and waiting 15 seconds for the script to force a shutdown. Normally, once the shutdown happens gracefully, unresponsive tablet servers are forcibly shut down after 5 seconds.

Configuration

Accumulo configuration information is stored in a xml file and ZooKeeper. System wide configuration information is stored in accumulo-site.xml. In order for accumulo to find this file its directory must be on the classpath. Accumulo will log a warning if it can not find it, and will use built-in default values. The accumulo scripts try to put the config directory on the classpath.

Starting with version 1.0, per-table configuration was introduced. This information is stored in ZooKeeper. This information can be manipulated using the config command in the accumulo shell. ZooKeeper will notify all tablet servers when config properties are modified. This makes it possible to change major compaction settings, for example, for a table while accumulo is running.

Per-table configuration settings override system settings.

See the possible configuration options and their default values here

Managing system resources

It is very important how disk and memory usage are allocated across the cluster and how servers processes are allocated across the cluster.

On larger clusters, run the namenode, secondary namenode, jobtracker, accumulo master, and zookeepers on dedicated nodes. On a smaller cluster you may want to run all master processes on one node. When doing this ensure that the max total memory that could be used by all master processes does not exceed system memory. Swapping on your single master node would not be good.
Accumulo 1.2 and earlier rely on zookeeper but do not use it heavily. On a large cluster setting up 3 or 5 zookeepers should be plenty. Since there is no performance gain when running more zookeepers, fault tolerance is the only benefit.
On slave nodes ensure the memory used by all slave processes is less than system memory. For example the following slave node config could use up to 38G of RAM : tablet server 3G, logger 1G, data node 2G, up to 10 mappers each using 2G, and up 6 reducers each using 2G. If the slave nodes only have 32G, then using 38G will result in swapping which could cause tablet server to lose their lock in zookeeper and die. Even if swapping does not cause tablet servers to die, it will kill performance.
Accumulo and map reduce will work with less memory, but it has an impact. Accumulo will minor compact more frequently when it has less map memory, resulting in more major compactions. The minor and major compactions both use CPU and HDFS I/O. The same goes for map reduce, the less memory you give it, the more it has to sort and spill. Try to minimize spilling and compactions as much as possible without causing swapping.
Accumulo writes data to disk before it sorts it in memory. This allows data that was in memory when a tablet server crashes to be recovered. Each slave node needs a local directory to write this data to. Ensure the file system holding this directory has at least 100G free on all nodes. Also, if this directory is in a filesystem used by map reduce or hdfs they may effect each others performance.

There are a few settings that determine how much memory accumulo tablet servers use. In accumulo-env.sh there is a setting called ACCUMULO_TSERVER_OPTS. By default this is set to something like "-Xmx512m -Xms512m". These are Java jvm options asking Java to use 512 megabytes of memory. By default accumulo stores data written to it outside of the Java memory space in order to avoid pauses caused by the Java garbage collector. The amount of memory it uses for this data is determined by the accumulo setting "tserver.memory.maps.max". Since this memory is outside of the Java managed memory, the process can grow larger than the -Xmx setting. So if -Xmx is set to 512M and tserver.memory.maps.max is set to 1G, a tablet server process can be expected to use 1.5G. If tserver.memory.maps.native.enabled is set to false, then accumulo will only use memory managed by Java and the process will not use more than what -Xmx is set to. In this case the tserver.memory.maps.max setting should be 75% of the -Xmx setting.

Swappiness

The linux kernel will swap out memory of running programs to increase the size of the disk buffers. This tendency to swap out is controlled by a kernel setting called "swappiness." This behavior does not work well for large java servers. When a java process runs a garbage collection, it touches lots of pages forcing all swapped out pages back into memory. It is suggested that swappiness be set to zero.

 # sysctl -w vm.swappiness=0
 # echo "vm.swappiness = 0" >> /etc/sysctl.conf

Hadoop timeouts

In order to detect failed datanodes, use shorter timeouts. Add the following to your hdfs-site.xml file:


  <property>
    <name>dfs.socket.timeout</name>
    <value>3000</value>
  </property>

  <property>
    <name>dfs.socket.write.timeout</name>
    <value>5000</value>
  </property>

  <property>
    <name>ipc.client.connect.timeout</name>
    <value>1000</value>
  </property>

  <property>
    <name>ipc.clident.connect.max.retries.on.timeouts</name>
    <value>2</value>
  </property>