Ambari Management Pack Development

Adding a new property

1. Add the property to the appropriate *-env.xml file found in METRON.CURRENT/configuration.

   <property>
       <name>new_property</name>
       <description>New Property description</description>
       <value>Default Value</value>
       <display-name>New Property Pretty Name</display-name>
   </property>
   

   The appropriate *-env.xml file should be selected based on which component depends on the property. This allows Ambari to accurately restart only the affected components when the property is changed. If a property is in metron-env.xml, Ambari will prompt you to restart all Metron components.

2. Add the property to the metron_theme.json file found in METRON.CURRENT/themes if the property was added to a component-specific *-env.xml file (metron-parsers-env.xml for example) and not metron-env.xml. This is necessary for the property to be displayed in the correct tab of the Metron Configs section in the Ambari UI. Using other properties as a guide, add the property to /configuration/placement/configs for proper placement and also to /configuration/widgets for a specific widget type.

3. Reference the property in METRON.CURRENT/package/scriptes/params/params_linux.py, unless it will be used in Ambari’s status command. It will be stored in a variable. The name doesn’t have to match, but it’s preferred that it does. Make sure to use replace metron-env the correct *-env file, as noted above.

     new_property = config['configurations']['metron-env']['new_property']
   

   If this property will be used in the status command, instead make this change in METRON.CURRENT/package/scriptes/params/status_params.py. Afterwards, in params_linux.py, reference the new property:

  new_property = status_params.new_property

This behavior is because Ambari doesn’t send all parameters to the status, so it needs to be explicitly provided.

1. Ambari master services can then import the params:

     from params import params
     env.set_params(params)
   

2. The *_commands.py files receive the params as an input from the master services. Once this is done, they can be accessed via the variable we set above:

     self.__params.new_property
   

<property>
    <name>metron_apps_hdfs_dir</name>
    <value>/apps/metron</value>
    <description>Metron apps HDFS dir</description>
    <display-name>Metron apps HDFS dir</display-name>
</property>

metron_apps_hdfs_dir = config['configurations']['metron-env']['metron_apps_hdfs_dir']

metron_apps_hdfs_dir = status_params.metron_apps_hdfs_dir

from resource_management.libraries.functions import format
# ...
hdfs_grok_patterns_dir = format("{metron_apps_hdfs_dir}/patterns")

def __init__(self, params):
    if params is None:
        raise ValueError("params argument is required for initialization")
    self.__params = params
    # Other initialization

def stop(self, env, upgrade_type=None):
    from params import params
    env.set_params(params)
    commands = ParserCommands(params)
    commands.stop_parser_topologies()

def init_parsers(self):
    Logger.info(
        "Copying grok patterns from local directory '{0}' to HDFS '{1}'".format(self.__params.local_grok_patterns_dir,
                                                                                self.__params.hdfs_grok_patterns_dir))

    self.__params.HdfsResource(self.__params.hdfs_grok_patterns_dir,
                               type="directory",
                               action="create_on_execute",
                               owner=self.__params.metron_user,
                               mode=0755,
                               source=self.__params.local_grok_patterns_dir)

    Logger.info("Done initializing parser configuration")

<property>
    <name>metron_log_dir</name>
    <value>/var/log/metron</value>
    <description>Log directory for metron</description>
    <display-name>Metron log dir</display-name>
</property>

metron_log_dir = config['configurations']['metron-env']['metron_log_dir']

METRON_LOG_DIR="{{metron_log_dir}}"

from resource_management.core.resources.system import File
from resource_management.core.source import Template

def configure(self, env, upgrade_type=None, config_dir=None):
    # Do stuff

    env.set_params(params)
    File(format("/etc/sysconfig/metron"),
         content=Template("metron.j2")
         )
    # Do stuff

...
METRON_LOG_DIR="/var/log/metron"
...

{
  "configuration": {
    "layouts": [...],
    "placement": {
      "configs": [
        {
          "config": "metron-enrichment-env/enrichment_workers",
          "subsection-name": "subsection-enrichment-storm"
        },
        ...
      ]
    },
    "widgets": [
      {
        "config": "metron-enrichment-env/enrichment_workers",
        "widget": {
          "type": "text-field"
        }
      }
    ]
  }
}

How to identify errors in MPack changes

Typically, any errors are thrown at one of two times:

1. Attempting to install Metron as a whole. These are typically service level definition errors, not property errors. Logs will often be found in /var/log/ambari-server/ambari-server.log. Often the log will indicate it was unable to load a particular service or subservice with some more detail.
2. Running the actual functionality. These typically tend to cause errors in the UI at runtime. Logs are usually found in /var/log/ambari-agent/ambari-agent.log.

Unfortunately, because errors tend to occur at runtime, it’s often necessary to add things like logging statements, or even just throw errors to print out in the Ambari UI.

The primary solution to these is to look in the logs for exceptions, see what’s going wrong (Property doesn’t exist? Malformed file couldn’t be loaded?), and adjust appropriately.

Testing changes without cycling Vagrant build

There are techniques we can use to avoid spinning down and spinning back up Vagrant.

/var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services
/var/lib/ambari-agent/cache/common-services

/var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py
/var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py

[root@node1 ~]# find /var/lib/ -name enrichment_commands.py
/var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py
/var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py

Configuration involving dependency services

Metron can define expectations on other services, e.g. Storm’s topology.classpath should be /etc/hbase/conf:/etc/hadoop/conf. This happens in METRON.CURRENT/service_advisor.py.

The value is defined in a map in getSTORMSiteDesiredValues(self, is_secured). This map is used by validateSTORMSiteConfigurations to ensure that the value used by the component is what we need it to be. Generally, only the map should need to be modified.

Properties from the other services can also be used and examined. We use this to build the appropriate URL for Storm, because it differs based on Kerberos security. For example, to retrieve from another service, access the configurations array, retrieve the appropriate config file, retrieve the properties, and finally the desired property.

stormUIServerPort = services["configurations"]["storm-site"]["properties"]["ui.port"]

The security of the cluster can be checked with

is_secured = self.isSecurityEnabled(services)

Note that the configuration expectations will be properly enforced if the cluster is upgraded (e.g. the Storm URL is properly updated) and it does not need to occur manually.

Kerberos

Any scripts interacting with underlying secured tech needs to have kinit run as needed. This includes anything interacting with HDFS, HBase, Kafka, Storm, etc. This kinit should be run in a conditional statement, to avoid breaking non-Kerberized clusters.

Ambari can run kinit, as a given user with the appropriate keytab like so:

if self.__params.security_enabled:
    metron_security.kinit(self.__params.kinit_path_local,
                          self.__params.metron_keytab_path,
                          self.__params.metron_principal_name,
                          execute_user=self.__params.metron_user)

The security_enabled param is already made available, along with appropriate keytabs for metron, hdfs, kafka, and hbase users.

Best practices

• Write scripts to be idempotent. The pattern currently used is to write a file out when a task is finished, e.g. setting up ACLs or tables. For example, when indexing is configured, a file is written out and checked based on a property.

    def set_configured(self):
    File(self.__params.indexing_configured_flag_file,
         content="",
         owner=self.__params.metron_user,
         mode=0755)
  

  This is checked in the indexing master

  if not commands.is_configured():
      commands.init_kafka_topics()
      commands.init_hdfs_dir()
      commands.set_configured()

• Ensure ACLs are properly managed. This includes Kafka and HBase. Often this involves a config file written out as above because this isn’t idempotent!
  • Make sure to kinit as the correct user for setting up ACLs in a secured cluster. This is usually kafka for Kafka and hbase for HBase.
  • See set_hbase_acls in METRON.CURRENT/package/scripts/enrichment_commands.py for an HBase example
  • See init_kafka_acls in METRON.CURRENT/package/scripts/enrichment_commands.py and METRON.CURRENT/package/scripts/metron_service.py for an Kafka example