This module provides a RESTful API for interacting with Metron.
Package the application with Maven:
mvn clean package
Untar the archive in the $METRON_HOME directory. The directory structure will look like:
config rest_application.yml bin metron-rest lib metron-rest-$METRON_VERSION.jar
Copy the $METRON_HOME/bin/metron-rest script to /etc/init.d/metron-rest
The REST application depends on several configuration parameters:
No optional parameter has a default.
Environment Variable | Description |
---|---|
METRON_JDBC_DRIVER | JDBC driver class |
METRON_JDBC_URL | JDBC url |
METRON_JDBC_USERNAME | JDBC username |
METRON_JDBC_PLATFORM | JDBC platform (one of h2, mysql, postgres, oracle |
ZOOKEEPER | Zookeeper quorum (ex. node1:2181,node2:2181) |
BROKERLIST | Kafka Broker list (ex. node1:6667,node2:6667) |
HDFS_URL | HDFS url or fs.defaultFS Hadoop setting (ex. hdfs://node1:8020) |
Environment Variable | Description | Required | Default |
---|---|---|---|
METRON_USER | Run the application as this user | Optional | metron |
METRON_LOG_DIR | Directory where the log file is written | Optional | /var/log/metron/ |
METRON_PID_DIR | Directory where the pid file is written | Optional | /var/run/metron/ |
METRON_REST_PORT | REST application port | Optional | 8082 |
METRON_JDBC_CLIENT_PATH | Path to JDBC client jar | Optional | H2 is bundled |
METRON_TEMP_GROK_PATH | Temporary directory used to test grok statements | Optional | ./patterns/temp |
METRON_DEFAULT_GROK_PATH | Defaults HDFS directory used to store grok statements | Optional | /apps/metron/patterns |
SECURITY_ENABLED | Enables Kerberos support | Optional | false |
Environment Variable | Description | Required |
---|---|---|
METRON_JVMFLAGS | JVM flags added to the start command | Optional |
METRON_SPRING_PROFILES_ACTIVE | Active Spring profiles (see below) | Optional |
METRON_SPRING_OPTIONS | Additional Spring input parameters | Optional |
METRON_PRINCIPAL_NAME | Kerberos principal for the metron user | Optional |
METRON_SERVICE_KEYTAB | Path to the Kerberos keytab for the metron user | Optional |
These are set in the /etc/sysconfig/metron file.
The REST application persists data in a relational database and requires a dedicated database user and database (see https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html for more detail).
The REST application comes with embedded database support for development purposes (https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support).
For example, edit these variables in /etc/sysconfig/metron before starting the application to configure H2:
METRON_JDBC_DRIVER="org.h2.Driver" METRON_JDBC_URL="jdbc:h2:file:~/metrondb" METRON_JDBC_USERNAME="root" METRON_JDBC_PASSWORD='root" METRON_JDBC_PLATFORM="h2"
The REST application should be configured with a production-grade database outside of development.
For example, the following configures the application for MySQL:
Install MySQL if not already available (this example uses version 5.7, installation instructions can be found here)
Create a metron user and REST database and permission the user for that database:
CREATE USER 'metron'@'node1' IDENTIFIED BY 'Myp@ssw0rd'; CREATE DATABASE IF NOT EXISTS metronrest; GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'node1';
Install the MySQL JDBC client onto the REST application host and configurate the METRON_JDBC_CLIENT_PATH variable:
cd $METRON_HOME/lib wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz tar xf mysql-connector-java-5.1.41.tar.gz
Edit these variables in /etc/sysconfig/metron to configure the REST application for MySQL:
METRON_JDBC_DRIVER="com.mysql.jdbc.Driver" METRON_JDBC_URL="jdbc:mysql://mysql_host:3306/metronrest" METRON_JDBC_USERNAME="metron" METRON_JDBC_PLATFORM="mysql" METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar
After configuration is complete, the REST application can be managed as a service:
service metron-rest start
If a production database is configured, the JDBC password should be passed in as the first argument on startup:
service metron-rest start Myp@ssw0rd
The REST application can be accessed with the Swagger UI at http://host:port/swagger-ui.html#/. The default port is 8082.
The metron-rest module uses Spring Security for authentication and stores user credentials in the relational database configured above. The required tables are created automatically the first time the application is started so that should be done first. For example (continuing the MySQL example above), users can be added by connecting to MySQL and running:
use metronrest; insert into users (username, password, enabled) values ('your_username','your_password',1); insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
Metron REST can be configured for a cluster with Kerberos enabled. A client JAAS file is required for Kafka and Zookeeper and a Kerberos keytab for the metron user principal is required for all other services. Configure these settings in the /etc/sysconfig/metron file:
SECURITY_ENABLED=true METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf" METRON_PRINCIPAL_NAME="metron@EXAMPLE.COM" METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
The REST application comes with a few Spring Profiles to aid in testing and development.
Profile | Description |
---|---|
test | sets variables to in-memory services, only used for integration testing |
dev | adds a test user to the database with credentials user/password |
vagrant | sets configuration variables to match the Metron vagrant environment |
docker | sets configuration variables to match the Metron dcoker environment |
Setting active profiles is done with the METRON_SPRING_PROFILES_ACTIVE variable. For example, set this variable in /etc/sysconfig/metron to configure the REST application for the Vagrant environment and add a test user:
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
Request and Response objects are JSON formatted. The JSON schemas are available in the Swagger UI.
Profiles are includes for both the metron-docker and Quick Dev environments.
Start the metron-docker environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:
mvn clean package mvn spring-boot:run -Drun.profiles=docker,dev
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
Start the Quick Dev environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:
mvn clean package mvn spring-boot:run -Drun.profiles=vagrant,dev
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
To run the application locally on the Quick Dev host (node1), follow the Installation instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in /etc/sysconfig/metron:
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
and start the application:
service metron-rest start
In a cluster with Kerberos enabled, update the security settings in /etc/sysconfig/metron. Security is disabled by default in the vagrant Spring profile so that setting must be overriden with the METRON_SPRING_OPTIONS variable:
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev" METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf" METRON_SPRING_OPTIONS="--kerberos.enabled=true"
The metron-rest application will be available at http://node1:8082/swagger-ui.html#/.
This project depends on the Java Transaction API. See https://java.net/projects/jta-spec/ for more details.