This processor is used to tail a file or multiple files according to multiple modes. The mode to choose depends of the logging pattern followed by the file(s) to tail. In any case, if there is a rolling pattern, the rolling files must be plain text files (compression is not supported at the moment).
In case the 'Rolling filename pattern' property is used, when the processor detects that the file to tail has rolled over, the processor will look for possible missing messages in the rolled file. To do so, the processor will use the pattern to find the rolling files in the same directory as the file to tail.
In order to keep this property available in the 'Multiple files' mode when multiples files to tail are in the same directory, it is possible to use the ${filename} tag to reference the name (without extension) of the file to tail. For example, if we have:
/my/path/directory/my-app.log.1
/my/path/directory/my-app.log
/my/path/directory/application.log.1
/my/path/directory/application.log
the 'rolling filename pattern' would be ${filename}.log.*.
The 'Single file' mode assumes that the file to tail has always the same name even if there is a rolling pattern. Example:
/my/path/directory/my-app.log.2
/my/path/directory/my-app.log.1
/my/path/directory/my-app.log
and new log messages are always appended in my-app.log file.
In case recursivity is set to 'true'. The regular expression for the files to tail must embrace the possible intermediate directories between the base directory and the files to tail. Example:
/my/path/directory1/my-app1.log
/my/path/directory2/my-app2.log
/my/path/directory3/my-app3.log
Base directory: /my/path
Files to tail: directory[1-3]/my-app[1-3].log
Recursivity: true
In the 'Multiple files' mode, it is possible to specify if the file to tail has always the same name or not. It is done through the property 'Rolling strategy'. The strategy can be 'Fixed name' in case the files to tail have always the same name (see example above) or can be 'Changing name' in case the files to tail do not always have the same name. Example:
/my/path/directory/my-app-2016-09-06.log
/my/path/directory/my-app-2016-09-07.log
/my/path/directory/my-app-2016-09-08.log
and new log messages are always appended in log file of the current day.
If the processor is configured with 'Multiple files' mode and 'Fixed name' rolling strategy, the processor will list the files to tail in the 'Base directory' (recursively or not) and matching the regular expression. This listing will only occur once when the processor is started. In this configuration, the processor will act as in 'Single file' mode for all files listed by the processor when started.
If the processor is configured with 'Multiple files' mode and 'Changing name' rolling strategy, two new properties are mandatory:
It is necessary to pay attention to 'Lookup frequency' and 'Maximum age' properties as well as the frequency at which the processor is triggered in order to keep good performances. It is recommended to keep 'Maximum age' > 'Lookup frequency' > processor scheduling frequency to avoid loosing messages. It also recommended not to set 'Maximum Age' too low because if messages are appended in a file after this file has been considered "too old", all the messages in the file may be read again leading to data duplication.
Besides, if the processor is configured with 'Multiple files' mode and 'Changing name' rolling strategy, the 'Rolling filename pattern' property must be specific enough to ensure that only the rolling files will be listed and not other currently tailed files in the same directory (this can be achieved using ${filename} tag).