Description:
This Processors puts the contents of a FlowFile to a Topic in
Apache Kafka. The full contents of
a FlowFile becomes the contents of a single message in Kafka.
This message is optionally assigned a key by using the
<Kafka Key> Property.
The Processor allows the user to configure an optional Message Delimiter that
can be used to send many messages per FlowFile. For example, a \n could be used
to indicate that the contents of the FlowFile should be used to send one message
per line of text. If the property is not set, the entire contents of the FlowFile
will be sent as a single message. When using the delimiter, if some messages are
successfully sent but other messages fail to send, the FlowFile will be FORKed into
two child FlowFiles, with the successfully sent messages being routed to 'success'
and the messages that could not be sent going to 'failure'.
Properties:
In the list below, the names of required properties appear
in bold. Any other properties (not in bold) are considered optional.
If a property has a default value, it is indicated. If a property
supports the use of the NiFi Expression Language (or simply,
"expression language"), that is also indicated.
- Known Brokers
-
A comma-separated list of known Kafka Brokers in the format
<host>:<port>. This list does not need to be
exhaustive but provides a mechanism for determining which
other nodes belong to the Kafka cluster.
- Default value: no default
- Supports expression language: false
- Topic Name
- The Kafka Topic to send messages to. While the GetKafka
Processor requires a statically named Topic so that it knows
where to fetch messages from, the PutKafka Processor does allow
the Expression Language to be used so that a single PutKafka
Processor can be used to send messages to many different Kafka
topics.
- Default value: no default
- Supports expression language: true
- Kafka Key
-
The Key to use for the Message. If no value is given, the message
will not be given a Key.
- Default value: no default
- Supports expression language: true
- Delivery Guarantee
-
Specifies the requirement for guaranteeing that a message is sent to Kafka.
This Property can have one of three different values:
-
Guarantee Replicated Delivery - FlowFile will be routed to
failure unless the message is replicated to the appropriate number
of Kafka Nodes according to the Topic configuration
-
Guarantee Single Node Delivery - FlowFile will be routed to
success if the message is received by a single Kafka node,
whether or not it is replicated. This is faster than
<Guarantee Replicated Delivery> but can result in data loss
if a Kafka node crashes
-
Best Effort - FlowFile will be routed to success after successfully
writing the content to a Kafka node, without waiting for a response.
This provides the best performance but may result in data loss.
- Default value: Best Effort
- Supports expression language: false
- Message Delimiter
-
Specifies the delimiter to use for splitting apart multiple messages within a single FlowFile.
If not specified, the entire content of the FlowFile will be used as a single message.
If specified, the contents of the FlowFile will be split on this delimiter and each section
sent as a separate Kafka message.
- Default value: no default
- Supports expression language: true
- Communications Timeout
-
The amount of time to wait for a response from Kafka before determining
that there is a communications error
- Default value: 30 secs
- Supports expression language: false
- Max Buffer Size
-
The maximum amount of data to buffer in memory before sending to Kafka
- Default value: 1 MB
- Supports expression language: false
- Client Name
- Client Name to use when communicating with Kafka
- Default value: "NiFi-" followed by the UUID of the Processor
- Supports expression language: false
Relationships:
- success
- All FlowFiles that are successfully sent to Kafka are routed
to this relationship. If using the <Message Delimiter> property,
it's possible for some messages to be sent while others fail. In this
case, only the messages that are successfully sent will be routed to
this Relationship while the other messages will be routed to the
'failure' relationship.
- failure
- All FlowFiles that cannot be sent to Kafka for any reason be routed
to this relationship. If a portion of a FlowFile is successfully sent
to Kafka but not all, only those messages that cannot be sent to Kafka
will be routed to this Relationship.