public class KafkaProducer
extends java.lang.Object
KafkaProducer
is a connector for publishing a stream of tuples
to Apache Kafka messaging system topics.
The connector uses and includes components from the Kafka 0.8.2.2 release. It has been successfully tested against kafka_2.11-0.10.1.0 and kafka_2.11-0.9.0.0 server as well. For more information about Kafka see http://kafka.apache.org
Sample use:
String bootstrapServers = "localhost:9092";
String topic = "mySensorReadingsTopic";
Map<String,Object> config = new HashMap<>();
config.put("bootstrap.servers", bootstrapServers);
Topology t = ...
KafkaProducer kafka = new KafkaProducer(t, () -> config);
TStream<JsonObject> sensorReadings = t.poll(
() -> getSensorReading(), 5, TimeUnit.SECONDS);
// publish as sensor readings as JSON
kafka.publish(sensonReadings, tuple -> tuple.toString(), topic);
Constructor and Description |
---|
KafkaProducer(Topology t,
Supplier<java.util.Map<java.lang.String,java.lang.Object>> config)
Create a producer connector for publishing tuples to Kafka topics.s
|
Modifier and Type | Method and Description |
---|---|
TSink<java.lang.String> |
publish(TStream<java.lang.String> stream,
java.lang.String topic)
Publish the stream of tuples as Kafka key/value records
to the specified partitions of the specified topics.
|
<T> TSink<T> |
publish(TStream<T> stream,
Function<T,java.lang.String> keyFn,
Function<T,java.lang.String> valueFn,
Function<T,java.lang.String> topicFn,
Function<T,java.lang.Integer> partitionFn)
Publish the stream of tuples as Kafka key/value records
to the specified partitions of the specified topics.
|
<T> TSink<T> |
publishBytes(TStream<T> stream,
Function<T,byte[]> keyFn,
Function<T,byte[]> valueFn,
Function<T,java.lang.String> topicFn,
Function<T,java.lang.Integer> partitionFn)
Publish the stream of tuples as Kafka key/value records
to the specified topic partitions.
|
public KafkaProducer(Topology t, Supplier<java.util.Map<java.lang.String,java.lang.Object>> config)
See the Apache Kafka documentation for KafkaProducer
configuration properties at http://kafka.apache.org.
Configuration property values are strings.
The Kafka "New Producer configs" are used. Minimal configuration typically includes:
bootstrap.servers
The full set of producer configuration items are specified in
org.apache.kafka.clients.producer.ProducerConfig
t
- Topology to add toconfig
- KafkaProducer configuration information.public <T> TSink<T> publishBytes(TStream<T> stream, Function<T,byte[]> keyFn, Function<T,byte[]> valueFn, Function<T,java.lang.String> topicFn, Function<T,java.lang.Integer> partitionFn)
If a valid partition number is specified that partition will be used when sending the message. If no partition is specified but a key is present a partition will be chosen using a hash of the key. If neither key nor partition is present a partition will be assigned in a round-robin fashion.
T
- Tuple typestream
- the stream to publishkeyFn
- A function that yields an optional byte[]
Kafka record's key from the tuple.
Specify null or return a null value for no key.valueFn
- A function that yields the byte[]
Kafka record's value from the tuple.topicFn
- A function that yields the topic from the tuple.partitionFn
- A function that yields the optional topic
partition specification from the tuple.
Specify null or return a null value for no partition specification.TSink
public <T> TSink<T> publish(TStream<T> stream, Function<T,java.lang.String> keyFn, Function<T,java.lang.String> valueFn, Function<T,java.lang.String> topicFn, Function<T,java.lang.Integer> partitionFn)
This is a convenience method for String
typed key/value
conversion functions.
T
- Tuple typestream
- the stream to publishkeyFn
- A function that yields an optional String
Kafka record's key from the tuple.
Specify null or return a null value for no key.valueFn
- A function that yields the String for the
Kafka record's value from the tuple.topicFn
- A function that yields the topic from the tuple.partitionFn
- A function that yields the optional topic
partition specification from the tuple.
Specify null or return a null value for no partition specification.TSink
publishBytes(TStream, Function, Function, Function, Function)
public TSink<java.lang.String> publish(TStream<java.lang.String> stream, java.lang.String topic)
This is a convenience method for a String stream published as a Kafka record with no key and a value consisting of the String tuple serialized as UTF-8, and publishing round-robin to a fixed topic's partitions.
stream
- the stream to publishtopic
- The topic to publish toTSink
publish(TStream, Function, Function, Function, Function)
Copyright © 2016 The Apache Software Foundation. All Rights Reserved - bbe71fa-20161201-1641