G
- topology typeT
- tuple typepublic class ConnectorStream<G extends Topology,T> extends AbstractTStream<G,T>
Modifier | Constructor and Description |
---|---|
protected |
ConnectorStream(G topology,
Connector<T> connector) |
Modifier and Type | Method and Description |
---|---|
TStream<T> |
alias(java.lang.String alias)
Set an alias for the stream.
|
protected <N extends Pipe<T,U>,U> |
connectPipe(N pipeOp) |
protected <U> ConnectorStream<G,U> |
derived(Connector<U> connector) |
<U> TStream<U> |
fanin(FanIn<T,U> fanin,
java.util.List<TStream<T>> others)
Declare a stream that contains the output of the specified
FanIn oplet applied to this stream and others . |
TStream<T> |
filter(Predicate<T> predicate)
Declare a new stream that filters tuples from this stream.
|
<U> TStream<U> |
flatMap(Function<T,java.lang.Iterable<U>> mapper)
Declare a new stream that maps tuples from this stream into one or
more (or zero) tuples of a different type
U . |
java.lang.String |
getAlias()
Returns the stream's alias if any.
|
java.util.Set<java.lang.String> |
getTags()
Returns the set of tags associated with this stream.
|
protected Graph |
graph() |
<J,U,K> TStream<J> |
join(Function<T,K> keyer,
TWindow<U,K> twindow,
BiFunction<T,java.util.List<U>,J> joiner)
Join this stream with a partitioned window of type
U with key type K . |
<J,U,K> TStream<J> |
joinLast(Function<T,K> keyer,
TStream<U> lastStream,
Function<U,K> lastStreamKeyer,
BiFunction<T,U,J> joiner)
Join this stream with the last tuple seen on a stream of type
U
with partitioning. |
<K> TWindow<T,K> |
last(int count,
Function<T,K> keyFunction)
Declare a partitioned window that continually represents the last
count
tuples on this stream for each partition. |
<K> TWindow<T,K> |
last(long time,
java.util.concurrent.TimeUnit unit,
Function<T,K> keyFunction)
Declare a partitioned window that continually represents the last
time seconds of
tuples on this stream for each partition. |
<U> TStream<U> |
map(Function<T,U> mapper)
Declare a new stream that maps (or transforms) each tuple from this stream into one
(or zero) tuple of a different type
U . |
TStream<T> |
peek(Consumer<T> peeker)
Declare a stream that contains the same contents as this stream while
peeking at each element using
peeker . |
<U> TStream<U> |
pipe(Pipe<T,U> pipe)
Declare a stream that contains the output of the specified
Pipe
oplet applied to this stream. |
TSink<T> |
sink(Sink<T> oplet)
Sink (terminate) this stream using a oplet.
|
<E extends java.lang.Enum<E>> |
split(java.lang.Class<E> enumClass,
Function<T,E> splitter)
Split a stream's tuples among
enumClass.size streams as specified by
splitter . |
java.util.List<TStream<T>> |
split(int n,
ToIntFunction<T> splitter)
Split a stream's tuples among
n streams as specified by
splitter . |
TStream<T> |
tag(java.lang.String... values)
Adds the specified tags to the stream.
|
java.lang.String |
toString()
Intended only as a debug aid and content is not guaranteed.
|
TStream<T> |
union(java.util.Set<TStream<T>> others)
Declare a stream that will contain all tuples from this stream and all the
streams in
others . |
protected <U> ConnectorStream<G,U> derived(Connector<U> connector)
protected Graph graph()
public TStream<T> filter(Predicate<T> predicate)
TStream
t
on this stream will appear in the returned stream if
filter.test(t)
returns true
. If
filter.test(t)
returns false
then then t
will not
appear in the returned stream.
Examples of filtering out all empty strings from stream s
of type
String
TStream<String> s = ...
TStream<String> filtered = s.filter(t -> !t.isEmpty());
predicate
- Filtering logic to be executed against each tuple.public <U> TStream<U> map(Function<T,U> mapper)
TStream
U
. For each tuple t
on this stream, the returned stream will contain a tuple that is the
result of mapper.apply(t)
when the return is not null
.
If mapper.apply(t)
returns null
then no tuple
is submitted to the returned stream for t
.
Examples of transforming a stream containing numeric values as
String
objects into a stream of Double
values.
// Using lambda expression
TStream<String> strings = ...
TStream<Double> doubles = strings.map(v -> Double.valueOf(v));
// Using method reference
TStream<String> strings = ...
TStream<Double> doubles = strings.map(Double::valueOf);
U
- Tuple type of output streammapper
- Mapping logic to be executed against each tuple.U
mapped from this
stream's tuples.public <U> TStream<U> flatMap(Function<T,java.lang.Iterable<U>> mapper)
TStream
U
. For each tuple
t
on this stream, the returned stream will contain all non-null tuples in
the Iterator<U>
that is the result of mapper.apply(t)
.
Tuples will be added to the returned stream in the order the iterator
returns them.
t
.
Examples of mapping a stream containing lines of text into a stream of words split out from each line. The order of the words in the stream will match the order of the words in the lines.
TStream<String> lines = ...
TStream<String> words = lines.flatMap(
line -> Arrays.asList(line.split(" ")));
U
- Type of mapped input tuples.mapper
- Mapper logic to be executed against each tuple.U
mapped and flattened from this
stream's tuples.public java.util.List<TStream<T>> split(int n, ToIntFunction<T> splitter)
TStream
n
streams as specified by
splitter
.
For each tuple on the stream, splitter.applyAsInt(tuple)
is
called. The return value r
determines the destination stream:
if r < 0 the tuple is discarded else it is sent to the stream at position (r % n) in the returned array.
Each split TStream
is exposed by the API. The user has full
control over the each stream's processing pipeline. Each stream's
pipeline must be declared explicitly. Each stream can have different
processing pipelines.
An N-way split()
is logically equivalent to a collection of N
filter()
invocations, each with a predicate to select the tuples
for its stream. split()
is more efficient. Each tuple is analyzed
only once by a single splitter
instance to identify the
destination stream. For example, these are logically equivalent:
List<TStream<String>> streams = stream.split(2, tuple -> tuple.length()); TStream<String> stream0 = stream.filter(tuple -> (tuple.length() % 2) == 0); TStream<String> stream1 = stream.filter(tuple -> (tuple.length() % 2) == 1);
Example of splitting a stream of log records by their level attribute:
TStream<LogRecord> lrs = ...
List<<TStream<LogRecord>> splits = lrr.split(3, lr -> {
if (SEVERE.equals(lr.getLevel()))
return 0;
else if (WARNING.equals(lr.getLevel()))
return 1;
else
return 2;
});
splits.get(0). ... // SEVERE log record processing pipeline
splits.get(1). ... // WARNING log record processing pipeline
splits.get(2). ... // "other" log record processing pipeline
n
- the number of output streamssplitter
- the splitter functionn
streamspublic <E extends java.lang.Enum<E>> java.util.EnumMap<E,TStream<T>> split(java.lang.Class<E> enumClass, Function<T,E> splitter)
TStream
enumClass.size
streams as specified by
splitter
.E
- Enum typeenumClass
- enum data to splitsplitter
- the splitter functionpublic TStream<T> peek(Consumer<T> peeker)
TStream
peeker
. t
on this stream, peeker.accept(t)
will be
called.peeker
- Function to be called for each tuple.this
public TSink<T> sink(Sink<T> oplet)
TStream
TStream.sink(Consumer)
with a full life-cycle of
the oplet as well as easy access to
runtime services
.oplet
- Oplet processes each tuple without producing output.public <U> TStream<U> pipe(Pipe<T,U> pipe)
TStream
Pipe
oplet applied to this stream.U
- Tuple type of the returned stream.pipe
- The Pipe
oplet.public <U> TStream<U> fanin(FanIn<T,U> fanin, java.util.List<TStream<T>> others)
TStream
FanIn
oplet applied to this stream and others
.U
- Tuple type of the returned streams.fanin
- The FanIn
oplet.others
- The other input streams.
Must not be empty or contain duplicates or this
TStream.union(Set)
,
TStream.pipe(Pipe)
,
TStream.sink(Sink)
public <K> TWindow<T,K> last(int count, Function<T,K> keyFunction)
TStream
count
tuples on this stream for each partition. Each partition independently maintains the last
count
tuples for each key seen on this stream.
If no tuples have been seen on the stream for a key then the corresponding partition will be empty.
keyFunction
.
For each tuple on the stream keyFunction.apply(tuple)
is called
and the returned value is the tuple's key. For any two tuples ta,tb
in a partition
keyFunction.apply(ta).equals(keyFunction.apply(tb))
is true.
equals()
and hashCode()
correctly.
To create a window partitioned using the tuple as the key use identity()
as the key function.
To create an unpartitioned window use a key function that returns a constant,
by convention unpartitioned()
is recommended.
K
- Key type.count
- Number of tuples to maintain in each partition.keyFunction
- Function that defines the key for each tuple.count
tuples for each partition.public <K> TWindow<T,K> last(long time, java.util.concurrent.TimeUnit unit, Function<T,K> keyFunction)
TStream
time
seconds of
tuples on this stream for each partition. If no tuples have been
seen on the stream for a key in the last time
seconds then the partition will be empty.
Each partition independently maintains the last
count
tuples for each key seen on this stream.
keyFunction
.
For each tuple on the stream keyFunction.apply(tuple)
is called
and the returned value is the tuple's key. For any two tuples ta,tb
in a partition
keyFunction.apply(ta).equals(keyFunction.apply(tb))
is true.
equals()
and hashCode()
correctly.
To create a window partitioned using the tuple as the key use identity()
as the key function.
To create an unpartitioned window use a key function that returns a constant,
by convention unpartitioned()
is recommended.
K
- Key type.time
- Time to retain a tuple in a partition.unit
- Unit for time
.keyFunction
- Function that defines the key for each tuple.count
tuple.public <J,U,K> TStream<J> join(Function<T,K> keyer, TWindow<U,K> twindow, BiFunction<T,java.util.List<U>,J> joiner)
TStream
U
with key type K
.
For each tuple on this stream, it is joined with the contents of window
for the key keyer.apply(tuple)
. Each tuple is
passed into joiner
and the return value is submitted to the
returned stream. If call returns null then no tuple is submitted.J
- Tuple type of result streamU
- Tuple type of window to join withK
- Key typekeyer
- Key function for this stream to match the window's key.twindow
- Keyed window to join this stream with.joiner
- Join function.window
.public <J,U,K> TStream<J> joinLast(Function<T,K> keyer, TStream<U> lastStream, Function<U,K> lastStreamKeyer, BiFunction<T,U,J> joiner)
TStream
U
with partitioning.
For each tuple on this
stream, it is joined with the last tuple seen on lastStream
with a matching key (of type K
).
t
on this stream will match the last tuple
u
on lastStream
if
keyer.apply(t).equals(lastStreamKeyer.apply(u))
is true.
equals
and
hashCode()
.
Each tuple is
passed into joiner
and the return value is submitted to the
returned stream. If call returns null then no tuple is submitted.
J
- Tuple type of result streamU
- Tuple type of stream to join withK
- Key typekeyer
- Key function for this streamlastStream
- Stream to join with.lastStreamKeyer
- Key function for lastStream
joiner
- Join function.lastStream
.public TStream<T> union(java.util.Set<TStream<T>> others)
TStream
others
. A stream cannot be unioned with itself, in
this case the union will only contain tuples from this stream once. If
others
is empty or only contains this
then this
is returned.others
- Stream to union with this stream.this
and others
.public TStream<T> tag(java.lang.String... values)
TStream
values
- Tag values.public java.util.Set<java.lang.String> getTags()
TStream
public TStream<T> alias(java.lang.String alias)
TStream
The alias must be unique within the topology. The alias may be used in various contexts:
alias
- an alias for the stream.ControlService
public java.lang.String getAlias()
TStream
public java.lang.String toString()
toString
in class java.lang.Object
Copyright © 2016 The Apache Software Foundation. All Rights Reserved - bbe71fa-20161201-1641