Object

kafka.tools

MirrorMaker

Related Doc: package tools

Permalink

object MirrorMaker extends Logging with KafkaMetricsGroup

The mirror maker has the following architecture: - There are N mirror maker thread shares one ZookeeperConsumerConnector and each owns a Kafka stream. - All the mirror maker threads share one producer. - Each mirror maker thread periodically flushes the producer and then commits all offsets.

Note

For mirror maker, the following settings are set by default to make sure there is no data loss:

  1. use new producer with following settings acks=all retries=max integer block.on.buffer.full=true max.in.flight.requests.per.connection=1 2. Consumer Settings auto.commit.enable=false 3. Mirror Maker Setting: abort.on.send.failure=true
Linear Supertypes
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. MirrorMaker
  2. KafkaMetricsGroup
  3. Logging
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Visibility
  1. Public
  2. All

Type Members

  1. trait MirrorMakerMessageHandler extends AnyRef

    Permalink

    If message.handler.args is specified.

    If message.handler.args is specified. A constructor that takes in a String as argument must exist.

  2. class MirrorMakerThread extends Thread with Logging with KafkaMetricsGroup

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def cleanShutdown(): Unit

    Permalink
  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. def commitOffsets(mirrorMakerConsumer: MirrorMakerBaseConsumer): Unit

    Permalink
  8. def createNewConsumers(numStreams: Int, consumerConfigPath: String, customRebalanceListener: Option[ConsumerRebalanceListener], whitelist: Option[String]): Seq[MirrorMakerBaseConsumer]

    Permalink
  9. def debug(msg: ⇒ String, e: ⇒ Throwable): Unit

    Permalink
    Definition Classes
    Logging
  10. def debug(e: ⇒ Throwable): Any

    Permalink
    Definition Classes
    Logging
  11. def debug(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  12. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  13. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  14. def error(msg: ⇒ String, e: ⇒ Throwable): Unit

    Permalink
    Definition Classes
    Logging
  15. def error(e: ⇒ Throwable): Any

    Permalink
    Definition Classes
    Logging
  16. def error(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  17. def fatal(msg: ⇒ String, e: ⇒ Throwable): Unit

    Permalink
    Definition Classes
    Logging
  18. def fatal(e: ⇒ Throwable): Any

    Permalink
    Definition Classes
    Logging
  19. def fatal(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  20. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  21. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  22. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  23. def info(msg: ⇒ String, e: ⇒ Throwable): Unit

    Permalink
    Definition Classes
    Logging
  24. def info(e: ⇒ Throwable): Any

    Permalink
    Definition Classes
    Logging
  25. def info(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  26. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  27. var logIdent: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  28. lazy val logger: Logger

    Permalink
    Definition Classes
    Logging
  29. val loggerName: String

    Permalink
    Definition Classes
    Logging
  30. def main(args: Array[String]): Unit

    Permalink
  31. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  32. def newGauge[T](name: String, metric: Gauge[T], tags: Map[String, String] = Map.empty): Gauge[T]

    Permalink
    Definition Classes
    KafkaMetricsGroup
  33. def newHistogram(name: String, biased: Boolean = true, tags: Map[String, String] = Map.empty): Histogram

    Permalink
    Definition Classes
    KafkaMetricsGroup
  34. def newMeter(name: String, eventType: String, timeUnit: TimeUnit, tags: Map[String, String] = Map.empty): Meter

    Permalink
    Definition Classes
    KafkaMetricsGroup
  35. def newTimer(name: String, durationUnit: TimeUnit, rateUnit: TimeUnit, tags: Map[String, String] = Map.empty): Timer

    Permalink
    Definition Classes
    KafkaMetricsGroup
  36. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  37. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  38. def removeMetric(name: String, tags: Map[String, String] = Map.empty): Unit

    Permalink
    Definition Classes
    KafkaMetricsGroup
  39. def swallow(action: ⇒ Unit): Unit

    Permalink
    Definition Classes
    Logging
  40. def swallowDebug(action: ⇒ Unit): Unit

    Permalink
    Definition Classes
    Logging
  41. def swallowError(action: ⇒ Unit): Unit

    Permalink
    Definition Classes
    Logging
  42. def swallowInfo(action: ⇒ Unit): Unit

    Permalink
    Definition Classes
    Logging
  43. def swallowTrace(action: ⇒ Unit): Unit

    Permalink
    Definition Classes
    Logging
  44. def swallowWarn(action: ⇒ Unit): Unit

    Permalink
    Definition Classes
    Logging
  45. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  46. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  47. def trace(msg: ⇒ String, e: ⇒ Throwable): Unit

    Permalink
    Definition Classes
    Logging
  48. def trace(e: ⇒ Throwable): Any

    Permalink
    Definition Classes
    Logging
  49. def trace(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  50. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  51. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  52. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  53. def warn(msg: ⇒ String, e: ⇒ Throwable): Unit

    Permalink
    Definition Classes
    Logging
  54. def warn(e: ⇒ Throwable): Any

    Permalink
    Definition Classes
    Logging
  55. def warn(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging

Inherited from KafkaMetricsGroup

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped