kafka.log

OffsetIndex

class OffsetIndex extends Logging

An index that maps offsets to physical file locations for a particular log segment. This index may be sparse: that is it may not hold an entry for all messages in the log.

The index is stored in a file that is pre-allocated to hold a fixed maximum number of 8-byte entries.

The index supports lookups against a memory-map of this file. These lookups are done using a simple binary search variant to locate the offset/location pair for the greatest offset less than or equal to the target offset.

Index files can be opened in two ways: either as an empty, mutable index that allows appends or an immutable read-only index file that has previously been populated. The makeReadOnly method will turn a mutable file into an immutable one and truncate off any extra bytes. This is done when the index file is rolled over.

No attempt is made to checksum the contents of this file, in the event of a crash it is rebuilt.

The file format is a series of entries. The physical format is a 4 byte "relative" offset and a 4 byte file location for the message with that offset. The offset stored is relative to the base offset of the index file. So, for example, if the base offset was 50, then the offset 55 would be stored as 5. Using relative offsets in this way let's us use only 4 bytes for the offset.

The frequency of entries is up to the user of this class.

All external APIs translate from relative offsets to full offsets, so users of this class do not interact with the internal storage format.

go to: companion
Inherited
  1. Hide All
  2. Show all
  1. Logging
  2. AnyRef
  3. Any
Visibility
  1. Public
  2. All

Instance constructors

  1. new OffsetIndex(file: File, baseOffset: Long, maxIndexSize: Int = -1)

Value Members

  1. def !=(arg0: AnyRef): Boolean

    attributes: final
    definition classes: AnyRef
  2. def !=(arg0: Any): Boolean

    o != arg0 is the same as !(o == (arg0)).

    o != arg0 is the same as !(o == (arg0)).

    arg0

    the object to compare against this object for dis-equality.

    returns

    false if the receiver object is equivalent to the argument; true otherwise.

    attributes: final
    definition classes: Any
  3. def ##(): Int

    attributes: final
    definition classes: AnyRef → Any
  4. def $asInstanceOf[T0](): T0

    attributes: final
    definition classes: AnyRef
  5. def $isInstanceOf[T0](): Boolean

    attributes: final
    definition classes: AnyRef
  6. def ==(arg0: AnyRef): Boolean

    o == arg0 is the same as if (o eq null) arg0 eq null else o.equals(arg0).

    o == arg0 is the same as if (o eq null) arg0 eq null else o.equals(arg0).

    arg0

    the object to compare against this object for equality.

    returns

    true if the receiver object is equivalent to the argument; false otherwise.

    attributes: final
    definition classes: AnyRef
  7. def ==(arg0: Any): Boolean

    o == arg0 is the same as o.equals(arg0).

    o == arg0 is the same as o.equals(arg0).

    arg0

    the object to compare against this object for equality.

    returns

    true if the receiver object is equivalent to the argument; false otherwise.

    attributes: final
    definition classes: Any
  8. def append(offset: Long, position: Int): Unit

    Append an entry for the given offset/location pair to the index.

    Append an entry for the given offset/location pair to the index. This entry must have a larger offset than all subsequent entries.

  9. def asInstanceOf[T0]: T0

    This method is used to cast the receiver object to be of type T0.

    This method is used to cast the receiver object to be of type T0.

    Note that the success of a cast at runtime is modulo Scala's erasure semantics. Therefore the expression1.asInstanceOf[String] will throw a ClassCastException at runtime, while the expressionList(1).asInstanceOf[List[String]] will not. In the latter example, because the type argument is erased as part of compilation it is not possible to check whether the contents of the list are of the requested typed.

    returns

    the receiver object.

    attributes: final
    definition classes: Any
  10. val baseOffset: Long

  11. def clone(): AnyRef

    This method creates and returns a copy of the receiver object.

    This method creates and returns a copy of the receiver object.

    The default implementation of the clone method is platform dependent.

    returns

    a copy of the receiver object.

    attributes: protected
    definition classes: AnyRef
  12. def close(): Unit

    Close the index

    Close the index

  13. def debug(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  14. def debug(e: ⇒ Throwable): Any

    definition classes: Logging
  15. def debug(msg: ⇒ String): Unit

    definition classes: Logging
  16. def delete(): Boolean

    Delete this index file

    Delete this index file

  17. def entries(): Int

    The number of entries in this index

    The number of entries in this index

  18. def entry(n: Int): OffsetPosition

    Get the nth offset mapping from the index

    Get the nth offset mapping from the index

    n

    The entry number in the index

    returns

    The offset/position pair at that entry

  19. def eq(arg0: AnyRef): Boolean

    This method is used to test whether the argument (arg0) is a reference to the receiver object (this).

    This method is used to test whether the argument (arg0) is a reference to the receiver object (this).

    The eq method implements an [http://en.wikipedia.org/wiki/Equivalence_relation equivalence relation] on non-null instances of AnyRef: * It is reflexive: for any non-null instance x of type AnyRef, x.eq(x) returns true. * It is symmetric: for any non-null instances x and y of type AnyRef, x.eq(y) returns true if and only if y.eq(x) returns true. * It is transitive: for any non-null instances x, y, and z of type AnyRef if x.eq(y) returns true and y.eq(z) returns true, then x.eq(z) returns true.

    Additionally, the eq method has three other properties. * It is consistent: for any non-null instances x and y of type AnyRef, multiple invocations of x.eq(y) consistently returns true or consistently returns false. * For any non-null instance x of type AnyRef, x.eq(null) and null.eq(x) returns false. * null.eq(null) returns true.

    When overriding the equals or hashCode methods, it is important to ensure that their behavior is consistent with reference equality. Therefore, if two objects are references to each other (o1 eq o2), they should be equal to each other (o1 == o2) and they should hash to the same value (o1.hashCode == o2.hashCode).

    arg0

    the object to compare against this object for reference equality.

    returns

    true if the argument is a reference to the receiver object; false otherwise.

    attributes: final
    definition classes: AnyRef
  20. def equals(arg0: Any): Boolean

    This method is used to compare the receiver object (this) with the argument object (arg0) for equivalence.

    This method is used to compare the receiver object (this) with the argument object (arg0) for equivalence.

    The default implementations of this method is an [http://en.wikipedia.org/wiki/Equivalence_relation equivalence relation]: * It is reflexive: for any instance x of type Any, x.equals(x) should return true. * It is symmetric: for any instances x and y of type Any, x.equals(y) should return true if and only if y.equals(x) returns true. * It is transitive: for any instances x, y, and z of type AnyRef if x.equals(y) returns true and y.equals(z) returns true, then x.equals(z) should return true.

    If you override this method, you should verify that your implementation remains an equivalence relation. Additionally, when overriding this method it is often necessary to override hashCode to ensure that objects that are "equal" (o1.equals(o2) returns true) hash to the same scala.Int (o1.hashCode.equals(o2.hashCode)).

    arg0

    the object to compare against this object for equality.

    returns

    true if the receiver object is equivalent to the argument; false otherwise.

    definition classes: AnyRef → Any
  21. def error(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  22. def error(e: ⇒ Throwable): Any

    definition classes: Logging
  23. def error(msg: ⇒ String): Unit

    definition classes: Logging
  24. def fatal(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  25. def fatal(e: ⇒ Throwable): Any

    definition classes: Logging
  26. def fatal(msg: ⇒ String): Unit

    definition classes: Logging
  27. var file: File

  28. def finalize(): Unit

    This method is called by the garbage collector on the receiver object when garbage collection determines that there are no more references to the object.

    This method is called by the garbage collector on the receiver object when garbage collection determines that there are no more references to the object.

    The details of when and if the finalize method are invoked, as well as the interaction between finalizeand non-local returns and exceptions, are all platform dependent.

    attributes: protected
    definition classes: AnyRef
  29. def flush(): Unit

    Flush the data in the index to disk

    Flush the data in the index to disk

  30. def getClass(): java.lang.Class[_]

    Returns a representation that corresponds to the dynamic class of the receiver object.

    Returns a representation that corresponds to the dynamic class of the receiver object.

    The nature of the representation is platform dependent.

    returns

    a representation that corresponds to the dynamic class of the receiver object.

    attributes: final
    definition classes: AnyRef
  31. def hashCode(): Int

    Returns a hash code value for the object.

    Returns a hash code value for the object.

    The default hashing algorithm is platform dependent.

    Note that it is allowed for two objects to have identical hash codes (o1.hashCode.equals(o2.hashCode)) yet not be equal (o1.equals(o2) returns false). A degenerate implementation could always return 0. However, it is required that if two objects are equal (o1.equals(o2) returns true) that they have identical hash codes (o1.hashCode.equals(o2.hashCode)). Therefore, when overriding this method, be sure to verify that the behavior is consistent with the equals method.

    returns

    the hash code value for the object.

    definition classes: AnyRef → Any
  32. def info(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  33. def info(e: ⇒ Throwable): Any

    definition classes: Logging
  34. def info(msg: ⇒ String): Unit

    definition classes: Logging
  35. def isFull: Boolean

    True iff there are no more slots available in this index

    True iff there are no more slots available in this index

  36. def isInstanceOf[T0]: Boolean

    This method is used to test whether the dynamic type of the receiver object is T0.

    This method is used to test whether the dynamic type of the receiver object is T0.

    Note that the test result of the test is modulo Scala's erasure semantics. Therefore the expression1.isInstanceOf[String] will return false, while the expression List(1).isInstanceOf[List[String]] will return true. In the latter example, because the type argument is erased as part of compilation it is not possible to check whether the contents of the list are of the requested typed.

    returns

    true if the receiver object is an instance of erasure of type T0; false otherwise.

    attributes: final
    definition classes: Any
  37. var lastOffset: Long

  38. var logIdent: String

    attributes: protected
    definition classes: Logging
  39. lazy val logger: Logger

    definition classes: Logging
  40. val loggerName: String

    definition classes: Logging
  41. def lookup(targetOffset: Long): OffsetPosition

    Find the largest offset less than or equal to the given targetOffset and return a pair holding this offset and it's corresponding physical file position.

    Find the largest offset less than or equal to the given targetOffset and return a pair holding this offset and it's corresponding physical file position.

    targetOffset

    The offset to look up.

    returns

    The offset found and the corresponding file position for this offset. If the target offset is smaller than the least entry in the index (or the index is empty), the pair (baseOffset, 0) is returned.

  42. var maxEntries: Int

    The maximum number of eight-byte entries this index can hold

    The maximum number of eight-byte entries this index can hold

  43. val maxIndexSize: Int

  44. def ne(arg0: AnyRef): Boolean

    o.ne(arg0) is the same as !(o.eq(arg0)).

    o.ne(arg0) is the same as !(o.eq(arg0)).

    arg0

    the object to compare against this object for reference dis-equality.

    returns

    false if the argument is not a reference to the receiver object; true otherwise.

    attributes: final
    definition classes: AnyRef
  45. def notify(): Unit

    Wakes up a single thread that is waiting on the receiver object's monitor.

    Wakes up a single thread that is waiting on the receiver object's monitor.

    attributes: final
    definition classes: AnyRef
  46. def notifyAll(): Unit

    Wakes up all threads that are waiting on the receiver object's monitor.

    Wakes up all threads that are waiting on the receiver object's monitor.

    attributes: final
    definition classes: AnyRef
  47. def readLastEntry(): OffsetPosition

    The last entry in the index

    The last entry in the index

  48. def renameTo(f: File): Boolean

    Rename the file that backs this offset index

    Rename the file that backs this offset index

    returns

    true iff the rename was successful

  49. def resize(newSize: Int): Unit

    Reset the size of the memory map and the underneath file.

    Reset the size of the memory map and the underneath file. This is used in two kinds of cases: (1) in trimToValidSize() which is called at closing the segment or new segment being rolled; (2) at loading segments from disk or truncating back to an old segment where a new log segment became active; we want to reset the index size to maximum index size to avoid rolling new segment.

  50. def sanityCheck(): Unit

    Do a basic sanity check on this index to detect obvious problems

    Do a basic sanity check on this index to detect obvious problems

  51. def sizeInBytes(): Int

    The number of bytes actually used by this index

    The number of bytes actually used by this index

  52. def swallow(action: ⇒ Unit): Unit

    definition classes: Logging
  53. def swallowDebug(action: ⇒ Unit): Unit

    definition classes: Logging
  54. def swallowError(action: ⇒ Unit): Unit

    definition classes: Logging
  55. def swallowInfo(action: ⇒ Unit): Unit

    definition classes: Logging
  56. def swallowTrace(action: ⇒ Unit): Unit

    definition classes: Logging
  57. def swallowWarn(action: ⇒ Unit): Unit

    definition classes: Logging
  58. def synchronized[T0](arg0: T0): T0

    attributes: final
    definition classes: AnyRef
  59. def toString(): String

    Returns a string representation of the object.

    Returns a string representation of the object.

    The default representation is platform dependent.

    returns

    a string representation of the object.

    definition classes: AnyRef → Any
  60. def trace(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  61. def trace(e: ⇒ Throwable): Any

    definition classes: Logging
  62. def trace(msg: ⇒ String): Unit

    definition classes: Logging
  63. def trimToValidSize(): Unit

    Trim this segment to fit just the valid entries, deleting all trailing unwritten bytes from the file.

    Trim this segment to fit just the valid entries, deleting all trailing unwritten bytes from the file.

  64. def truncate(): Unit

    Truncate the entire index, deleting all entries

    Truncate the entire index, deleting all entries

  65. def truncateTo(offset: Long): Unit

    Remove all entries from the index which have an offset greater than or equal to the given offset.

    Remove all entries from the index which have an offset greater than or equal to the given offset. Truncating to an offset larger than the largest in the index has no effect.

  66. def wait(): Unit

    attributes: final
    definition classes: AnyRef
  67. def wait(arg0: Long, arg1: Int): Unit

    attributes: final
    definition classes: AnyRef
  68. def wait(arg0: Long): Unit

    attributes: final
    definition classes: AnyRef
  69. def warn(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  70. def warn(e: ⇒ Throwable): Any

    definition classes: Logging
  71. def warn(msg: ⇒ String): Unit

    definition classes: Logging