Skip to main content

Kafka Offsets

Kafka offsets are used in Kafka Mappings to specify when to start and stop querying data from a Kafka cluster for a Magpie table or stream.

Earliest​

Query from the earliest records in the topic. This type of offset is specified as the string earliest.

Specification​

"earliest"

Latest​

Query to the latest records in the topic. This type of offset is specified as the string latest.

Specification​

"latest"

By Topic Partition​

Specify offsets for each topic partition.

Topic Offset Specification​

{
  "<topic name>": <topic partition offset spec>
}

Structure values​

Field NameTypeDescriptionRequiredDefault
Topic partition offset object (defined below)The offsets for partitions in this topic. This field can be repeated multiple times to define offsets for multiple topics.Required

Topic Partition Offset Specification​

{
  "<partition>": <integer>
}

Structure values​

Field NameTypeDescriptionRequiredDefault
IntegerThe offset for this partition. A value of -2 specifies an earliest offset for this partition. A value of -1 specifies a latest offset for this partition. Non-negative values specify a specific offset for this partition. This field can be repeated multiple times to define offsets for multiple topic partitionsRequired

Example​

This example specifies offsets of 23 for topicA partition 0, latest for topicA partition 1, and earliest for topicB partition 0.

{
  "topicA": {
    "0": 23,
    "1": -1
  },
  "topicB": {
    "0": -2
  }
}