Kafka offsets are used in Kafka Mappings to specify when to start and stop querying data from a Kafka cluster for a Magpie table or stream.
Earliest
Query from the earliest records in the topic. This type of offset is specified as the string earliest.
Specification
"earliest"
Latest
Query to the latest records in the topic. This type of offset is specified as the string latest.
Specification
"latest"
By Topic Partition
Specify offsets for each topic partition.
Topic Offset Specification
{
"<topic name>": <topic partition offset spec>
}
Structure Values
Field Name | Type | Description | Required | Default |
|---|---|---|---|---|
<topic name> | Topic partition offset object (defined below) | The offsets for partitions in this topic. This field can be repeated multiple times to define offsets for multiple topics. | Required |
Topic Partition Offset Specification
{
"<partition>": <integer>
}
Structure Values
Field Name | Type | Description | Required | Default |
|---|---|---|---|---|
<partition> | Integer | The offset for this partition. A value of This field can be repeated multiple times to define offsets for multiple topic partitions | Required |
Example
This example specifies offsets of 23 for topicA partition 0, latest for topicA partition 1, and earliest for topicB partition 0.
{
"topicA": {
"0": 23,
"1": -1
},
"topicB": {
"0": -2
}
}