Kafka Subscription

June 02, 2020 21:33

A Kafka subscription is used in Kafka Mappings to define the data that is included in a Kafka-backed Magpie table or stream. There are many types of Kafka subscriptions with different required fields. The type of subscription is defined by the subscriptionType field.

Topic Subscription

A subscription to a single Kafka topic.

Specification

{
  "subscriptionType": "Topic",
  "topic": "<string>"
}

Structure Values

Field Name	Type	Description	Required	Default
topic	String	Name of the topic to subscribe to.	Required
subscriptionType	String	Subscription type. `Topic` for topic subscriptions.	Required

Example

{
  "topic": "my_topic",
  "subscriptionType": "Topic"
}

Topics Subscription

A subscription to multiple Kafka topics.

Specification

{
  "subscriptionType": "Topics",
  "topics": [
    "<string>"
  ]
}

Structure Values

Field Name	Type	Description	Required	Default
topics	Array of String	Names of topics to subscribe to.	Required
subscriptionType	String	Subscription type. `Topics` for topics subscriptions.	Required

Example

{
  "topics": ["topic1", "topic2"],
  "subscriptionType": "Topics"
}

Pattern Subscription

A subscription to multiple Kafka topics defined by a regular expression.

Specification

{
  "subscriptionType": "Pattern",
  "pattern": "<string>"
}

Structure Values

Field Name	Type	Description	Required	Default
pattern	String	A Java-style regular expression to use to select topics to subscribe to.	Required
subscriptionType	String	Subscription type. `Pattern` for pattern subscriptions.	Required

Example

{
  "pattern": "top.*",
  "subscriptionType": "Pattern"
}

Topic Partition Subscription

A subscription to specific partitions in a set of topics.

Specification

{
  "subscriptionType": "TopicPartition",
  "partitions": {
    "<string>": [<integer>]
  }
}

Structure Values

Field Name	Type	Description	Required	Default
partitions	Map from string to array of integer	An object that defines a set of partitions to load for a set of topics. Topic names are mapped to an array of partition indices for each topic.	Required
subscriptionType	String	Subscription type. `TopicPartition` for topic partition subscriptions.	Required

Example

{
  "partitions": {
    "topicA": [0,1],
    "topicB": [2,4]
  },
  "subscriptionType": "TopicPartition"
}