Create Table From Kafka

Creates a Magpie table that references a topic or set of topics on a Kafka data source.

Syntax

create [temp | temporary] table { <table spec> | <table name> } [in schema <schema reference>]
  from kafka data source <data source reference>
  { topic "<topic name>" | topics "<topic list>" | topic pattern "<topic pattern>" | topic partitions <topic partitions spec> }
  [with key type <key type>]
  [with value type <value type>]
  [with starting offsets <starting offsets spec>]
  [with ending offsets <ending offsets spec>]

Parameters

table spec

JSON. A specification for saving the table. Note that persistence mapping and fields are not used when creating a table from Kafka.

table name

String. The name of the table to save the data as.

schema reference

String. The name of the schema to create the table in. Defaults to the current schema.

data source reference

String. The name of the kafka data source to reference.

topic name

String. The name of a single topic to load as a table.

topic list

String. A comma separated list of topics to load as a single table.

topic pattern

String. A java-style regular expression to use to search for topics to load.

topic partitions spec

JSON. A specification of specific topic partitions to load. For example: {"topicA":[0,1],"topicB":[2,4]}.

key type

String. The data type of keys in the loaded topic or topics. Must be either string or binary. Default is string.

value type

String. The data type of message values in the loaded topic or topics. Must be either string or binary. Default is string.

starting offsets spec

JSON. The start point for loading messages from the configured topic or topics. Must be either "earliest" or a specification or which offsets to use by topic partition, for example: {"topicA":{"0":23,"1":-2},"topicB":{"0":-2}}. In the JSON, -2 refers to earliest. Default is "earliest".

ending offsets spec

JSON. The end point for loading messages from the configured topic or topics. Must be either "latest" or a specification or which offsets to use by topic partition, for example: {"topicA":{"0":23,"1":-1},"topicB":{"0":-1}}. In the JSON, -1 refers to latest. Default is "latest".


Was this article helpful?
0 out of 0 found this helpful