Skip to main content

Create Table From Kafka

Creates a Magpie table that references a topic or set of topics on a Kafka data source.

Syntax​

create [temp | temporary] table { <table spec> | <table name> } [in schema <schema reference>]
  from kafka data source <data source reference>
  { topic "<topic name>" | topics "<topic list>" | topic pattern "<topic pattern>" | topic partitions <topic partitions spec> }
  [with key type <key type>]
  [with value type <value type>]
  [with starting offsets <starting offsets spec>]
  [with ending offsets <ending offsets spec>]

Parameters​

table spec​

JSON — A specification for saving the table. Note that persistence mapping and fields are not used when creating a table from Kafka.

table name​

String — The name of the table to save the data as.

schema reference​

String — The name of the schema to create the table in. Defaults to the current schema.

data source reference​

String — The name of the kafka data source to reference.

topic name​

String — The name of a single topic to load as a table.

topic list​

String — A comma separated list of topics to load as a single table.

topic pattern​

String — A java-style regular expression to use to search for topics to load.

topic partitions spec​

JSON — A specification of specific topic partitions to load. For example: {"topicA":[0,1],"topicB":[2,4]}.

key type​

String — The data type of keys in the loaded topic or topics. Must be either string or binary. Default is string.

value type​

String — The data type of message values in the loaded topic or topics. Must be either string or binary. Default is string.

starting offsets spec​

JSON — The start point for loading messages from the configured topic or topics. Must be either "earliest" or a specification or which offsets to use by topic partition, for example: {"topicA":{"0":23,"1":-2},"topicB":{"0":-2}}. In the JSON, -2 refers to earliest. Default is "earliest".

ending offsets spec​

JSON — The end point for loading messages from the configured topic or topics. Must be either "latest" or a specification or which offsets to use by topic partition, for example: {"topicA":{"0":23,"1":-1},"topicB":{"0":-1}}. In the JSON, -1 refers to latest. Default is "latest".