Create Table From Kafka

September 13, 2018 20:59

Creates a Magpie table that references a topic or set of topics on a Kafka data source.

Syntax

create [temp | temporary] table { <table spec> | <table name> } [in schema <schema reference>]
  from kafka data source <data source reference>
  { topic "<topic name>" | topics "<topic list>" | topic pattern "<topic pattern>" | topic partitions <topic partitions spec> }
  [with key type <key type>]
  [with value type <value type>]
  [with starting offsets <starting offsets spec>]
  [with ending offsets <ending offsets spec>]

Parameters

`table spec`

JSON. A specification for saving the table. Note that persistence mapping and fields are not used when creating a table from Kafka.

`table name`

String. The name of the table to save the data as.

`schema reference`

String. The name of the schema to create the table in. Defaults to the current schema.

`data source reference`

String. The name of the kafka data source to reference.

`topic name`

String. The name of a single topic to load as a table.

`topic list`

String. A comma separated list of topics to load as a single table.

`topic pattern`

String. A java-style regular expression to use to search for topics to load.

`topic partitions spec`

JSON. A specification of specific topic partitions to load. For example: {"topicA":[0,1],"topicB":[2,4]}.

`key type`

String. The data type of keys in the loaded topic or topics. Must be either string or binary. Default is string.

`value type`

String. The data type of message values in the loaded topic or topics. Must be either string or binary. Default is string.

`starting offsets spec`

JSON. The start point for loading messages from the configured topic or topics. Must be either "earliest" or a specification or which offsets to use by topic partition, for example: {"topicA":{"0":23,"1":-2},"topicB":{"0":-2}}. In the JSON, -2 refers to earliest. Default is "earliest".

`ending offsets spec`

JSON. The end point for loading messages from the configured topic or topics. Must be either "latest" or a specification or which offsets to use by topic partition, for example: {"topicA":{"0":23,"1":-1},"topicB":{"0":-1}}. In the JSON, -1 refers to latest. Default is "latest".