Task

July 25, 2019 13:39

Specification

Magpie, Scala, Python, and R Script Tasks

{
  "name": String,
  "description": String,
  "state": String,
  "predecessorName": String,
  "failureMode": String,
  "maxRetries": Int,
  "timeout": String
}

SQL Mapper Task

{
  "name": String,
  "description": String,
  "state": String,
  "predecessorName": String,
  "targetTableName": String,
  "targetTableSpec": <table spec>,
  "partitionOptions": 
    {
      "count": Int
      "partitionBy": Array
    },
  "saveMode": String,
  "failureMode": String,
  "maxRetries": Int,
  "timeout": String
}

Nested Job Task

{
  "name": String,
  "description": String,
  "state": String,
  "predecessorName": String,
  "failureMode": String,
  "maxRetries": Int,
  "timeout": String
}

Structure Values

Field Name	Type	Description	Required	Default	Magpie, Scala, Python, or R Script	SQL Mapper	Nested Job
name	String	Task name.	Required		X	X	X
description	String	Expanded task information.		None	X	X	X
state	String	Whether the task is enabled or disabled. Supported values: `Enabled`, `Disabled`.		`Enabled`	X	X	X
predecessorName	String	Name of task. Links task to another task. The completion of the predecessor task will trigger execution.		None	X	X	X
targetTableName	String	Table that will is created from the SQL query result.	targetTableName or targetTableSpec Required			X
targetTableSpec	`table` `object`	Specifications for the table that will is created from the SQL query result.	targetTableName or targetTableSpec Required			X
partitionOptions	`partition` object					X
dataSourceName	String	Data source that the table will be saved to. If not specified, the default data source will be used.		None		X
count	Int	The number of partitions to save the resulting table with. Default is to coalesce the result into the number of cores available on the cluster.		None		X
partitionBy	Array	A comma-separated list of column names to partition the resulting table by. Default is unpartitioned.		None		X
saveMode	String	Supported values: `ReplaceWithDelete`, `Replace`, `Create`. `Create` creates a new table, erroring if the table already exists. `Replace` either creates a new table if the table does not already exist, or replaces the existing version of the table if it does exist. `ReplaceWithDelete` is the same as replace, but it deletes the underlying data for the existing table after replacing.		`Create`		X
failureMode	String	Task behavior in the event of a failure. Supported value: `Halt, Retry`.		`Halt`	X	X	X
maxRetries	Int	Only applicable to `Retry` failureMode. The number of times to retry a task that fails before halting.		None	X	X	X
timeout	String	Time to wait for task completion before failing. i.e. `"90 seconds"`, `"5 minutes"`, `"1 hour"`		None	X	X	X

Example

{
  "name": "profile_table"
  "description": "Profile table f_transaction."
  "predecessorName": "create_table",
  "failureMode": "Retry",
  "maxRetries": 3,
  "timeout": "5 minutes"
}