Task

Specification

Magpie, Scala, and Python Script Tasks

{
  "name": String,
  "description": String,
  "state": String,
  "predecessorName": String,
  "failureMode": String,
  "maxRetries": Int,
  "timeout": String,
  "isFinalizer": Boolean
}

SQL Mapper Task

{
  "name": String,
  "description": String,
  "state": String,
  "predecessorName": String,
  "targetTableName": String,
  "targetTableSpec": <table spec>,
  "targetSchemaName": String,
  "partitionOptions": 
    {
      "count": Int
      "partitionBy": Array
    },
  "saveMode": String,
  "failureMode": String,
  "maxRetries": Int,
  "timeout": String,
  "isFinalizer": Boolean
}

Nested Job Task

{
  "name": String,
  "description": String,
  "state": String,
  "predecessorName": String,
  "failureMode": String,
  "maxRetries": Int,
  "timeout": String,
  "isFinalizer": Boolean
}

Structure values

Field Name	Type	Description	Required	Default	Magpie, Scala, or Python Script	SQL Mapper	Nested Job
name	String	Task name.	Required		X	X	X
description	String	Expanded task information.		None	X	X	X
state	String	Whether the task is enabled or disabled. Supported values: Enabled, Disabled.		Enabled	X	X	X
predecessorName	String	Name of task. Links task to another task. The completion of the predecessor task will trigger execution.		None	X	X	X
targetTableName	String	Table that will is created from the SQL query result.	targetTableName or targetTableSpec Required			X
targetTableSpec	tableobject	Specifications for the table that will is created from the SQL query result.	targetTableName or targetTableSpec Required			X
targetSchemaName	String	Schema to which the table created by the SQL query result should belong. Default is to use the active session schema.		None		X
partitionOptions	partition object					X
dataSourceName	String	Data source that the table will be saved to. If not specified, the default data source will be used.		None		X
count	Int	The number of partitions to save the resulting table with. Default is to coalesce the result into the number of cores available on the cluster.		None		X
partitionBy	Array	A comma-separated list of column names to partition the resulting table by. Default is unpartitioned.		None		X
saveMode	String	Supported values: ReplaceWithDelete, Replace, Create. Create creates a new table, erroring if the table already exists. Replace either creates a new table if the table does not already exist, or replaces the existing version of the table if it does exist. ReplaceWithDelete is the same as replace, but it deletes the underlying data for the existing table after replacing.		Create		X
failureMode	String	Task behavior in the event of a failure. Supported value: Halt, Retry.		Halt	X	X	X
maxRetries	Int	Only applicable to Retry failureMode. The number of times to retry a task that fails before halting.		None	X	X	X
timeout	String	Time to wait for task completion before failing. i.e. "90 seconds", "5 minutes", "1 hour"		None	X	X	X
isFinalizer	Boolean	If present and true, indicates that the task is a Finalizer. Each job may only have one finalizer task, which will run at the end of the job and may not have any predecessors or successors.		None	X	X	X

Example

{
  "name": "profile_table"
  "description": "Profile table f_transaction."
  "predecessorName": "create_table",
  "failureMode": "Retry",
  "maxRetries": 3,
  "timeout": "5 minutes"
}