Task

Specification

Magpie, Scala, Python, and R Script Tasks

{
  "name": String,
  "description": String,
  "state": String,
  "predecessorName": String,
  "failureMode": String,
  "maxRetries": Int,
  "timeout": String
}

SQL Mapper Task

{
  "name": String,
  "description": String,
  "state": String,
  "predecessorName": String,
  "targetTableName": String,
  "targetTableSpec": <table spec>,
  "partitionOptions": 
    {
      "count": Int
      "partitionBy": Array
    },
  "createMode": String,
  "writeMode": String,
  "failureMode": String,
  "maxRetries": Int,
  "timeout": String
}

Nested Job Task

{
  "name": String,
  "description": String,
  "state": String,
  "predecessorName": String,
  "failureMode": String,
  "maxRetries": Int,
  "timeout": String
}

Structure Values

Field NameTypeDescriptionRequiredDefaultMagpie, Scala, Python, or R ScriptSQL MapperNested Job
nameStringTask name.Required
XXX
descriptionStringExpanded task information.
NoneXXX
stateStringWhether the task is enabled or disabled. Supported values: Enabled, Disabled.
EnabledXXX
predecessorNameStringName of task. Links task to another task. The completion of the predecessor task will trigger execution.
NoneXXX
targetTableNameStringTable that will is created from the SQL query result.targetTableName or targetTableSpec Required

X
targetTableSpectable objectSpecifications for the table that will is created from the SQL query result.targetTableName or targetTableSpec Required

X
partitionOptionspartition object



X
countIntThe number of partitions to save the resulting table with. Default is to coalesce the result into the number of cores available on the cluster.
None
X
partitionByArrayA comma-separated list of column names to partition the resulting table by. Default is unpartitioned.
None
X
createModeString

Supported values: ReplaceWithDelete, Replace, Create, UseExisting.

UseExisting requires that the target table already exist and writes to that table.

Create creates a new table, erroring if the table already exists.

Replace either creates a new table if the table does not already exist, or replaces the existing version of the table if it does exist.

ReplaceWithDelete is the same as replace, but it deletes the underlying data for the existing table after replacing.

Required

X
writeModeStringOnly applicable to UseExisting createMode. Supported value: Overwrite.
None
X
failureModeStringTask behavior in the event of a failure. Supported value: Halt, Retry.
HaltXXX
maxRetriesIntOnly applicable to Retry failureMode. The number of times to retry a task that fails before halting.
NoneXX

X

timeoutString

Time to wait for task completion before failing.
i.e. "90 seconds", "5 minutes", "1 hour"


NoneXXX

Example

{
  "name": "profile_table"
  "description": "Profile table f_transaction."
  "predecessorName": "create_table",
  "failureMode": "Retry",
  "maxRetries": 3,
  "timeout": "5 minutes"
}
Was this article helpful?
0 out of 0 found this helpful