Cluster

Specification

{
  "name": "<string>",
  "description": "<string>",
  "driverType": "<string>",
  "workerCount": <integer>,
  "workerType": "<string>",
  "executorCores": <integer>,
  "storageVolumeSize": "<string>",
  "bootstrapScript": "<string>"
}

Structure Values

Field NameTypeDescriptionRequiredDefault
nameStringCluster name.Required
descriptionStringExpanded cluster information.OptionalNone
driverTypeString

The type of instance to use for the cluster driver. Supported values can be found by running the command list instance types in Magpie. Example values are m5.4xlarge, n2-standard-16, or Standard_D16s_v4


Required
workerCountIntegerHow many worker nodes to provision for the cluster. Omit or set to zero for a single-node (driver only) cluster.Optional0
workerTypeStringThe type of instance to use for the cluster workers. See driverType for supported values.OptionaldriverType
executorCoresIntegerHow many cores to use per Spark executor.Optional4
storageVolumeSizeStringHow much local storage to provision for each cluster node. Takes the form of a string with units, such as "500 GiB"Optional100 GiB
bootstrapScriptStringBash script lines to execute at cluster startup.OptionalNone

Example

{
  "name": "etl_cluster",
  "description": "Cluster to use for nightly ETL runs",
  "driverType": "m5.2xlarge",
  "workerCount": 2,
  "workerType": "m5.4xlarge",
  "executorCores": 4,
  "storageVolumeSize": "500 GiB",
  "bootstrapScript": "pip install some-package"
}
Was this article helpful?
0 out of 0 found this helpful