Starting and Stopping Clusters

Starting

After creating a cluster, it must be started and set as your current cluster in order to use it.

start cluster main;
use cluster main;

After starting and using the cluster, you are able to load data, run queries, run scripts, and perform any of the other runtime actions that Magpie supports.

Tracking Usage

Once a cluster is started, you will be billed credits for its usage based on the terms of your Magpie subscription agreement.

To see how many cluster credits your organization has used, use the Cluster Usage command. This command will show credit usage by the time period you specify, and optionally filtered to a particular cluster or time range. By default, it shows total usage for the organization.

Viewing History

Magpie stores history of cluster runs. Each cluster run is the period of time between when a cluster was started and eventually stopped. Magpie stores log messages emitted from the cluster as well as health events and other information about the success or failure of the cluster run. To view this information, use the Cluster History command. History can optionally be filtered to a particular cluster. By default, history for all clusters is shown.

Stopping

When you are finished using a cluster, if a cluster definition needs to be altered, or if a cluster has entered an unhealthy state, the Stop Cluster command can be used to terminate the cluster.

When a cluster is stopped, any information stored locally on the cluster is destroyed and is unrecoverable. This includes anything saved to the local filesystem by custom user Scala, Python, and R scripts. When the cluster is started again, it will use an entirely new set of instances.

For example, to stop the cluster main:

use cluster none;
stop cluster main;

Before stopping a cluster, you must stop using it by either using another cluster or opting to use no cluster (use cluster none). Magpie will not let you stop your current cluster. If you decide to use no cluster, you are still able to execute metadata-only commands such as describe, list, and stop cluster. However, in order to perform analysis or data processing again, you must first select a new cluster to use.

Magpie will not allow you to change your current cluster in the middle of a Scala, Python, or R script. Executing a use cluster command inside a script will result in an error.

Was this article helpful?
0 out of 0 found this helpful