HDFS Credentials

AWS Access Key Credentials

Specification

{
  "credentialType": "AwsAccessKey",
  "accessKeyId": "<string>",
  "secretAccessKey": "<string>"
}

Structure values

Field Name	Type	Description	Required	Default
credentialType	String	The type of credential. AwsAccessKey for AWS Access Keys. Only for use with S3A sources.	Required
accessKeyId	String	The access key ID part of the keys.	Required
secretAccessKey	String	The secret access key part of the keys.	Required

Example

{
  "credentialType": "AwsAccessKey",
  "accessKeyId": " AKIAIOSFODNN7EXAMPLE",
  "secretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

---

AWS Assumed Role Credentials

Specification

{
  "credentialType": "AwsAssumedRole",
  "roleArn": "<string>",
  "roleSessionName": "<string>",
  "duration": <integer>,
  "crossAccount": <boolean>
}

Structure values

Field Name	Type	Description	Required	Default
credentialType	String	The type of credential. AwsAssumedRole for AWS Assumed Roles. Only for use with S3A sources configured in Magpie that is instantiated in the AWS Cloud environment. Current AWS role must include permissions for invoking STS:AssumeRole.	Required
roleArn	String	The arn of the role you will be assuming.	Required
roleSessionName	String	The name of the session in aws.		None
duration	integer	The duration of the expected session in seconds.		3600
crossAccount	boolean	Set to true if the role to be assumed is in a different account than the role you are currently using.		false

Example

{
  "credentialType": "AwsAssumedRole",
  "roleArn": "arn:aws:iam::123456789:role/aws-service-role",
  "roleSessionName": "my_session",
  "duration": 3600,
  "crossAccount": true
}

---

Azure Service Principal Credentials

Specification

{
  "credentialType": "AzureServicePrincipal",
  "tenantId": "<string>",
  "clientId": "<string>",
  "clientSecret": "<string>"
}

Structure values

Field Name	Type	Description	Required	Default
credentialType	String	The type of credential. AzureServicePrincipal for authenticating using an Azure AD service principal. Available for use with abfss type sources.	Required
tenantId	String	The tenant (directory) ID that target resource resides in.	Required
clientId	String	The client (application) ID of the Azure AD service principal. By default, Magpie uses the cluster’s default client ID and secret.	Optional	Magpie cluster client ID
clientSecret	String	The client secret of the Azure AD service principal. By default, Magpie uses the cluster’s default client ID and secret.	Optional	Magpie cluster client secret

Example

{
  "credentialType": "AzureServicePrincipal",
  "tenantId": "fb74c09d-d7de-4f68-b826-35bc80cc288f"
}

---

Azure Shared Access Signature Token Credentials

Specification

{
  "credentialType": "AzureSasToken",
  "token": "<string>"
}

Structure values

Field Name	Type	Description	Required	Default
credentialType	String	The type of credential. AzureSasToken for authenticating using a shared access signature token. Available for use with abfss and wasbs type sources.	Required
token	String	A generated SAS token for the storage account or container.	Required

Example

{
  "credentialType": "AzureSasToken",
  "token": "?sv=2018-03-28&sr=c&si=myPolicyPS&sig=NAxefF%2BrR2ubjZtyUtuAvLQgt%2FJIN5aHJMj6OsDwyy4%3D"
}

---

Azure Shared Key Credentials

Specification

{
  "credentialType": "AzureSharedKey",
  "key": "<string>"
}

Structure values

Field Name	Type	Description	Required	Default
credentialType	String	The type of credential. AzureSharedKey for authenticating using a storage account access key. Available for use with abfss and wasbs type sources.	Required
key	String	The access key for the storage account.	Required

Example

{
  "credentialType": "AzureSharedKey",
  "key": "lLrCcuiWVsLCoEXAMPLEBjs/ivkLEEXAMPLESU3Nu27ybphEXAMPLEmwjMwMHoHxEXAMPLE/UhLowWQ=="
}