1. Silectis
  2. Usage Guide
  3. What's New

Release Notes

Follow

Magpie 2.3.4

Version 2.3.4 is a minor release in the 2.x series. It was released on February 28th, 2025. Changes include:

  • Updated Job Description window to move the long form description to the second tab leaving with Execution Graph as first

Magpie 2.3.3

Version 2.3.3 is a minor release in the 2.x series. It was released on January 31th, 2025. Changes include:

  • Added SSO support for Microsoft Entra Identity Providers

  • Updated Job Description window to remove long form description and replace with Execution Graph

Magpie 2.3.2

Version 2.3.2 is a minor release in the 2.x series. It was released on January 24th, 2025. Changes include:

  • Bug fix to support displaying Private Keys in cluster description

Magpie 2.3.1

Version 2.3.1 is a minor release in the 2.x series. It was released on December 13th, 2024. Changes include:

  • Add support for Private Keys to store SSH Keys and GCP Keys

  • Upgrade Snowflake Spark Connector to version 2.16.0

  • Enhancement to allow creating tables from database sources as writable

Magpie 2.3.0

Version 2.3.0 is a minor release in the 2.x series. It was released on October 22nd, 2024. There are no user-facing changes.

Magpie 2.2.1

Version 2.2.1 is a patch release in the 2.x series. It was released on September 28th, 2024. Changes include:

  • Enhancement to allow setting materializationLocation in BigQuery Sources

Magpie 2.1.0

Version 2.1.0 is a minor release in the 2.x series. It was released on September 27th, 2024. Changes include:

Magpie 2.1.2

Version 2.1.2 is a patch release in the 2.x series. It was released on September 6th, 2024. Changes include:

  • Enhancement to disable `Forgot Password` link in the login page when in SSO-only mode

  • Enhancement to allow multiline strings for alter metadata commands

  • Enhancement to Spark 3.5 memory defaults to improve cluster stability under heavy workloads

  • Bug fix to support `Exists` SQL function

  • Bug fix to remove error that printed on CLI startup since version 2.x

  • Bug fix for BigQuery support for date types encoded as strings

Magpie 2.1.1

Version 2.1.1 is a patch release in the 2.x series. It was released on August 2nd, 2024. Changes include:

  • Add support for Spark SQL `PIVOT` and `UNPIVOT` keywords

  • Enhancement to Lookup Validator to support case-insensitive matching

  • Enhancement to Spark Configuration to improve compatibility with legacy datetime formats

  • Bug fix to Single Sign-on when the identity provider returns a mixed-case email address

  • Bug fix for copying files within the same ABFSS source

  • Bug fix for table profiles that failed to render properly if the metadata contained `{{`

Magpie 2.1.0

Version 2.1.0 is a minor release in the 2.x series. It was released on July 19th, 2024. Changes include:

  • Upgrade Spark to 3.5.1

  • Enhancement to Magpie SQL to allow VALUES keyword.

  • Enhancement to Magpie SQL to support CASTing subtypes.

  • Enhancement to Magpie SQL to support new operators: <>, %, div, |, ||, ~, &, ==, ^.

  • Bug fix to allow users to validate tables with SQL keywords as the table name.

  • Bug fix to set Spark SQL config legacy.timeParserPolicy to value LEGACY in order to maintain compatibility with Magpie 1.x releases.

  • Bug fix to ensure that org admins can generate and view access tokens without explicitly being granted permission.

Magpie 2.0.0

Version 2.0.0 is a major release in the 2.x series. It was released on May 10th, 2024. This release removes all previously deprecated features and upgrades several components of Magpie in preparation for the next Magpie release. Changes include:

  • Upgrade Scala Runtime to 2.12.10

  • Bug fix to ensure Access Token is displayed to organization admins in Zeppelin

  • Remove deprecated MagpieContext commands for getVariable, setVariable, hasVariable, clearVariable, clearVariables, printVariables, substituteVariables

  • Remove deprecated create schedule command

  • Remove deprecated alter user set password command

  • Remove deprecated create notification subscription command

  • Remove deprecated MagpieNotebookContext

  • Remove deprecated :cluster CLI command

  • Remove deprecated R runtime

Magpie 1.60.1

Version 1.60.1 is a minor release in the 1.x series. It was released on March 15th, 2024. Changes include:

  • Bug fix to SSO login redirect following successful authentication. Fixes regression introduced in 1.59.0

  • Bug fix to explicitly require organization.access_token permission to create personal access tokens. The ability to create an access token is no longer granted to organization admins

Magpie 1.60.0

Version 1.60.0 is a minor release in the 1.x series. It was released on March 8th, 2024. Changes include:

  • Deprecating support for running R scripts. We recommend using R studio and connecting to Magpie as a data source. R support will be fully removed in an upcoming Magpie release

  • Enhancement to user account management that allows accounts to be designated as service accounts. Service accounts can have their access tokens managed by Organization administrators.

  • Add new notifications for expiring personal access tokens. Users with expiring tokens will receive an email 30, 15, 7, 3 days prior to token expiration and another upon expiration. Contact your Silectis support representative to adjust these thresholds for your organization.

  • Enhancement to display a success message when logging into the CLI with single Sign-on

  • Enhancement to SSO to add a login_hint to the OIDC provider. This provides a better single sign-on experience for users logged into multiple accounts.

  • Bug fix to correct inaccurate error message when cluster start fails on Azure

Magpie 1.59.0

Version 1.59.0 was skipped.

Magpie 1.58.0

Version 1.58.0 is a minor release in the 1.x series. It was released on February 2nd, 2024. Changes include:

  • Support for personal access tokens when logging in via the CLI or a JDBC connector

  • Enhancement to Magpie job schedule email notifications to include the Magpie hostname in the email

  • Bug fix to correct error message displayed when starting an Azure-hosted Magpie cluster failed

  • Update MS SQL JDBC Driver to v12.4.0

Magpie 1.57.0

Version 1.57.0 was skipped.

Magpie 1.56.0

Version 1.56.0 is a minor release in the 1.x series. It was released on December 1st, 2023. Changes include:

  • Support for federated login with OpenID Connect. Contact your Silectis account team to enable this feature within your organization.

Magpie 1.55.3

Version 1.55.3 is a minor release in the 1.x series. It was released on September 22nd, 2023. Changes include:

  • Enhancement to alter metadata commands to allow editing individual fields using a alter <type> <reference> set <fieldName> <fieldValue> command

  • Bug fix for activity history commands that prevented output from displaying

  • Performance improvement to lineage calculations in describe table commands

Magpie 1.54.0

Version 1.54.0 is a minor release in the 1.x series. It was released on August 4th, 2023. Changes include:

  • Enhancement to notebooks that allow downloading files from object data sources (S3, Azure Blob Storage, GCS)

  • Bug fix to prevent validators from being created with invalid names

  • Bug fix to allow creating regex and in list validators containing single quotes (Single quotes must be escaped with a backslash)

Magpie 1.53.0

Version 1.53.0 was skipped.

Magpie 1.52.0

Version 1.52.0 is a minor release in the 1.x series. It was released on May 19th, 2023. Changes include:

  • Remove support for Azure Data Lake Gen 1 (Deprecated by Microsoft. Use Azure Data Lake Gen 2, instead)

  • Bug fix to cluster health monitoring that resulted in non-existant clusters being monitored

  • Improvement to error message when listing files in a data source with unsupported characters in filenames

Magpie 1.51.0

Version 1.51.0 is a minor release in the 1.x series. It was released on February 17th, 2023. Changes include:

  • Bug fix to only show warning if limit is reduced to hard max

  • Enhancement to SQL Parser to support aliasing Stack function

  • Enhancement to Regex and In List validators to optionally cast fields to strings prior to validation

Magpie 1.50.0

Version 1.50.0 is a minor release in the 1.x series. It was released on December 21st, 2022. There are no user-facing changes.

Magpie 1.49.0

Version 1.49.0 is a minor release in the 1.x series. It was released on December 9th, 2022. Changes include:

  • Bug fix to correct syntax for list notifications for job schedule command

  • Enhancement to data profiling display

Magpie 1.48.0

Version 1.48.0 is a minor release in the 1.x series. It was released on October 28th, 2022. Changes include:

  • Enhancement to null handling in Data Validators

  • Bug fix to cluster status reporting when running non-Magpie Spark applications

Magpie 1.47.0

Version 1.47.0 is a minor release in the 1.x series. It was released on September 30th, 2022. Changes include:

  • Support for SQL validator

  • Support for filtering Activity History commands by Cluster

  • Bug fix to cluster status reporting false unhealthy status in specific scenarios

Magpie 1.46.0

Version 1.46.0 is a minor release in the 1.x series. It was released on September 9th, 2022. Changes include:

  • Improves performance when querying small tables with BI tools connected via Magpie’s JDBC server.

Magpie 1.45.0

Version 1.45.0 is a minor release in the 1.x series. It was released on August 19th, 2022. Changes include:

  • Support for Date Range validator

  • Support for MySQL data sources

  • Adds reassign job command

  • Support for altering job schedules

Magpie 1.44.0

Version 1.44.0 is a minor release in the 1.x series. It was released on July 22nd, 2022. Changes include:

  • Support for start date and end date filters to execution history, job execution history, task execution history, cluster history, and usage history commands

  • Adds restart cluster command

  • Support for Lookup and In List Validators

  • Support for additional syntax options when specifying regex validators and field selectors

  • Adds new default R libraries: tictoc, here, and wrapr

  • Bug fix for drop project with cascade command partially succeeding

Magpie 1.43.0

Version 1.43.0 is a minor release in the 1.x series. It was released on June 10th, 2022. There are no user-facing changes.

Magpie 1.42.0

Version 1.42.0 was skipped.

Magpie 1.41.0

Version 1.41.0 is a minor release in the 1.x series. It was released on June 3rd, 2022. Changes include:

  • Bug fix that improves protection when dropping a currently executing job

Magpie 1.40.0

Version 1.40.0 is a minor release in the 1.x series. It was released on May 6th, 2022. Changes include:

  • Adds new validator types: Not and Not Null

  • Bug fix that fixes inconsistent naming of parameters of a Regex Validator

  • Upgrade to login that disables accounts after many consecutive login failures

  • Improvement to spark shuffle partition configuration when starting a cluster

  • Alpha support for replacing underlying VM instances of clusters on AWS and Azure backed Magpie Instances

Magpie 1.39.0

Version 1.39.0 is a minor release in the 1.x series. It was released on March 10th, 2022. Changes include:

  • Bug fix that prevents dropping currently running jobs

  • Adds timestamp filters to activity history commands

  • Performance improvements for activity history commands

  • Upgrade Redshift JDBC Driver to v2.x

  • Adds support in SQL parsing for indexing a function result

  • Adds Equals validator type for Data Validators feature preview

  • Bug fix for Data Validator field selector case sensitivity (single and list selectors are now case insensitive)

This release includes a new Magpie CLI client. All users are encouraged to update their CLI client to the latest version.

Magpie 1.38.0

Version 1.38.0 was skipped.

Magpie 1.37.0

Version 1.37.0 is a minor release in the 1.x series. It was released on February 25th, 2022. Changes include:

  • Adds if stopped option to start cluster commands and if started option to stop cluster commands to facilitate automated cluster management

  • Adds support in SQL for parsing three-part table references

  • Alpha support for data validators. Reach out to your Silectis data engineering support representative for a feature demo.

Magpie 1.36.0

Version 1.36.0 is a minor release in the 1.x series. It was released on January 17th, 2022. There are no user-facing changes.

Magpie 1.35.0

Version 1.35.0 is a minor release in the 1.x series. It was released on January 16th, 2022. Changes include:

  • Bug fix for incorrectly returned metadata in the Hive JDBC driver

  • Adds support in SQL for additional parentheses surrounding a subquery

Magpie 1.34.0

Version 1.34.0 is a minor release in the 1.x series. It was released on December 10th, 2021. Changes include:

  • Improves the look and feel of Magpie Notebooks, including a new color palette for charts

  • New icons in data lineage graphs and job execution flow graphs

  • Adds support in SQL for scientific notation

  • Adds support in SQL for creating decimal literals with a scale of 0 by following an integer literal with a trailing decimal

Magpie 1.33.0

Version 1.33.0 is a minor release in the 1.x series. It was released on November 19th, 2021. Changes include:

  • Bug fix for incorrectly cached S3 credentials when using multiple secret keys to access the same S3 bucket

  • Improves masking of secrets in job logs

Magpie 1.32.0

Version 1.32.0 is a minor release in the 1.x series. It was released on October 29th, 2021. There are no user-facing changes.

Magpie 1.31.0

Version 1.31.0 was skipped.

Magpie 1.30.0

Version 1.30.0 is a minor release in the 1.x series. It was released on October 22nd, 2021. Changes include:

  • Improves Magpie autocomplete with suggestions for metadata names

  • Deprecates :cluster CLI special command. Use use cluster instead

This release includes a new Magpie CLI client. All users are encouraged to update their CLI client to the latest version.

Magpie 1.29.0

Version 1.29.0 is a minor release in the 1.x series. It was released on October 8th, 2021. Changes include:

  • Upgrade to Spark 2.4.8

  • Fix file permissions for CLI executables on some operating systems

  • Automatically disconnect from unhealthy clusters

  • Adds limits to history commands

This release includes a new Magpie CLI client. All users are encouraged to update their CLI client to the latest version.

Magpie 1.28.3

Version 1.28.3 is a patch release in the 1.28.x series. It was released on September 19th, 2021. Changes include:

  • Adds extract data source file command to expand archives in filesystem data sources

Magpie 1.28.2

Version 1.28.2 is a patch release in the 1.28.x series. It was released on September 12th, 2021. Changes include:

  • Adds use cluster command to set the current cluster

  • Adds none as an option for use commands to enable unsetting the current cluster, repository, schema, or project

  • Improves performance of describe commands for certain metadata

Magpie 1.28.1

Version 1.28.1 is a patch release in the 1.28.x series. It was released on September 7th, 2021. Changes include:

  • Bug fix for escaping variables during substitution

Magpie 1.28.0

Version 1.28.0 is a minor release in the 1.x series. It was released on September 5th, 2021. Changes include:

  • Upgrades the underlying image for Magpie clusters

  • Bug fix for timeouts of cancellation requests

  • Bug fix for escaping variables during substitution

  • Bug fix for situations where jobs could have been constructed in a way that allowed infinite loops

This release includes a new Magpie CLI client. All users are encouraged to update their CLI client to the latest version in order to take advantage of the cancellation timeout bug fix.

Magpie 1.27.2

Version 1.27.2 is a patch release in the 1.27.x series. It was released on August 8th, 2021. Changes include:

  • Bug fix for Spark UI formatting

Magpie 1.27.1

Version 1.27.1 was skipped.

Magpie 1.27.0

Version 1.27.0 is a minor release in the 1.x series. It was released on August 8th, 2021. Changes include:

  • Adds support for running metadata-only jobs without a cluster

  • Adds support for cluster health notifications

  • Deprecates CustomEmail and UserEmail notification channel in favor of Email notification channel

  • Deprecates create notification subscription and create schedule for job commands in favor of alter job schedule add notification subscription and alter job add schedule commands

  • Deprecates magpie.interpret and magpie.profile functions in scripts in favor of mc.interpret and mc.profile

  • Improvement for error message when attempting to export a result when there is no last result set

  • Bug fix for executor log overflow on long-running clusters

Magpie 1.26.3

Version 1.26.3 is a patch release in the 1.26.x series. It was released on July 18th, 2021. Changes include:

  • Bug fix for granting permissions on clusters to an organization

Magpie 1.26.2

Version 1.26.2 is a patch release in the 1.26.x series. It was released on July 16th, 2021. Changes include:

  • Bug fix for executing SQL queries with large numbers of binary expressions

  • Bug fix for dropping the default cluster for an organization

  • Adds an optional with bootstrap script clause to alter cluster commands

Magpie 1.26.1

Version 1.26.1 is a patch release in the 1.26.x series. It was released on July 3rd, 2021. Changes include:

  • Adds more detail on storage types to cluster descriptions

Magpie 1.26.0

Version 1.26.0 is a minor release in the 1.x series. It was released on June 27th, 2021. Changes include:

  • Adds a cluster usage command to summarize usage of user-managed Magpie clusters

  • Adds the ability to grant permissions at the organization level

  • Bug fix for display of history commands in the notebook when the output contains large log messages

  • Deprecates the createMode and writeMode parameters in SQL mapper tasks in favor of the saveMode parameter

Magpie 1.25.1

Version 1.25.1 is a patch release in the 1.25.x series. It was released on June 16th, 2021. Changes include:

  • Bug fix for display of certain code blocks in task descriptions rendered in the notebook

Magpie 1.25.0

Version 1.25.0 is a minor release in the 1.x series. It was released on June 13th, 2021. Changes include:

  • Adds a cluster health monitor to periodically monitor the health of user-managed Magpie clusters

Magpie 1.24.3

Version 1.24.3 is a patch release in the 1.24.x series. It was released on May 16th, 2021. Changes include:

  • Bug fix for setting the default schema when connected to PowerBI

Magpie 1.24.2

Version 1.24.2 is a patch release in the 1.24.x series. It was released on April 23rd, 2021. Changes include:

  • Adds stack trace details to runtime error messages to aid in user debugging

Magpie 1.24.1

Version 1.24.1 is a patch release in the 1.24.x series. It was released on March 5th, 2021. Changes include:

  • Adds limits to show commands. Full result objects can still be accessed using the save and export commands

  • Bug fix where paragraphs could sometimes execute out of order in the notebook

  • Bug fix that prevents creating cycles in jobs via the use of nested job tasks

  • Bug fix that prevents creating cycles in tables with SQL persistence mappings

  • Bug fix where tables that had already been loaded in user sessions would not be reloaded if their Magpie data source definition changed

Magpie 1.24.0

Version 1.24.0 is a minor release in the 1.x series. It was released on February 26th, 2021. Changes include:

  • Adds the admin permission as an option on all permission-managed objects

  • Adds an optional timeout for tasks

  • Bug fix where tables that were cached prior to profiling would sometimes become uncached after profiling

Magpie 1.23.2

Version 1.23.2 is a patch release in the 1.23.x series. It was released on February 12th, 2021. Changes include:

  • Adds additional lineage details for tables that were saved from a result

  • Adds additional lineage details for tables that were saved from a URL

  • Bug fix where legacy variable substitution syntax would be replaced with new syntax when creating script tasks

Magpie 1.23.1

Version 1.23.1 is a patch release in the 1.23.x series. It was released on January 29th, 2021. There are no user-facing changes.

Magpie 1.23.0

Version 1.23.0 is a minor release in the 1.x series. It was released on January 22nd, 2021. Changes include:

  • Adds project name to job notification emails

  • Bug fix for uploading large files via the CLI

This release contains a new version of the CLI. All users are encouraged to upgrade.

Magpie 1.22.0

Version 1.22.0 is a minor release in the 1.x series. It was released on January 8th, 2021. Changes include:

  • Adds a job execution graph view when describing jobs in notebook

  • Adds new lineage permission type to control access to metadata lineage in descriptions

  • Adds recent execution history to job, task, and project descriptions

  • Bug fix for uploading files of certain sizes via the CLI

Magpie 1.21.0

Version 1.21.0 is a minor release in the 1.x series. It was released on December 13th, 2020. Changes include:

  • Adds new Variables and Secrets API on the MagpieContext for Scala, Python, and R. Existing variable management functions are deprecated and will be removed in a future release.

  • Bug fix for timing out cancellation of hanging Python scripts

  • Bug fix for copying files from the local filesystem to a remote filesystem data source

  • Beta support for self-service cluster management. Reach out to your Silectis data engineering support representative if you would like access.

Magpie 1.20.3

Version 1.20.3 is a patch release in the 1.20.x series. It was released on December 6th, 2020. Changes include:

  • Improved display of table lineage graphs for tables with long lineage

  • Adds script language to activity history display for script actions

  • Bug fix that could cause cancelled actions that timed out during cancellation to display as “in progress”

Magpie 1.20.2

Version 1.20.2 is a patch release in the 1.20.x series. It was released on December 1st, 2020. Changes include:

  • Bug fix for table lineage graphs

Magpie 1.20.1

Version 1.20.1 is a patch release in the 1.20.x series. It was released on November 30th, 2020. Changes include:

  • Bug fix for table lineage graphs

  • Bug fix for listing tables and schemas in Snowflake data sources

Magpie 1.20.0

Version 1.20.0 is a minor release in the 1.x series. It was released on November 29th, 2020. Changes include:

  • Adds secret metadata objects and updates how variables and secrets can be substituted into commands. See the changes here.

  • Beta support for table lineage graphs in table descriptions

Magpie 1.19.1

Version 1.19.1 is a patch release in the 1.19.x series. It was released on October 30th, 2020. Changes include:

  • Adds support for Snowflake as a data source

  • Normalizes the format of all metadata description JSON

Magpie 1.19.0

Version 1.19.0 is a minor release in the 1.x series. It was released on October 18th, 2020. Changes include:

  • Adds metadata lineage information to metadata descriptions

Magpie 1.18.6

Version 1.18.6 is a patch release in the 1.18.x series. It was released on October 4th, 2020. Changes include:

  • Adds schedule history command to enable querying job history by schedule

  • Improves display of times on job history commands in notebooks

  • Bug fix for display of top values in profiles in the notebook for very long values

Magpie 1.18.5

Version 1.18.5 is a patch release in the 1.18.x series. It was released on September 27th, 2020. Changes include:

  • Adds additional validation for BigQuery data source credentials

  • Adds with ignore failed tables option to create schema from data source commands

  • Adds retry failure mode for tasks

Magpie 1.18.4

Version 1.18.4 was not released.

Magpie 1.18.3

Version 1.18.3 is a patch release in the 1.18.x series. It was released on August 30th, 2020. Changes include:

  • Bug fix for profiling date and timestamp fields that are entirely null

  • Prefixes notebook commits with username when using revision control

  • Variable names are now validated the same as other metadata names

  • Adds handling for array, map, and struct types in profiles

Magpie 1.18.2

Version 1.18.2 is a patch release in the 1.18.x series. It was released on August 23rd, 2020. Changes include:

  • Bug fix for setting up notebook revision control with an uninitialized Git repository

  • Removes origin field from notebook revision control config

  • Adds earliest and latest dates and timestamps to profiles

Magpie 1.18.1

Version 1.18.1 is a patch release in the 1.18.x series. It was released on August 16th, 2020. Changes include:

  • Bug fix for setting notebook revision control configuration in some environments

Magpie 1.18.0

Version 1.18.0 is a minor release in the 1.x series. It was released on August 16th, 2020. Changes include:

  • Beta support for notebook version control

  • Ignores NaN values when calculating statistics for numerical fields during profiles

  • Shows null values as separate item in top values list for field profiles

  • Upgrades underlying BigQuery connector library

  • Bug fix for cancellation of spark jobs in python scripts

  • Removes metadata displayName fields, deprecated in January

Magpie 1.17.4

Version 1.17.4 is a patch release in the 1.17.x series. It was released on August 2nd, 2020. Changes include:

  • Bug fix for cancellation of tasks queued behind other tasks

Magpie 1.17.3

Version 1.17.3 is a patch release in the 1.17.x series. It was released on July 28th, 2020. Changes include:

  • Bug fix for cancellation of Scala scripts

  • Bug fix for performance regression of Spark UI

Magpie 1.17.2

Version 1.17.2 is a patch release in the 1.17.x series. It was released on July 26th, 2020. Changes include:

  • Adds list schedules in project command

  • Adds list fields in schema command

  • Beta support for uploading files via the CLI

  • Removes the 1 million row limit for export result commands, and removes the associated limit parameter from the command

  • Upgrade to Spark 2.4.6

  • Fixes warning messages that would appear on CLI startup on newer Java versions

  • Bug fix where error messages would sometimes not be formatted correctly in the notebook

Magpie 1.17.1

Version 1.17.1 is a patch release in the 1.17.x series. It was released on July 5th, 2020. Changes include:

  • Bug fix for python scripts that were not able to be cancelled

  • Bug fix where data sources could be deleted even when tables still referenced them

Magpie 1.17.0

Version 1.17.0 is a minor release in the 1.x series. The rolling release started on June 21st, 2020. Changes include:

  • Adds autocomplete for users and roles when managing notebook permissions

  • Adds assert command

  • Improves display of describe field and describe task command output in notebook

  • Adds field count to table profiles

  • Adds support for writing to BigQuery data sources

  • Adds Delta Lake python library

  • Bug fix for situations where foreign key constraints would appear in the result set when listing tables in a Postgres data source

Magpie 1.16.1

Version 1.16.1 is a patch release in the 1.16.x series. It was released on May 24th, 2020. Changes include:

  • Adds support for syntax autocomplete in notebook for Magpie, SQL, Python, and Scala

  • Bug fix for casting fields as decimal with precision and scale in SQL

  • Bug fix that disables CSV column pruning by default, which could improve performance but also could cause silent data corruption for certain CSV files. Users can re-enable column pruning by executing set spark.sql.csv.parser.columnPruning.enabled = true in a sql statement

Magpie 1.16.0

Version 1.16.0 is a minor release in the 1.x series. It was released on May 10th, 2020. Changes include:

  • Upgrade to Python 3.7

  • Adds Postgres bulk data source, a more efficient option for bulk transfers to and from Postgres databases

Magpie 1.15.1

Version 1.15.1 is a patch release in the 1.15.x series. It was released on May 3rd, 2020. Changes include:

  • Adds quoteAll option for writing tables with csv persistence mappings

Magpie 1.15.0

Version 1.15.0 is a minor release in the 1.x series. It was released on April 19th, 2020. Changes include:

  • Ability for users to set and reset their passwords via a password reset email

  • Replaces the set password command with the reset password command which sends a reset email to the user

  • Adds list variables command to show all the currently set variables in a user’s session

  • Bug fix for some numerical field profiles

Magpie 1.14.1

Version 1.14.1 is a patch release in the 1.14.x series. It was released on April 12th, 2020. Changes include:

  • Bug fix where some notebook paragraphs were not able to be cancelled

Magpie 1.14.0

Version 1.14.0 is a minor release in the 1.x series. The rolling release started on March 29th, 2020. Changes include:

  • Ability for admins to restart the Magpie interpreter from the notebook

  • More descriptive error messages for JSON parsing errors

  • Support for setting a custom null value when exporting results

  • Support for setting a custom encoding when reading and writing csv and json files

  • Support for setting a custom encoding when exporting results

  • Support for sharing Scala, Python, and R functions and variables throughout tasks in a job

  • Bug fix for closure serialization in Scala script tasks

  • Bug fix for displaying items after data tables in a single notebook paragraph

  • Bug fix to display an error message when using the with replace with delete option with unsupported data sources

Magpie 1.13.6

Version 1.13.6 is a patch release in the 1.13.x series. It was released on March 22nd, 2020. Changes include:

  • More descriptive error messages for command parsing errors

  • Bug fix for intermittent errors when defining Spark Scala UDFs in the notebook

Magpie 1.13.5

Version 1.13.5 is a patch release in the 1.13.x series. It was released on March 15th, 2020. Changes include:

  • Support for accessing views in JDBC data sources

  • Support for setting a custom null value when reading and writing csv files

Magpie 1.13.4

Version 1.13.4 is a patch release in the 1.13.x series. It was released on March 9th, 2020. Changes include:

  • Support for Short, Byte, and Null field types

  • Bug fix to support referencing tables with case sensitive names in create table from database table commands

Magpie 1.13.3

Version 1.13.3 is a patch release in the 1.13.x series. It was released on March 8th, 2020. Changes include:

  • New HDFS sources have the default path set as / when not specified

  • Bug fix for display of Spark jobs link in notebook for paragraphs that have a large number of Spark jobs

  • Bug fix for loading CSV files that have illegal characters in the header row

Magpie 1.13.2

Version 1.13.2 is a patch release in the 1.13.x series. It was released on March 1st, 2020. Changes include:

  • Beta support for reading and writing Delta Lake tables

  • Bug fix for viewing profiles of very wide tables

Magpie 1.13.1

Version 1.13.1 is a patch release in the 1.13.x series. It was released on February 23rd, 2020. Changes include:

  • Support for Azure Blob Storage data sources

  • Support for setting context defaults at the user level (default schema, default project, default repository)

  • Stability improvements to S3A connector

Magpie 1.13.0

Version 1.13.0 is a minor release in the 1.x series. It was released on February 16th, 2020. Changes include:

  • Support for Azure Data Lake Gen 2 data sources

  • Support for Azure Data Lake Gen 1 data sources

  • Upgrade to Spark 2.4.5

  • Upgrade to Hadoop 3.2.1

Magpie 1.12.5

Version 1.12.5 is a patch release in the 1.12.x series. It was released on February 9th, 2020. Changes include:

  • Bug fix to handle R process failures in R interpreter

  • Bug fix to handle incorrect parameters in magpie::sql R method

Magpie 1.12.4

Version 1.12.4 is a patch release in the 1.12.x series. It was released on February 2nd, 2020. Changes include:

  • Bug fix for multiple simultaneous R notebook sessions

  • Bug fix for connection timeouts with CLI

Magpie 1.12.3

Version 1.12.3 is a patch release in the 1.12.x series. It was released on January 19th, 2020. Changes include:

  • Bug fix for display of script tasks in notebook

Magpie 1.12.2

Version 1.12.2 is a patch release in the 1.12.x series. It was released on January 19th, 2020. Changes include:

  • Adds support for max partition count in save result and save table as commands

  • Adds support for specifying an alternate data source in save result and save table as commands

  • Adds support for specifying a default schema in database data sources

  • Adds feedback and support links to Magpie homepage

  • Deprecates "display name" property in metadata objects

Magpie 1.12.1

Version 1.12.1 is a patch release in the 1.12.x series. It was released on January 12th, 2020. There are no user-facing changes.

Magpie 1.12.0

Version 1.12.0 is a minor release in the 1.x series. It was released on January 5th, 2020. Changes include:

  • Beta support for R language blocks in notebook and R script tasks in jobs.

  • Bug fix for the notebook that could cause columns to become hidden as wide tables were scrolled horizontally.

  • Improvement for BigQuery connector that enables retries for certain internal BigQuery API errors when reading from BigQuery tables.

Magpie 1.11.2

Version 1.11.2 is a patch release in the 1.11.x series. It was released on December 15th, 2019. Changes include:

  • Bug fix that caused very large integers to display incorrectly in the notebook

  • Adds support for rlike operator in SQL

Magpie 1.11.1

Version 1.11.1 is a patch release in the 1.11.x series. It was released on December 8th, 2019. Changes include:

  • Spark jobs started by the BI tool interface now include a username label

  • Adds support for reading from BigQuery views

  • Bug fix that caused the BI tool interface to leak connections in some cases

Magpie 1.11.0

Version 1.11.0 is a minor release in the 1.x series. It was released on December 8th, 2019. Changes include:

  • List commands now only display objects that users can use or read rather than always displaying all objects. If a user has create access to a parent object, all child objects are displayed regardless of use or read permissions.

  • Changes default timestamp format to yyyy-MM-dd HH:mm:ss for csv files in export result, create table from file, and save URL as table commands

  • Improves performance of BI tool interface when incremental mode is used

  • Adds more detail to authorization exceptions, explaining why permission was denied

  • Sorts permissions returned by describe user and role commands

  • Adds all files matched by a list data source files command to the result object, regardless of how many were displayed in the original command output

  • Removes color characters from CLI output for easier batch processing

  • Adds additional CLI flags for logging level to control output verbocity

  • Adds CLI flag to echo command input before executing commands via the CLI in batch mode

  • Bug fix where stale table references could exist in a user's context if another user renamed the table

  • Bug fix that prevented a table from being accessed in a SQL script in some circumstances

  • Bug fix that prevented using the left and right SQL functions

This release includes a new Magpie CLI client. All users are encouraged to update their CLI client to the latest version in order to utilize the new command line flags.

Magpie 1.10.3

Version 1.10.3 is a patch release in the 1.10.x series. It was released on October 20th, 2019. Changes include:

  • Upgrade to Spark 2.4.4

  • Hides inaccessible schemas and tables from BI tools. Previously, schemas and tables that were not accessible due to insufficient user permissions may have been listed, even though the user was not able to query them. Now, those inaccessible schemas and tables are not listed.

  • Adds ability to specify repository name in a Microstrategy JDBC connection

Magpie 1.10.2

Version 1.10.2 is a patch release in the 1.10.x series. It was released on October 13th, 2019. Changes include:

  • Improves command parsing times for long commands

  • Bug fix for running long commands from the CLI

  • Bug fix for python path of scheduled jobs

This release includes a new Magpie CLI client. All users are encouraged to update their CLI client to the latest version in order to benefit from the improvement in command parsing times.

Magpie 1.10.1

Version 1.10.1 is a patch release in the 1.10.x series. It was released on October 6th, 2019. Changes include:

  • Bug fix for supporting data source options when using Redshift bulk persistence mappings

  • Adds ability to specify a custom AWS IAM Role for Redshift bulk persistence mappings

  • Adds with merge schema option to create table from file commands

  • Bug fix for chart x-axis ordering of zero values

Magpie 1.10.0

Version 1.10.0 is a minor release in the 1.x series. It was released on September 27th, 2019. Changes include:

  • Adds ability for organization admins to manage users in their organization (adding, dropping, altering, enabling, disabling, and resetting passwords)

  • Granting notebook permissions to roles now simply uses the role name rather than the organization-qualified name (i.e., role_name vs org_name.role_name). Granting permissions via the organization-qualified name has been deprecated and will be removed in a future release.

  • Improves cluster resource management so that cluster resources are shared between jobs executing concurrently rather than jobs queuing and waiting for resources in sequence

  • Adds support for wildcards in list data source files, copy data source file, delete data source file, and create table from file commands as well as in table persistence mappings (e.g., *.csv)

  • Improves behavior of describe job command by displaying tasks in execution order

  • Actual code (magpie commands, SQL script, python script, or Scala script) that was run for each task in a job is present in the job execution log

  • Commands executed via the Magpie CLI are now able to be cancelled by sending a process interrupt to the CLI. With a keyboard in an interactive session, this can be done by pressing CTRL+C.

  • Removes cancel job command in favor of cancelling via the Notebook or CLI

  • Bug fix where create table from sql erroneously always required a current schema to be set

  • Bug fix that prevented cancelling execute sql statements

  • Object ACL is now set on cross-account S3 buckets, enabling access to files written by Magpie outside of the platform

  • Adds better handling for refresh table commands so that the refresh is pushed to all Magpie users

  • Bug fix where aggregations in notebook charts were incorrect when they included null values

This release includes a new Magpie CLI client. All users are encouraged to update their CLI client to the latest version in order to take advantage of the new command cancellation feature.

Magpie 1.9.7

Version 1.9.7 is a patch release in the 1.9.x series. It was released on September 12th, 2019. Changes include:

  • Adds if exists option to delete data source file command

  • Adds fetch size option to create table from database data source commands

Magpie 1.9.6

Version 1.9.6 is a patch release in the 1.9.x series. It was released on August 28th, 2019. Changes include:

  • Access to the Spark UI for admin users

  • Job history display now includes the user that executed the job

Magpie 1.9.5

Version 1.9.5 is a patch release in the 1.9.x series. It was released on August 21st, 2019. Changes include:

  • Bug fix to stop writing potentially sensitive information to the activity log

Magpie 1.9.4

Version 1.9.4 is a patch release in the 1.9.x series. It was released on August 13th, 2019. Changes include:

  • Notebook permissions are now case insensitive

  • Bug fix that required billed project ID to be set on BigQuery data sources

  • Bug fix that prevented execute SQL commands from working on BigQuery data sources

Magpie 1.9.3

Version 1.9.3 is a patch release in the 1.9.x series. It was released on July 30th, 2019. Changes include:

  • Support for avro file format

Magpie 1.9.2

Version 1.9.2 is a patch release in the 1.9.x series. It was released on July 24th, 2019. Changes include:

  • Improved handling for job schedule start and end times. Time range is now validated to ensure schedule will run at least once.

  • Bug fix that sometimes caused the notebook progress bar to report over 100% progress

  • Bug fix that prevented some escape sequences from being entered into strings in commands

  • Bug fix that checks whether an existing table's persistence mapping is writable before attempting to replace it when it is the target of a save ... as table with replace with delete command

Magpie 1.9.1

Version 1.9.1 is a patch release in the 1.9.x series. It was released on July 16th, 2019. Changes include:

  • Improved progress reporting for python script paragraphs in the notebook

  • Bug fix that prevented reading scripts from the console standard input on Windows

Magpie 1.9.0

Version 1.9.0 is a minor release in the 1.x series. It was released on July 12th, 2019. Changes include:

  • Improved table and graph widgets for viewing results in the notebook

  • Running paragraphs display their progress in the notebook

  • Increased the amount of memory available for the Magpie process on each cluster

Was this article helpful?
1 out of 1 found this helpful