Release Notes

Magpie 1.24.1

Version 1.24.1 is a patch release in the 1.24.x series. It was released on March 5th, 2021. Changes include:

  • Adds limits to show commands. Full result objects can still be accessed using the save and export commands

  • Bug fix where paragraphs could sometimes execute out of order in the notebook

  • Bug fix that prevents creating cycles in jobs via the use of nested job tasks

  • Bug fix that prevents creating cycles in tables with SQL persistence mappings

  • Bug fix where tables that had already been loaded in user sessions would not be reloaded if their Magpie data source definition changed

Magpie 1.24.0

Version 1.24.0 is a minor release in the 1.x series. It was released on February 26th, 2021. Changes include:

  • Adds the admin permission as an option on all permission-managed objects

  • Adds an optional timeout for tasks

  • Bug fix where tables that were cached prior to profiling would sometimes become uncached after profiling

Magpie 1.23.2

Version 1.23.2 is a patch release in the 1.23.x series. It was released on February 12th, 2021. Changes include:

  • Adds additional lineage details for tables that were saved from a result

  • Adds additional lineage details for tables that were saved from a URL

  • Bug fix where legacy variable substitution syntax would be replaced with new syntax when creating script tasks

Magpie 1.23.1

Version 1.23.1 is a patch release in the 1.23.x series. It was released on January 29th, 2021. There are no user-facing changes.

Magpie 1.23.0

Version 1.23.0 is a minor release in the 1.x series. It was released on January 22nd, 2021. Changes include:

  • Adds project name to job notification emails

  • Bug fix for uploading large files via the CLI

This release contains a new version of the CLI. All users are encouraged to upgrade.

Magpie 1.22.0

Version 1.22.0 is a minor release in the 1.x series. It was released on January 8th, 2021. Changes include:

  • Adds a job execution graph view when describing jobs in notebook

  • Adds new lineage permission type to control access to metadata lineage in descriptions

  • Adds recent execution history to job, task, and project descriptions

  • Bug fix for uploading files of certain sizes via the CLI

Magpie 1.21.0

Version 1.21.0 is a minor release in the 1.x series. It was released on December 13th, 2020. Changes include:

  • Adds new Variables and Secrets API on the MagpieContext for Scala, Python, and R. Existing variable management functions are deprecated and will be removed in a future release.

  • Bug fix for timing out cancellation of hanging Python scripts

  • Bug fix for copying files from the local filesystem to a remote filesystem data source

  • Beta support for self-service cluster management. Reach out to your Silectis data engineering support representative if you would like access.

Magpie 1.20.3

Version 1.20.3 is a patch release in the 1.20.x series. It was released on December 6th, 2020. Changes include:

  • Improved display of table lineage graphs for tables with long lineage

  • Adds script language to activity history display for script actions

  • Bug fix that could cause cancelled actions that timed out during cancellation to display as “in progress”

Magpie 1.20.2

Version 1.20.2 is a patch release in the 1.20.x series. It was released on December 1st, 2020. Changes include:

  • Bug fix for table lineage graphs

Magpie 1.20.1

Version 1.20.1 is a patch release in the 1.20.x series. It was released on November 30th, 2020. Changes include:

  • Bug fix for table lineage graphs

  • Bug fix for listing tables and schemas in Snowflake data sources

Magpie 1.20.0

Version 1.20.0 is a minor release in the 1.x series. It was released on November 29th, 2020. Changes include:

  • Adds secret metadata objects and updates how variables and secrets can be substituted into commands. See the changes here.

  • Beta support for table lineage graphs in table descriptions

Magpie 1.19.1

Version 1.19.1 is a patch release in the 1.19.x series. It was released on October 30th, 2020. Changes include:

  • Adds support for Snowflake as a data source

  • Normalizes the format of all metadata description JSON

Magpie 1.19.0

Version 1.19.0 is a minor release in the 1.x series. It was released on October 18th, 2020. Changes include:

  • Adds metadata lineage information to metadata descriptions

Magpie 1.18.6

Version 1.18.6 is a patch release in the 1.18.x series. It was released on October 4th, 2020. Changes include:

  • Adds schedule history command to enable querying job history by schedule

  • Improves display of times on job history commands in notebooks

  • Bug fix for display of top values in profiles in the notebook for very long values

Magpie 1.18.5

Version 1.18.5 is a patch release in the 1.18.x series. It was released on September 27th, 2020. Changes include:

  • Adds additional validation for BigQuery data source credentials

  • Adds with ignore failed tables option to create schema from data source commands

  • Adds retry failure mode for tasks

Magpie 1.18.4

Version 1.18.4 was not released.

Magpie 1.18.3

Version 1.18.3 is a patch release in the 1.18.x series. It was released on August 30th, 2020. Changes include:

  • Bug fix for profiling date and timestamp fields that are entirely null

  • Prefixes notebook commits with username when using revision control

  • Variable names are now validated the same as other metadata names

  • Adds handling for array, map, and struct types in profiles

Magpie 1.18.2

Version 1.18.2 is a patch release in the 1.18.x series. It was released on August 23rd, 2020. Changes include:

  • Bug fix for setting up notebook revision control with an uninitialized Git repository

  • Removes origin field from notebook revision control config

  • Adds earliest and latest dates and timestamps to profiles

Magpie 1.18.1

Version 1.18.1 is a patch release in the 1.18.x series. It was released on August 16th, 2020. Changes include:

  • Bug fix for setting notebook revision control configuration in some environments

Magpie 1.18.0

Version 1.18.0 is a minor release in the 1.x series. It was released on August 16th, 2020. Changes include:

  • Beta support for notebook version control

  • Ignores NaN values when calculating statistics for numerical fields during profiles

  • Shows null values as separate item in top values list for field profiles

  • Upgrades underlying BigQuery connector library

  • Bug fix for cancellation of spark jobs in python scripts

  • Removes metadata displayName fields, deprecated in January

Magpie 1.17.4

Version 1.17.4 is a patch release in the 1.17.x series. It was released on August 2nd, 2020. Changes include:

  • Bug fix for cancellation of tasks queued behind other tasks

Magpie 1.17.3

Version 1.17.3 is a patch release in the 1.17.x series. It was released on July 28th, 2020. Changes include:

  • Bug fix for cancellation of Scala scripts

  • Bug fix for performance regression of Spark UI

Magpie 1.17.2

Version 1.17.2 is a patch release in the 1.17.x series. It was released on July 26th, 2020. Changes include:

  • Adds list schedules in project command

  • Adds list fields in schema command

  • Beta support for uploading files via the CLI

  • Removes the 1 million row limit for export result commands, and removes the associated limit parameter from the command

  • Upgrade to Spark 2.4.6

  • Fixes warning messages that would appear on CLI startup on newer Java versions

  • Bug fix where error messages would sometimes not be formatted correctly in the notebook

Magpie 1.17.1

Version 1.17.1 is a patch release in the 1.17.x series. It was released on July 5th, 2020. Changes include:

  • Bug fix for python scripts that were not able to be cancelled

  • Bug fix where data sources could be deleted even when tables still referenced them

Magpie 1.17.0

Version 1.17.0 is a minor release in the 1.x series. The rolling release started on June 21st, 2020. Changes include:

  • Adds autocomplete for users and roles when managing notebook permissions

  • Adds assert command

  • Improves display of describe field and describe task command output in notebook

  • Adds field count to table profiles

  • Adds support for writing to BigQuery data sources

  • Adds Delta Lake python library

  • Bug fix for situations where foreign key constraints would appear in the result set when listing tables in a Postgres data source

Magpie 1.16.1

Version 1.16.1 is a patch release in the 1.16.x series. It was released on May 24th, 2020. Changes include:

  • Adds support for syntax autocomplete in notebook for Magpie, SQL, Python, and Scala

  • Bug fix for casting fields as decimal with precision and scale in SQL

  • Bug fix that disables CSV column pruning by default, which could improve performance but also could cause silent data corruption for certain CSV files. Users can re-enable column pruning by executing set spark.sql.csv.parser.columnPruning.enabled = true in a sql statement

Magpie 1.16.0

Version 1.16.0 is a minor release in the 1.x series. It was released on May 10th, 2020. Changes include:

  • Upgrade to Python 3.7

  • Adds Postgres bulk data source, a more efficient option for bulk transfers to and from Postgres databases

Magpie 1.15.1

Version 1.15.1 is a patch release in the 1.15.x series. It was released on May 3rd, 2020. Changes include:

  • Adds quoteAll option for writing tables with csv persistence mappings

Magpie 1.15.0

Version 1.15.0 is a minor release in the 1.x series. It was released on April 19th, 2020. Changes include:

  • Ability for users to set and reset their passwords via a password reset email

  • Replaces the set password command with the reset password command which sends a reset email to the user

  • Adds list variables command to show all the currently set variables in a user’s session

  • Bug fix for some numerical field profiles

Magpie 1.14.1

Version 1.14.1 is a patch release in the 1.14.x series. It was released on April 12th, 2020. Changes include:

  • Bug fix where some notebook paragraphs were not able to be cancelled

Magpie 1.14.0

Version 1.14.0 is a minor release in the 1.x series. The rolling release started on March 29th, 2020. Changes include:

  • Ability for admins to restart the Magpie interpreter from the notebook

  • More descriptive error messages for JSON parsing errors

  • Support for setting a custom null value when exporting results

  • Support for setting a custom encoding when reading and writing csv and json files

  • Support for setting a custom encoding when exporting results

  • Support for sharing Scala, Python, and R functions and variables throughout tasks in a job

  • Bug fix for closure serialization in Scala script tasks

  • Bug fix for displaying items after data tables in a single notebook paragraph

  • Bug fix to display an error message when using the with replace with delete option with unsupported data sources

Magpie 1.13.6

Version 1.13.6 is a patch release in the 1.13.x series. It was released on March 22nd, 2020. Changes include:

  • More descriptive error messages for command parsing errors

  • Bug fix for intermittent errors when defining Spark Scala UDFs in the notebook

Magpie 1.13.5

Version 1.13.5 is a patch release in the 1.13.x series. It was released on March 15th, 2020. Changes include:

  • Support for accessing views in JDBC data sources

  • Support for setting a custom null value when reading and writing csv files

Magpie 1.13.4

Version 1.13.4 is a patch release in the 1.13.x series. It was released on March 9th, 2020. Changes include:

  • Support for Short, Byte, and Null field types

  • Bug fix to support referencing tables with case sensitive names in create table from database table commands

Magpie 1.13.3

Version 1.13.3 is a patch release in the 1.13.x series. It was released on March 8th, 2020. Changes include:

  • New HDFS sources have the default path set as / when not specified

  • Bug fix for display of Spark jobs link in notebook for paragraphs that have a large number of Spark jobs

  • Bug fix for loading CSV files that have illegal characters in the header row

Magpie 1.13.2

Version 1.13.2 is a patch release in the 1.13.x series. It was released on March 1st, 2020. Changes include:

  • Beta support for reading and writing Delta Lake tables

  • Bug fix for viewing profiles of very wide tables

Magpie 1.13.1

Version 1.13.1 is a patch release in the 1.13.x series. It was released on February 23rd, 2020. Changes include:

  • Support for Azure Blob Storage data sources

  • Support for setting context defaults at the user level (default schema, default project, default repository)

  • Stability improvements to S3A connector

Magpie 1.13.0

Version 1.13.0 is a minor release in the 1.x series. It was released on February 16th, 2020. Changes include:

  • Support for Azure Data Lake Gen 2 data sources

  • Support for Azure Data Lake Gen 1 data sources

  • Upgrade to Spark 2.4.5

  • Upgrade to Hadoop 3.2.1

Magpie 1.12.5

Version 1.12.5 is a patch release in the 1.12.x series. It was released on February 9th, 2020. Changes include:

  • Bug fix to handle R process failures in R interpreter

  • Bug fix to handle incorrect parameters in magpie::sql R method

Magpie 1.12.4

Version 1.12.4 is a patch release in the 1.12.x series. It was released on February 2nd, 2020. Changes include:

  • Bug fix for multiple simultaneous R notebook sessions

  • Bug fix for connection timeouts with CLI

Magpie 1.12.3

Version 1.12.3 is a patch release in the 1.12.x series. It was released on January 19th, 2020. Changes include:

  • Bug fix for display of script tasks in notebook

Magpie 1.12.2

Version 1.12.2 is a patch release in the 1.12.x series. It was released on January 19th, 2020. Changes include:

  • Adds support for max partition count in save result and save table as commands

  • Adds support for specifying an alternate data source in save result and save table as commands

  • Adds support for specifying a default schema in database data sources

  • Adds feedback and support links to Magpie homepage

  • Deprecates "display name" property in metadata objects

Magpie 1.12.1

Version 1.12.1 is a patch release in the 1.12.x series. It was released on January 12th, 2020. There are no user-facing changes.

Magpie 1.12.0

Version 1.12.0 is a minor release in the 1.x series. It was released on January 5th, 2020. Changes include:

  • Beta support for R language blocks in notebook and R script tasks in jobs.

  • Bug fix for the notebook that could cause columns to become hidden as wide tables were scrolled horizontally.

  • Improvement for BigQuery connector that enables retries for certain internal BigQuery API errors when reading from BigQuery tables.

Magpie 1.11.2

Version 1.11.2 is a patch release in the 1.11.x series. It was released on December 15th, 2019. Changes include:

  • Bug fix that caused very large integers to display incorrectly in the notebook

  • Adds support for rlike operator in SQL

Magpie 1.11.1

Version 1.11.1 is a patch release in the 1.11.x series. It was released on December 8th, 2019. Changes include:

  • Spark jobs started by the BI tool interface now include a username label

  • Adds support for reading from BigQuery views

  • Bug fix that caused the BI tool interface to leak connections in some cases

Magpie 1.11.0

Version 1.11.0 is a minor release in the 1.x series. It was released on December 8th, 2019. Changes include:

  • List commands now only display objects that users can use or read rather than always displaying all objects. If a user has create access to a parent object, all child objects are displayed regardless of use or read permissions.

  • Changes default timestamp format to yyyy-MM-dd HH:mm:ss for csv files in export result, create table from file, and save URL as table commands

  • Improves performance of BI tool interface when incremental mode is used

  • Adds more detail to authorization exceptions, explaining why permission was denied

  • Sorts permissions returned by describe user and role commands

  • Adds all files matched by a list data source files command to the result object, regardless of how many were displayed in the original command output

  • Removes color characters from CLI output for easier batch processing

  • Adds additional CLI flags for logging level to control output verbocity

  • Adds CLI flag to echo command input before executing commands via the CLI in batch mode

  • Bug fix where stale table references could exist in a user's context if another user renamed the table

  • Bug fix that prevented a table from being accessed in a SQL script in some circumstances

  • Bug fix that prevented using the left and right SQL functions

This release includes a new Magpie CLI client. All users are encouraged to update their CLI client to the latest version in order to utilize the new command line flags.

Magpie 1.10.3

Version 1.10.3 is a patch release in the 1.10.x series. It was released on October 20th, 2019. Changes include:

  • Upgrade to Spark 2.4.4

  • Hides inaccessible schemas and tables from BI tools. Previously, schemas and tables that were not accessible due to insufficient user permissions may have been listed, even though the user was not able to query them. Now, those inaccessible schemas and tables are not listed.

  • Adds ability to specify repository name in a Microstrategy JDBC connection

Magpie 1.10.2

Version 1.10.2 is a patch release in the 1.10.x series. It was released on October 13th, 2019. Changes include:

  • Improves command parsing times for long commands

  • Bug fix for running long commands from the CLI

  • Bug fix for python path of scheduled jobs

This release includes a new Magpie CLI client. All users are encouraged to update their CLI client to the latest version in order to benefit from the improvement in command parsing times.

Magpie 1.10.1

Version 1.10.1 is a patch release in the 1.10.x series. It was released on October 6th, 2019. Changes include:

  • Bug fix for supporting data source options when using Redshift bulk persistence mappings

  • Adds ability to specify a custom AWS IAM Role for Redshift bulk persistence mappings

  • Adds with merge schema option to create table from file commands

  • Bug fix for chart x-axis ordering of zero values

Magpie 1.10.0

Version 1.10.0 is a minor release in the 1.x series. It was released on September 27th, 2019. Changes include:

  • Adds ability for organization admins to manage users in their organization (adding, dropping, altering, enabling, disabling, and resetting passwords)

  • Granting notebook permissions to roles now simply uses the role name rather than the organization-qualified name (i.e., role_name vs org_name.role_name). Granting permissions via the organization-qualified name has been deprecated and will be removed in a future release.

  • Improves cluster resource management so that cluster resources are shared between jobs executing concurrently rather than jobs queuing and waiting for resources in sequence

  • Adds support for wildcards in list data source filescopy data source file, delete data source file, and create table from file commands as well as in table persistence mappings (e.g., *.csv)

  • Improves behavior of describe job command by displaying tasks in execution order

  • Actual code (magpie commands, SQL script, python script, or Scala script) that was run for each task in a job is present in the job execution log

  • Commands executed via the Magpie CLI are now able to be cancelled by sending a process interrupt to the CLI. With a keyboard in an interactive session, this can be done by pressing CTRL+C.

  • Removes cancel job command in favor of cancelling via the Notebook or CLI

  • Bug fix where create table from sql erroneously always required a current schema to be set

  • Bug fix that prevented cancelling execute sql statements

  • Object ACL is now set on cross-account S3 buckets, enabling access to files written by Magpie outside of the platform

  • Adds better handling for refresh table commands so that the refresh is pushed to all Magpie users

  • Bug fix where aggregations in notebook charts were incorrect when they included null values

This release includes a new Magpie CLI client. All users are encouraged to update their CLI client to the latest version in order to take advantage of the new command cancellation feature.

Magpie 1.9.7

Version 1.9.7 is a patch release in the 1.9.x series. It was released on September 12th, 2019. Changes include:

  • Adds if exists option to delete data source file command

  • Adds fetch size option to create table from database data source commands

Magpie 1.9.6

Version 1.9.6 is a patch release in the 1.9.x series. It was released on August 28th, 2019. Changes include:

  • Access to the Spark UI for admin users

  • Job history display now includes the user that executed the job

Magpie 1.9.5

Version 1.9.5 is a patch release in the 1.9.x series. It was released on August 21st, 2019. Changes include:

  • Bug fix to stop writing potentially sensitive information to the activity log

Magpie 1.9.4

Version 1.9.4 is a patch release in the 1.9.x series. It was released on August 13th, 2019. Changes include:

  • Notebook permissions are now case insensitive

  • Bug fix that required billed project ID to be set on BigQuery data sources

  • Bug fix that prevented execute SQL commands from working on BigQuery data sources

Magpie 1.9.3

Version 1.9.3 is a patch release in the 1.9.x series. It was released on July 30th, 2019. Changes include:

  • Support for avro file format

Magpie 1.9.2

Version 1.9.2 is a patch release in the 1.9.x series. It was released on July 24th, 2019. Changes include:

  • Improved handling for job schedule start and end times. Time range is now validated to ensure schedule will run at least once.

  • Bug fix that sometimes caused the notebook progress bar to report over 100% progress

  • Bug fix that prevented some escape sequences from being entered into strings in commands

  • Bug fix that checks whether an existing table's persistence mapping is writable before attempting to replace it when it is the target of a save ... as table with replace with delete command

Magpie 1.9.1

Version 1.9.1 is a patch release in the 1.9.x series. It was released on July 16th, 2019. Changes include:

  • Improved progress reporting for python script paragraphs in the notebook

  • Bug fix that prevented reading scripts from the console standard input on Windows

Magpie 1.9.0

Version 1.9.0 is a minor release in the 1.x series. It was released on July 12th, 2019. Changes include:

  • Improved table and graph widgets for viewing results in the notebook

  • Running paragraphs display their progress in the notebook

  • Increased the amount of memory available for the Magpie process on each cluster

Was this article helpful?
1 out of 1 found this helpful