Alluxio 2.2.0 Release

We are extremely excited to announce the release of Alluxio 2.2.0!

This release is the General Availability (GA) release of Alluxio Structured Data Services (SDS), the subsystem of Alluxio responsible for managing and transforming structured data, such as databases, tables, and partitions.

Alluxio 2.2.0 also contains several major improvements to the stability, reliability, and performance of the core system. It is intended to be backwards compatible with previous releases on the Alluxio 2.x line.

Downloads can be found here. Thanks to everyone who has contributed to this release!

Release highlights

Alluxio Structured Data Service (SDS)

The Alluxio SDS provides database, table, and partition level metadata to compute frameworks, such as Presto. Based on the data access pattern or explicit user request, it invokes the transformation engine to transparently reorganize data to best fit the workload’s needs, which can provide up to 5x performance improvement.

Alluxio 2.2.0 marks the GA of SDS for read-only workloads. A native integration with Presto is available in the latest Presto release. For older versions of Presto, the connector can be found in the Alluxio release artifact.

Welcome to try it out by reading more about how to get started with Alluxio’s structured data service in the documentation!

Alluxio Data Service

Data Service Engine

The Data Service Engine has been rearchitected to support high performance and scalability required in large scale deployments, especially when utilizing Alluxio SDS.

Data Service Monitoring

In this release, Alluxio Data Service also includes a new CLI for querying information about the state of the job service and throttling capabilities based on the load a node is experiencing. See the documentation for more details.

Alluxio Core

Journal

Online journal backups are now available. A secondary master will take a backup of the journal while the primary master continues to serve requests, allowing for backups to be taken with no downtime.

DevOps – Log Collection

The Alluxio CLI packages a cluster diagnostic tool, `collectInfo`. This allows users to easily gather relevant cluster information for debugging purposes. See the docs for more information.

Important System Improvements

  • Improvements to HDFS Active Sync to allow parallel syncs on multiple mount points (08a194cb33)
  • Support for wasbs URIs (secure wasb) (6bcfc0b23a)
  • Optimization of reads to remote HDFS clusters (4caad25cab)
  • Optimization of list worker RPCs (8c74baf0e3)
  • Optimization of worker metrics collection (c48cb92c78)

Important Bug fixes

  • An issue with clients hanging within long-running processes has been resolved (5c4c32d193, 7311a7063c)
  • An issue when backing-up and restoring a cluster which used the RocksDB block store has been fixed (3f52c815c9)
  • An issue with tagged metrics not being aggregated has been fixed (8e59b2b296)
  • An issue with short-lived meter metrics not being recorded accurately has been fixed (997d57be0c)

A full list of all system improvements and bug fixes can be found here. There were a total of 328 PRs to Alluxio since the last feature release (v2.1.0) A big thanks to their contributions!

Enjoy the new release and look forward to hearing your feedback (community slack channel)!