We are extremely excited to announce the release of Alluxio 2.2.0!
With this release comes the General Availability (GA) of Alluxio Structured Data Services (SDS), the subsystem of Alluxio responsible for managing and transforming structured data, such as databases, tables, and partitions.
Alluxio 2.2.0 also contains several major improvements to the stability, reliability, and performance of the core system. It is intended to be backwards compatible with previous releases on the Alluxio 2.x line.
Downloads can be found here. Thanks to everyone who has contributed to this release! And here are some release highlights:
Alluxio Structured Data Service (SDS)
The Alluxio SDS provides database, table, and partition level metadata to compute frameworks, such as Presto. Based on the data access pattern or explicit user request, it invokes the transformation engine to transparently reorganize data to best fit the workload’s needs, which can provide up to 5x performance improvement.
Alluxio 2.2.0 marks the GA of SDS for read-only workloads. A native integration with Presto is available in the latest Presto release. For older versions of Presto, the connector can be found in the Alluxio release artifact.
Welcome to try it out by reading more about how to get started with Alluxio's structured data service in the documentation!
Alluxio Data Service
The new data service includes two main components - the Data Service Engine and Data Service Monitoring.
Data Service Engine
The Data Service Engine has been rearchitected to support high performance and scalability required in large scale deployments, especially when utilizing Alluxio SDS.
Data Service Monitoring
In this release, Alluxio Data Service also includes a new CLI for querying information about the state of the job service and throttling capabilities based on the load a node is experiencing. See the documentation for more details.
Alluxio Core
Journal
Online journal backups are now available. A secondary master will take a backup of the journal while the primary master continues to serve requests, allowing for backups to be taken with no downtime.
DevOps - Log Collection
The Alluxio CLI packages a cluster diagnostic tool, `collectInfo`. This allows users to easily gather relevant cluster information for debugging purposes. See the docs for more information.
You can see full details of Alluxio 2.2.0 in the release notes.
As a next step, take a look at our first blog in the Serving Structured Data in Alluxio series.
Enjoy the new release and look forward to hearing your feedback (community slack channel)!
Bin, Calvin, Gene, Zac and the Alluxio Developer Community
.png)
Blog
Coupang, a Fortune 200 technology company, manages a multi-cluster GPU architecture for their AI/ML model training. This architecture introduced significant challenges, including:
- Time-consuming data preparation and data copy/movement
- Difficulty utilizing GPU resources efficiently
- High and growing storage costs
- Excessive operational overhead maintaining storage for localized data silos
To resolve these challenges, Coupang’s AI platform team implemented a distributed caching system that automatically retrieves training data from their central data lake, improves data loading performance, unifies access paths for model developers, automates data lifecycle management, and extends easily across Kubernetes environments. The new distributed caching architecture has improved model training speed, reduced storage costs, increased GPU utilization across clusters, lowered operational overhead, enabled training workload portability, and delivered 40% better I/O performance compared to parallel file systems.
Suresh Kumar Veerapathiran and Anudeep Kumar, engineering leaders at Uptycs, recently shared their experience of evolving their data platform and analytics architecture to power analytics through a generative AI interface. In their post on Medium titled Cache Me If You Can: Building a Lightning-Fast Analytics Cache at Terabyte Scale, Veerapathiran and Kumar provide detailed insights into the challenges they faced (and how they solved them) scaling their analytics solution that collects and reports on terabytes of telemetry data per day as part of Uptycs Cloud-Native Application Protection Platform (CNAPP) solutions.
.png)