Alluxio now provides just-in-time data transform of data to be compute-optimized for applications like Presto, independent of the storage solution or format
SAN MATEO, CA – March 11, 2020 - Alluxio, the developer of open source cloud data orchestration software, today announced the availability of Alluxio Structured Data Service (SDS) featuring a data Catalog Service and Transformation Service, two new major architectural components of its Data Orchestration Platform. Data engineers, architects and developers can now spend less resources storing data and more time delivering data to analytical compute engines.
As users and enterprises leverage widely-available analytics engines such as Presto, Apache Spark SQL or Apache Hive, they often run into inefficient data formats and face performance challenges. Typically, those engines consume structured data in different databases with "tables" consisting of "rows" and "columns", rather than "offset" and "length" in files or objects. This gap creates multiple challenges and inefficiencies, such as mappings or creating converted copies of the data. With this announcement, users benefit from a more simplified data platform that enables connections to different catalogs for access to structured data, with less copies and pipelines and more compute-optimized data.
"Alluxio now provides just-in-time data transform of data to be compute-optimized, independent of the storage format for OLAP engines, such as Presto and Apache Spark," said Haoyuan Li, founder and CTO, Alluxio. "These schema-aware optimizations are made possible with the new Alluxio Catalog Service which abstracts the widely-used Apache Hive Metastore, so regardless of how the data was initially stored - CSV and text formatted files, for example – the data is now transformed into the generally recognized compute-optimized parquet format. Almost every organization has a surprising amount of data in CSV or other text formats and this removes the manual work to make that data more usable. A second type of transformation will coalesce many smaller files, enabling the data to be combined into fewer files, which is more efficient to process for SQL engines. And yet a third type of transformation is for sorting, enabling table columns to be sorted adding to the efficiency of queries, newly available in our Enterprise Edition."
"We can thank Kubernetes for distributed compute; and Alluxio for distributed data. The combination of these technologies offers tremendous promise for our data-driven hybrid and multicloud future," said Eric Kavanagh, CEO, Bloor Group.
Alluxio Structured Data Service
With Alluxio Structured Data Service, Alluxio can expose the data to be effectively accessed by the SQL engines, independent of how and where the data is stored. New capabilities and services include:
- Presto Connector for Alluxio - A new Presto connector for Alluxio is now available. This allows easy integration and configuration of Alluxio with Presto.
- Catalog Service - The new Alluxio Catalog Service manages the metadata of structured data in the system. It is responsible for all the database, table, and schema information, as well as the location of all the stored data. There is no longer a need to change any table locations in the Hive metastore, or to restart or reconfigure any Hive services. The Alluxio Catalog Service enables schema-aware optimizations for any type of structured data. For example, once the Hive metastore is attached to the Alluxio Catalog Service, the catalog service will automatically mount the appropriate table locations, and automatically serve the table metadata with the Alluxio locations.
- Transformation Service – The new Alluxio Transformation Service transforms data into a compute-optimized representation of the data, which is independent from the storage-optimized format. This enables physical data independence. Three types of transformations are available for tables: coalesce, format conversion, and sorting. While results depend on the specific formats and workloads, internal tests have shown increase in query performance by over 2.5x.
Availability
Alluxio 2.2 Community and Enterprise Edition with Structured Data Service are generally available for download here: https://www.alluxio.io/download/
Tweet this: .@Alluxio unveils Structured Data Service, new services that include data Catalog Service and Transform Service #DataCatalog #DataTransform #DataOrchestration https://bit.ly/2PVvKY7
Resources
Blog: What's new in Alluxio 2.2
Download: https://www.alluxio.io/download/
Blog: Serving Structured Data in Alluxio: Concept
Blog: Serving Structured Data in Alluxio: Example
About Alluxio
Alluxio is a leading provider of accelerated data access platforms for AI workloads. Alluxio’s distributed caching layer accelerates AI and data-intensive workloads by enabling high-speed data access across diverse storage systems. By creating a global namespace, Alluxio unifies data from multiple sources—on-premises and in the cloud—into a single, logical view, eliminating the need for data duplication or complex data movement.
Designed for scalability and performance, Alluxio brings data closer to compute frameworks like TensorFlow, PyTorch, and Spark, significantly reducing I/O bottlenecks and latency. Its intelligent caching, data locality optimization, and seamless integration with modern data platforms make it a powerful solution for teams building and scaling AI pipelines across hybrid and multi-cloud environments. Backed by leading investors, Alluxio powers technology, internet, financial services, and telecom companies, including 9 out of the top 10 internet companies globally. To learn more, visit www.alluxio.io.
Media Contact:
Beth Winkowski
Winkowski Public Relations, LLC for Alluxio
978-649-7189
beth@alluxio.com
.png)
News & Press
AMSTERDAM, NETHERLANDS, JUNE 10, 2025 — In today’s confusing and messy enterprise software market, innovative technology solutions that realize real customer results are hard to come by. As an industry analyst firm that focuses on enterprise digital transformation and the disruptive vendors that support it, Intellyx interacts with numerous innovators in the enterprise IT marketplace.
Alluxio, supplier of open source virtual distributed file systems, announced Alluxio Enterprise AI 3.6. This delivers capabilities for model distribution, model training checkpoint writing optimization, and enhanced multi-tenancy support. It can, we’re told, accelerate AI model deployment cycles, reduce training time, and ensure data access across cloud environments. The new release uses Alluxio Distributed Cache to accelerate model distribution workloads; by placing the cache in each region, model files need only be copied from the Model Repository to the Alluxio Distributed Cache once per region rather than once per server.