Now available, Alluxio adds new capabilities to unify data lakes at enormous scale, both in data size and number of files

‍SAN MATEO, CA – October 21, 2020 - Alluxio, the developer of open source cloud data orchestration software, today announced the immediate availability of the next major release of its Data Orchestration platform featuring an expanded metadata service, a new management console for hybrid and multi-cloud deployments, and more cloud native deployments. Data platform teams can now save on infrastructure and operational costs, while easily managing data access across multiple environments.

Enterprises leverage Alluxio at enormous scale, both in data size and number of files. Data orchestration decouples compute from the location of data to optimize for which data resides where and for how long without the management overhead. With this announcement, users will be able to manage namespaces with billions of files without relying on third party systems, greatly reducing the overall deployment footprint of the solution. The new management console will make it easy to connect an analytics cluster, with engines such as Presto and Spark, with data sources across multiple clouds, single cloud or on-premises using Alluxio.

“Organizations have adopted an infrastructure with compute engines and data sets spread across private data centers and public clouds for business agility and cost effectiveness. Our customers have turned to Alluxio to bridge the gap between applications and the storage systems spread across regions and cloud providers,” said Haoyuan Li, founder and CEO, Alluxio.

Alluxio 2.4 Community and Enterprise Edition features new capabilities, including:

Expanded Metadata Service - At the core of the Alluxio Data Orchestration Platform is a metadata service, a scalable, distributed data service for management across multiple sources like traditional Hadoop-based data lakes on-premises or modern cloud-based data lakes. Leveraged to unify data lakes at enormous scale, both in data size and number of files, Alluxio has expanded this service to provide support for billions of files while removing third-party system dependencies. Breaking away from dependencies on traditional Hadoop components, Alluxio has bolstered support for cloud native and container-based deployments. The lifecycle management of the Alluxio’s metadata service now also supports automatic backups without impacting the live system to further reduce the platform management overhead.

New management console – The Alluxio Data Orchestration Hub is a new web-based management console that makes it easy to connect an analytics or machine learning clusters with multiple data sources to unify data lakes. The new service provides an easy-to-use unified management view for configuration and monitoring.

Cloud native deployment - Spawning analytics clusters in AWS and GCP is now easier than ever. Based on Terraform, Alluxio now makes it easy to launch pre-configured clusters programmatically using a single command. Alluxio has been featured as a recommended data lake partner for data lake modernization solution with Google Cloud, including the ability to launch an Alluxio-enabled cluster using the Dataproc component exchange console.

Sensitive data management - Alluxio now integrates with Vault for secure management of sensitive information for data access with dynamic infrastructure across multiple clouds and on-premises environments. With the shift from on-premises infrastructure to multiple cloud-providers, protection of data access tokens and credentials is more important than ever.

Simplified DevOps and system monitoring - Alluxio 2.4 adds several system enhancements to simplify and improve cluster management and maintenance. The system provides an aggregated cluster view of key performance metrics like I/O throughput and metadata request rate through the UI and programmatic monitoring endpoints. Internal monitoring for failures and system slow-downs has been added, further improving the operator view of the health and performance of the system.

Support for Java 11 - Java 11 is the latest long term support version of Java. Alluxio 2.4 provides compatibility with Java 11 while maintaining support for Java 8. Users looking to move their compute engines or Alluxio systems to Java 11 can now do so without any concerns.

Supporting Customer Quotes

‍“We have worked closely with Alluxio and see the Alluxio Data Orchestration System as an integral component of our machine learning and AI platform on Kubernetes, with engines like Tensorflow, now available for both internal and external customers,” said Yang Che, Sr. Staff Engineer, Alibaba Cloud.

"With Alluxio on Kubernetes, we observed 2x-10x speed up of large-scale model training and we’re in the process of building Alluxio as part of our DL infrastructure to power thousands of deep learning training workloads across Microsoft,” said Roger Yu, Partner Software Engineer Manager, Microsoft.

"The Alluxio Data Orchestration System slashed query run times by half when running analytics jobs like Spark in Tencent Cloud, using our EMR platform to allow for greater I/O performance, and provides the ability to provision elastic compute with significantly reduced network resources," said Xiaoping Lei, Vice General Manager of Big Data, Tencent Cloud.

Availability

‍Alluxio 2.4 Community and Enterprise Edition are generally available for download here: https://www.alluxio.io/download/

Resources

Read product blog: Introducing Alluxio 2.4

‍Download Alluxio version 2.4: https://www.alluxio.io/download/

‍Register for the Alluxio 2.4 online tech talk: Introduction to what’s new in Alluxio 2.4

‍Download image: Connect Alluxio to all your data sources across multiple clouds, single cloud or on-premises using self-guided wizards

Download image: Monitor the status of an Alluxio cluster anywhere from an intuitive UI

Tweet this: @Alluxio accelerates scale, introduces new Data Orchestration Hub #OpenSource #Analytics #BigData #Cloud https://bit.ly/33P3spg