The Alluxio team is excited to announce the release of Alluxio 2.7 for both the free open source Community Edition and Alluxio Enterprise Edition, generally available now.
Alluxio 2.7 highlights the functionality and performance of machine learning and AI training workloads, resulting in 8-12x improved I/O efficiency at a significantly lower cost. Alluxio 2.7 also greatly enhances the scalability of deployment of large clusters as well as loading of large datasets.
With this release, Alluxio has strengthened its position as a de-facto data unification and acceleration solution in data analytics and machine learning pipelines. The solution is optimized to support Spark, Presto, Tensorflow, and PyTorch, and is available on multiple cloud platforms such as AWS, GCP, and Azure Cloud, and also on Kubernetes in private data centers or public clouds.
You can download a copy of the free Alluxio Community Edition or a trial of the full Alluxio Enterprise Edition here. You can also join our community slack channel to provide your feedback. We want to thank the community for their valuable contributions to the Alluxio 2.7 release.
Deeper Integration with Machine Learning Workloads
The Alluxio platform has been continuously optimized for AI and ML applications which include a data loading and preprocessing step as a precursor to model training and inference. By accelerating I/O heavy stages and allowing parallel processing of the following compute-intensive training, end-to-end training on the Alluxio data platform observes huge performance gains over traditional solutions. Alluxio 2.7 provides support for a large number of small files and highly concurrent data access, proven to achieve 8-12x improved I/O efficiency for ML training at a significantly lower cost.
Improved Scalability from Deployment to Data Loading
Alluxio platform now supports a massive scale of data management. As the use of Alluxio has grown for compute and storage spanning multiple geographical locations, the platform continues to evolve and scale. Alluxio 2.7 provides a significant improvement in scalability, all the way from deploying Alluxio on clusters with thousands of nodes to loading data sets with billions of files in production workloads. The new “Worker Registration Lease” feature enables launching clusters with a large number of workers. The new “Batched Job” feature reduces the resource requirements for the management controller, supporting data loading at scale. These improved scalabilities further lower the cost of provisioned infrastructure.
Ease of Use on Kubernetes
Alluxio now includes a preview of a native Container Storage Interface (CSI) Driver for Kubernetes, as well as a Kubernetes operator for ML making it easier than ever before to operate ML pipelines on the Alluxio platform in containerized environments. The Alluxio volume type is now natively available for Kubernetes environments. Alluxio 2.7 makes major improvements in Helm charts, CSI, and Fluid integration which help deploy Alluxio in Kubernetes clusters co-located with training jobs, bringing agility and ease-of-use.
More Info
For a comprehensive list of major features of Alluxio 2.7, please refer to the Alluxio 2.7.0 official release notes.
Any questions? Join the Community Slack Channel or schedule a demo today.