Alluxio AI Infra Day 2024

Alluxio is the world's first memory-speed virtual distributed storage system that bridges applications and underlying storage systems, providing unified data access orders of magnitudes faster than existing solutions. The Hadoop Distributed File System (HDFS) is a distributed file system for storing large volumes of data. HDFS popularized the paradigm of bringing computation to data and the co-located compute and storage architecture. In this blog, we highlight two key benefits Alluxio brings to a compute cluster co-located with HDFS.

This is an excerpt from the Accelerating On-Demand Data Analytics with Alluxio whitepaper, which includes a detailed implementation guide in addition to this high level overview. In the Big Data world, it is often the case that only a subset of the total data is relevant for answering the question at hand. As a result, the total cost of ownership for long running clusters for analytics is high while utilization is low, especially when adopting an architecture of co-locating compute and storage.

Qunar Performs Real-Time Data Analytics up to 300x Faster with Alluxio

Alluxio 1.1 release includes many great features and improvements from the community. Alluxio would not be what it is today without the growing open source community, and we would like to thank everyone involved in this project. This release brings many new features and improvements, and in this post, we will highlight a few of the developments: performance improvements, access control features, and usability and integration improvements.

We’re excited to announce that the Alluxio open source project is adopting the Benevolent Dictator For Life (BDFL) model. The day-to-day management of the project will be carried out by the Project Management Committee (PMC). Within the PMC, there are Maintainers, who are responsible for upholding the quality of the code in their respective components. With the adoption of a project management mechanism, we believe it will further accelerate the project growth and enable contributors around the world to easily collaborate to bring exciting new features and improvements to Alluxio.

Alluxio provides Spark with a reliable data sharing layer, enabling Spark to excel at performing application logic while Alluxio handles storage.

Baidu Queries Data 30 Times Faster with Alluxio

Making the Impossible Possible with Alluxio: Accelerate Spark Jobs from Hours to Seconds

Today, we are very excited to announce the 1.0 release of Alluxio, the world’s first memory-centric virtual distributed storage system, which unifies data access and bridges computation frameworks and underlying storage systems. Applications only need to connect with Alluxio to access data stored in any underlying storage systems. Additionally, Alluxio’s memory-centric architecture enables data access orders of magnitude faster than existing solutions.

In this blog, Greg Lindstrom, Vice President of ML Trading at Blackout Power Trading, an electricity trading firm in North American power markets, shares how they leverage Alluxio to power their offline feature store. This approach delivers multi-join query performance in the double-digit millisecond range, while maintaining the cost and durability benefits of Amazon S3 for persistent storage. As a result, they achieved a 22 to 37x reduction in large-join query latency for training and a 37 to 83x reduction in large-join query latency for inference.

‍

International Data Corporation (IDC) reported that the global datasphere will grow from 33 zettabytes in 2018 to 175 zettabytes by 20251. This trend becomes more and more complicated with the variety and velocity of data growth, and it continuously changes the ways data is collected, stored, processed, and analyzed. New analytics solutions, including machine learning, deep learning, and artificial intelligence (AI), and new architectures and tools are being developed to extract and deliver value from the huge datasphere.

Today, many organizations are running a multitude of data-driven applications and data platforms that span multiple geographic regions and across heterogeneous environments – public, private, hybrid, or multi-cloud. Further, the trend of separating compute resources from storage resources makes it easier to scale compute and storage independently, allowing organizations to keep up with new trends in data analytics and AI. In response, more organizations are modernizing their data platforms to meet their needs.

Intel Deep Learning (DL) Boost with BFloat16 (BF16) demonstrates benefits across deep learning training workloads with the same accuracy as 32-bit floating-point (single-precision) (FP32). Recently Amazon introduced EC2 M6i instances powered by the latest-generation Intel Xeon Scalable Processors. Intel and Alluxio collaborate to measure a 20-25% price/performance improvement over the prior generation for machine learning models with PyTorch on AWS. This collaboration demonstrates data preprocessing and training at lower cost on CPUs using Alluxio as the data access layer to cloud storage.

This datasheet introduces the Presto + Alluxio Solution. Alluxio enables caching for Presto as well as hybrid deployments.

Many organizations have taken advantage of the scalability and cost-savings of cloud computing as well as cloud storage services to meet their data-powered workload demands. In addition, as data is increasingly siloed and lives everywhere, there’s a need for data orchestration to bring the needed data closer to compute. With Alluxio’s data orchestration platform, bring back data locality for your compute with in-memory & tiered data access.

Key Benefits:

• Cache data from S3 for Spark, Presto or Hive co-locating it on the same instance as compute
• Scale analytics workloads directly on remote, on-prem data without copying and syncing data into the cloud
• Improve performance with better data locality and get HDFS & S3 compatible data access layer on AWS EMR automatically synced with S3.