Lenovo is an Alluxio customer with a common problem and use case in the world of data analytics. They have petabytes of data in multiple data centers in different geographic locations. Analyzing it requires an ETL process to get all of the data in the right place. This is both slow, because data has to be transferred across the network, and costly because multiple copies of the data need to be stored. Freshness and quality of the data can also suffer as the data is also potentially out of date and incomplete because regulatory issues prevent certain data from being transferred. Lenovo deployed Alluxio to address these issues and eliminate the need for ETL altogether. Data from multiple locations is cached in Alluxio and multiple applications can access it for analysis. The data is either accessed locally from memory or fetched by Alluxio for new requests as needed from persistent storage. The working data set is always available without the need for ETL. Alluxio fits within the existing security frameworks and enforces the policies in place, ensuring regulatory and compliance requirements from different countries and jurisdictions are met. You can learn more in the full case study
.png)
Blog

Alluxio’s MLPerf Storage v3.0 results demonstrate that organizations can keep persistent AI data in Amazon S3 while delivering high-performance training and checkpointing through a distributed cache close to compute. Alluxio achieved 147.03 GiB/s checkpoint write bandwidth for Llama 3 405B and near-linear scaling from one to 32 workers.

