Join us for our 4th Alluxio Day community virtual event featuring speakers from Facebook, TikTok,
Tencent, and Intel.
Announcing Alluxio 2.5 – advanced interfaces and acceleration for analytics and AI/ML pipelines
Now with an accelerated POSIX API for unified storage access, performance, and ease of management
Join us for Alluxio Day IV, a community-driven virtual event on June 24th to hear from Facebook, Tencent Cloud and Intel!
Read about how to evaluate Alluxio’s data orchestration platform as a distributed cache for Apache Spark in a public cloud or on-premises
We’re hiring! Join our team and build the future of data orchestration. See open positions >
Alluxio enables compute
Data locality
Bring your data close to compute.
Make your data local to compute workloads for Spark caching, Presto caching, Hive caching and more.
Data Accessibility
Make your data accessible.
No matter if it sits on-prem or in the cloud, HDFS or S3, make your files and objects accessible in many different ways.
Data On-Demand
Make your data as elastic as compute.
Effortlessly orchestrate your data for compute in any cloud, even if data is spread across multiple clouds.
“zero-copy” burst user spotlight: walmart
Why Walmart chose Alluxio’s “Zero-Copy” burst solution:
- No requirement to persist data into the cloud
- Improved query performance and no network hops on recurrent queries
- Lower costs without the need for creating data copies
See more on how Alluxio powers Walmart’s “zero-copy” burst solution in their presentation >
Featured Use Cases and Deployments
Managing data copies/app changes when bursting compute to cloud?
Zero-copy hybrid bursting with no app changes to intelligently make remote data accessible in the public cloud.
Expanding compute capacity across geo-distributed data centers?
Zero-copy bursting across data centers for Presto, Spark, and Hive with no app changes on data stored in HDFS.
Interact with Alluxio in any stack
Pick a compute. Pick a storage. Alluxio just works.
-- Pointing Table location to Alluxio CREATE SCHEMA hive.web WITH (location = 'alluxio://master:port/my-table/‘)
// Using Alluxio as input and output for RDD
scala> sc.textFile("alluxio://master:19998/Input")
scala> rdd.saveAsTextFile("alluxio://master:19998/Output")
// Using Alluxio as input and output for Dataframe
scala> df = sqlContext.read.parquet("alluxio://master:19998/Input.parquet")
scala> df.write.parquet("alluxio://master:19998/Output.parquet”)
-- Pointing Table location to Alluxio hive> CREATE TABLE u_user ( userid INT, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LOCATION 'alluxio://master:port/table_data';
Create and Query table stored in Alluxio hbase(main):001:0> create 'test', 'cf' hbase(main):002:0> list ‘test'
# Accessing Alluxio after mounting Alluxio service to local file system $ ls /mnt/alluxio_mount $ cat /mnt/alluxio_mount/mydata.txt
ALLUXIO
$ ./bin/alluxio fs mount \ --option aws.accessKeyId=<AWS_ACCESS_KEY_ID> \ --option aws.secretKey=<AWS_SECRET_KEY_ID> \ alluxio://master:port/s3 s3a://<S3_BUCKET>/<S3_DIRECTORY>
$ ./bin/alluxio fs mount \ alluxio://master:port/hdfs hdfs://namenode:port/dir/
$ ./bin/alluxio fs mount \ --option fs.azure.account.key.<AZURE_ACCOUNT>.blob.core.windows.net=<AZURE_ACCESS_KEY> \ alluxio://master:port/azure wasb://<AZURE_CONTAINER>@<AZURE_ACCOUNT>.blob.core.windows.net/<AZURE_DIRECTORY>/
$ ./bin/alluxio fs mount \ --option fs.gcs.accessKeyId=<GCS_ACCESS_KEY_ID> \ --option fs.gcs.secretAccessKey=<GCS_SECRET_ACCESS_KEY> \ alluxio://master:port/gcs gs://<GCS_BUCKET>/<GCS_DIRECTORY>
$ ./bin/alluxio fs mount \ --option aws.accessKeyId=<AWS_ACCESS_KEY_ID> \ --option aws.secretKey=<AWS_SECRET_KEY_ID> \ --option alluxio.underfs.s3.endpoint=http://<rgw-hostname>:<rgw-port> \ --option alluxio.underfs.s3.disable.dns.buckets=true \ alluxio://master:port/ceph s3a://<S3_BUCKET>/<S3_DIRECTORY>
$ ./bin/alluxio fs mount alluxio://master:port/nfs /mnt/nfs
powered by alluxio
What’s Happening
Alluxio’s capabilities as a Data Orchestration framework have encouraged users to onboard more of their data-driven applications to an Alluxio powered data access layer. … Continued
Alluxio 2.5 focuses on improving interface support to broaden the set of data driven applications which can benefit from data orchestration. The POSIX and … Continued
Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data … Continued
This whitepaper details how to evaluate Alluxio’s data orchestration platform as a distributed cache for Apache Spark in a public cloud or on-premises. We … Continued
Data processing is increasingly making use of NVIDIA computing for massive parallelism. Advancements in accelerated compute mean that access to storage must also be quicker, whether in analytics, artificial intelligence (AI), or machine learning (ML) pipelines.