Announcing the first Data Orchestration Summit in November 2019! This Summit brings together data engineers, cloud engineers, data scientists, and industry thought leaders who are solving data problems at the intersection of cloud, AI, and data.
Bring your data to compute with open source
Data orchestration for analytics and machine learning in the cloud
Alluxio enables compute
Bring your data close to compute.
Make your data local to compute workloads for Spark caching, Presto caching, Hive caching and more.
Make your data accessible.
No matter if it sits on-prem or in the cloud, HDFS or S3, make your files and objects accessible in many different ways.
Make your data as elastic as compute.
Effortlessly orchestrate your data for compute in any cloud, even if data is spread across multiple clouds.
alluxio for data engineers
Are your Presto/Spark queries slow on S3?
Do your Presto/Spark queries have inconsistent performance?
Are your metadata operations slow on S3?
Are your egress costs too high?
alluxio for data architects
Can you share data across your app framework?
Do you have problems running remote/multiple storage systems?
Is running HDFS in the cloud for temporary storage expensive?
Do you have the directive to use cloud for analytics?
Interact with Alluxio in any stack
Pick a compute. Pick a storage. Alluxio just works.
// Using Alluxio as input and output for RDD scala> sc.textFile("alluxio://master:19998/Input") scala> rdd.saveAsTextFile("alluxio://master:19998/Output") // Using Alluxio as input and output for Dataframe scala> df = sqlContext.read.parquet("alluxio://master:19998/Input.parquet") scala> df.write.parquet("alluxio://master:19998/Output.parquet”)
-- Pointing Table location to Alluxio hive> CREATE TABLE u_user ( userid INT, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LOCATION 'alluxio://master:port/table_data';
Create and Query table stored in Alluxio hbase(main):001:0> create 'test', 'cf' hbase(main):002:0> list ‘test'
# Accessing Alluxio after mounting Alluxio service to local file system $ ls /mnt/alluxio_mount $ cat /mnt/alluxio_mount/mydata.txt
Featured Use Cases and Deployments
Get in-memory access caching Spark and Presto data on AWS S3, Google Cloud Platform, or Microsoft Azure.
Zero-copy hybrid bursting with no app changes to intelligently burst processing to the cloud.
Accelerate your Spark, Presto, and Tensorflow workloads for object stores on-premise or in the cloud.
powered by alluxio
The selected companies come from our massive data set of vendors and industry metrics. Yes, we use machine learning to analyze the industry in a detailed manner to determine a ranking for this list.
KubeCon 2019 is shaping up to be a fantastic event! And the number of sponsoring vendors at this year’s show is extremely impressive as the show continues to grow in scope and size! One of the companies that will be on VMblog’s MUST SEE list this year at the event is Alluxio.
Learn how to set up EMR Spark with Alluxio so Spark jobs can seamlessly read from and write to S3. See the performance comparison … Continued
Alluxio is a new layer on top of under storage systems that can not only improve raw I/O performance but also enables applications flexible options to read, write and manage files. This article focuses on describing different ways to write files to Alluxio, realizing the tradeoffs in performance, consistency, and also the level of fault tolerance compared to HDFS.
Cloud has changed the dynamics of data engineering in many ways, from changing expectations of on-demand platform services to the popularity of the object … Continued