On-Demand Videos

The hybrid cloud model, where cloud resources run Spark or Presto jobs against data stored on-premises, is an appealing solution to reduce resource contention in on-premise environments while also saving in overall costs. One key flaw in a hybrid model is the overhead associated with transferring data between the two environments. Data and metadata locality within the compute application must be achieved in order to maintain the similar performance of analytics jobs as if the entire workload was run on-premises.

In this office hour, we demonstrate how a “zero-copy burst” solution helps to speed up Spark and Presto queries in the public cloud while eliminating the process of manually copying and synchronizing data from the on-premise data lake to cloud storage. This approach allows compute frameworks to decouple from on-premise data sources and scale efficiently by leveraging Alluxio and public cloud resources such as AWS.

We will cover:

• Typical challenges of moving data to the cloud and expanding compute capacity.
• Details about “zero-copy” hybrid cloud solution for burst computing
• A demo of running Presto analytic queries using remote on-prem HDFS data with Alluxio deployed in AWS EMR

ALLUXIO TECH TALK

As the amount of data analyzed and stored continues to grow exponentially, fixed on-premises infrastructure like Apache Hadoop data lakes becomes costly. Add to that the need to support newer and popular frameworks on an already busy data lake, it is not uncommon to see Hadoop-based data lakes running at beyond 100% utilization and hybrid processing split between physical and cloud infrastructure. As a result, companies are looking to leverage the flexibility and cost savings of the cloud.

Join us for this tech talk where we will show you how Alluxio can help burst your private computing environment to Google Cloud, minimizing costs and I/O overhead. Alluxio coupled with Google’s open source data and analytics processing engine, Dataproc, enables zero-copy burst for faster query performance in the cloud so you can take advantage of resources that are not local to your data, without the need for managing the copying or syncing of that data.

We’ll also show a demo on how to get up and running with Alluxio and Dataproc, including how to:

• Setup your hybrid environment between your private datacenter and Google Cloud Platform
• Burst a Spark based machine learning algorithm to Dataproc while accessing on-prem data
• Scale analytic workloads directly on data on-prem without copying and synchronizing the data into the cloud

ALLUXIO COMMUNITY OFFICE HOUR

Today’s conventional wisdom states that network latency across the two ends of a hybrid cloud prevents you from running analytic workloads in the cloud with the data on-prem. As a result, most companies copy their data into a cloud environment and maintain that duplicate data. All of this means that it is challenging to make both on-prem HDFS data accessible with the desired application performance.

In this talk, we will show you how to leverage any public cloud (AWS, Google Cloud Platform, or Microsoft Azure) to scale analytics workloads directly on on-prem data without copying and synchronizing the data into the cloud.

In this Office Hour, we will go over:

• A strategy to embrace the hybrid cloud, including an architecture for running ephemeral compute clusters using on-prem HDFS.
• An example of running on-demand Presto, Spark, and Hive with Alluxio in the public cloud.
• An analysis of experiments with TPC-DS to demonstrate the benefits of the given architecture.

ALLUXIO COMMUNITY OFFICE HOUR

Alluxio (alluxio.io) is an open-source data orchestration system that provides a single namespace federating multiple external distributed storage systems. It is critical for Alluxio to be able to store and serve the metadata of all files and directories from all mounted external storage both at scale and at speed.

This talk shares our design, implementation, and optimization of Alluxio metadata service (master node) to address the scalability challenges. Particularly, we will focus on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc. As a result of the combined above techniques, Alluxio 2.0 is able to store at least 1 billion files with a significantly reduced memory requirement, serving 3000 workers and 30000 clients concurrently.

In this Office Hour, we will go over how to:

• Metadata storage challenges
• How to combine different open source technologies as building blocks
• The design, implementation, and optimization of Alluxio metadata service

ALLUXIO COMMUNITY OFFICE HOUR

Presto, an open-source distributed SQL engine, is commonly used to query an existing Hive data warehouse. Due to existing applications, tech debt or operational challenges in the past, Presto may not be able to achieve its full potential but bound and limited by the past decisions. Particularly, challenges include overloaded Hive Metastore with slow and unpredictable access, unoptimized data formats and layouts such as too many small files, or lack of influence over the existing Hive system and other Hive applications.

Ideally, Presto would access data independently from how the data was originally stored or managed. Alluxio, as a data orchestration layer provides the physical data independence, for Presto to interact with the data more efficiently. In addition to caching for IO acceleration, Alluxio also provides a catalog service to abstract the metadata in the Hive Metastore, and transformations to expose the data in compute-optimized way. In this talk, we describe some of the challenges of using Presto with Hive, and introduce Alluxio data orchestration for solving those challenges.

In this Office Hour, we will go over:

• Typical challenges of using Presto with Hive
• Overview of the different services of Alluxio Structured Data Management in Alluxio 2.1
• A demo of using Alluxio Structured Data Management with Presto

ALLUXIO COMMUNITY OFFICE HOUR

Accessing data to run analytic workloads in Spark across data centers and/or clouds can be challenging. Additionally, network I/O can bottleneck Spark jobs that need to read a large amount of data. A common solution is to deploy an HDFS cluster closer to Spark as a caching layer and manually copy the input data to HDFS first, purging it afterward. But this ETL process can be both time-consuming and also error-prone.

A more efficient and simpler solution is to run Spark on Alluxio as a distributed cache on top of the remote data source. While caching data transparently based on access patterns and storing the working set closer, Alluxio provides Spark jobs much higher I/O throughput with enhanced data locality. In addition, Alluxio also provides data accessibility and abstraction for deployments in hybrid and multi-cloud environments.

In this Office Hour, we will go over how to:

• Burst on-prem Spark workloads to the cloud with Alluxio so Spark can seamlessly read from and write to remote data storage
• Use Alluxio as the input/output for Spark applications
• Save and load Spark RDDs and Dataframes with Alluxio

ALLUXIO COMMUNITY OFFICE HOUR

Building distributed systems is no small feat. Software testing is just one of many critical practices that engineers who build these systems need to utilize to ensure the quality and usability of their software. For distributed systems, scaling out testing frameworks to ensure that enterprises who run our in highly distributed environments is a complicated (and expensive task!)

In this online meetup, you will learn about:

• How the engineers at Alluxio have approached testing at scale
• Approaches to maintaining distributed systems at scale

ALLUXIO COMMUNITY OFFICE HOUR

Many organizations are leveraging EMR to run big data analytics on public cloud. However, reading and writing data to S3 directly can result in slow and inconsistent performance. Alluxio is a data orchestration layer for the cloud, and in this use case it caches data for S3, ensuring high and predictable performance as well as reduced network traffic.

In this office hour, you will learn about:

• How to set up Alluxio with the EMR stack so that Presto jobs can seamlessly read from and write to S3
• Compare the performance between Presto on EMR with Presto and Alluxio on EMR
• Open Session for discussion on any topics such as solving the separation of compute and storage problem, and more

With the advent of the public clouds and data increasingly siloed across many locations — on premises and in the public cloud — enterprises are looking for more flexibility and higher performance approaches to analyze their structured data.

Join us for this tech talk where we’ll introduce the Starburst Presto, Alluxio, and cloud object store stack for building a highly-concurrent and low-latency analytics platform. This stack provides a strong solution to run fast SQL across multiple storage systems including HDFS, S3, and others in public cloud, hybrid cloud, and multi-cloud environments. You’ll learn more about:

• The architecture of Presto, an open source distributed SQL engine
• How the Presto + Alluxio stack queries data from cloud object storage like S3 for faster and more cost-effective analytics
• Achieving data locality and cross-job caching with Alluxio regardless of where data is persisted

‍

feat. Apple Case Study using Tensorflow, NFS, DC/OS, and Alluxio

ALLUXIO ONLINE MEETUP

Data scientists or platform engineers often face the following challenge when the input data for machine learning jobs are stored in remote storage like NFS or cloud storage like S3. Making direct data access is slow, unstable and expensive; manually duplicating data to the training clusters also introduces large overhead, complicated data curation and often requires engineers to build ETL pipelines.

This talk will guide the audience on how Alluxio can greatly simplify the data preparation phase in with remote and possibly multiple data sources. We will share the lessons and benchmark from Bill Zhao an engineer led in Apple when building a Machine Learning platform using Tensorflow, NFS, DC/OS and Alluxio.

In this online meetup, you will learn about:

• When Alluxio can help for machine learning platform;
• How to setup and create POSIX endpoint for Alluxio service to unify the file system data access to S3, HDFS and Azure blob storage;
• How to run TensorFlow to train models backed by accessing remote input data like access local file system.

ALLUXIO COMMUNITY OFFICE HOUR

Users deploy Alluxio in a wide range of use cases from analytics to AI platforms, for Alluxio’s unified access to data and transparent caching for acceleration. However, many frameworks are SQL engines, like Presto, Apache Spark SQL, or Apache Hive, and consume data structured as tables of rows and columns. Since Alluxio is commonly used as a filesystem of files and directories, there is a mismatch between how Alluxio exposes data (files, directories), and how SQL engines deal with data (tables, rows, columns). This gap creates various challenges and inefficiencies.

Therefore, in the Alluxio 2.1 release, we introduce Alluxio Structured Data Management, which is a new set of services that enables structured data applications to interact with data more efficiently. The new services include the catalog service and a transformation service, which all work together to bridge the gap between storage and SQL engines and enable physical data independence.

In this office hour, we introduce the concepts and components of Alluxio Structured Data Management, and go through a demo with Presto.

In this Office Hour we’ll go over:

• Introduction and motivation of Alluxio Structured Data Management
• Overview of the different services of Alluxio Structured Data Management in Alluxio 2.1
• A demo of using Alluxio Structured Data Management with Presto

ALLUXIO COMMUNITY OFFICE HOUR

While adoption of the Cloud & Kubernetes has made it exceptionally easy to scale compute, the increasing spread of data across different systems and clouds has created new challenges for data engineers. Effectively accessing data from AWS S3 or on-premises HDFS becomes harder and data locality is also lost – how do you move data to compute workers efficiently, how do you unify data across multiple or remote clouds, and many more. Open source project Alluxio approaches this problem in a new way. It helps elastic compute workloads, such as Apache Spark, realize the true benefits of the cloud while bringing data locality and data accessibility to workloads orchestrated by Kubernetes.

One important performance optimization in Apache Spark is to schedule tasks on nodes with HDFS data nodes locally serving the task input data. However, more users are running Apache Spark natively on Kubernetes where HDFS is not an option. This office hour describes the concept and dataflow with respect to using the stack of Spark/Alluxio in Kubernetes with enhanced data locality even if the storage service is outside or remote.

In this Office Hour we’ll go over:

• Why Spark is able to make a locality-aware schedule when working with Alluxio in K8s environment using the host network
• Why a pod running Alluxio can share data efficiently with a pod running Spark on the same host using domain socket and host path volume
• The roadmap to improve this Spark / Alluxio stack in the context of K8s