Apache Hudi : The Path Forward
deep dive into two important areas of active development going forward – table metadata management and caching.
Tags: alluxio day, apache hudi, caching, data lake, metadata management
Tag: caching
Alluxio-FUSE as a data access layer for Dask
At Aspect Analytics we intend to use Dask, a distributed computation library for Python, to deal with MSI data stored as large tensors. In this talk we explore using Alluxio and Alluxio FUSE as a data consolidation and caching layer for some of our bioinformatics workflows.
Tags: alluxio day, aspect analytics, caching, dask, fuse
Community Office Hour: Improving Memory Utilization of Spark Jobs Using Alluxio
Many Spark users may not be aware of the differences in memory utilization between caching data directly in-memory into the Spark JVM versus storing data off-heap via an in-memory storage service like Alluxio. In this office hour, I will highlight the two approaches with a demo and open up for discussions
Tags: caching, memory, office hour, spark
Ultra-fast SQL Analytics using PAS (Presto on Alluxio Stack)
This talk describes a stack of open-source projects to serve high-concurrent and low-latency SQL queries using Presto with Alluxio on big data in the cloud. Deploying Alluxio as a data orchestration layer to access cloud storage object storage (e.g., AWS S3), this architecture greatly enhances the data locality of Presto with distributed and cross-query caching, thus avoids reading same data repeatedly from the cloud storage.
Tags: caching, cloud storage, data locality, meetup, object storage, presto
Ultra-fast SQL Analytics using PAS (Presto on Alluxio Stack)
Presto Meetup *
Presto is widely used for data science, business analytics, and operations. Presto’s SQL is a main driver for this, as it is ANSI-compliant, easy to ramp-up, and has rich functionality. Given the versatility and flexibility of this software, there is also a huge demand to develop interfaces for other critical data domains like real-time dashboards, stream processing, and large-scale batch computations. We will explore some interesting systems and prototypes to bring Presto to these new domains.
Tutorial: Presto + Alluxio + Hive Metastore on Your Laptop in 10 min
This tutorial guides users to set up a stack of Presto, Alluxio and Hive Metastore on your local server, and it demonstrates how to use Alluxio as the caching layer for Presto queries.
Building a Large-scale Interactive SQL Query Engine using Presto and Alluxio in JD.com
This article describes how JD built this interactive OLAP platform combining two open-source technologies: Presto and Alluxio.
Accelerating Write-intensive Data Workloads on AWS S3
Alluxio is an open-source data orchestration system widely used to speed up data-intensive workloads in the cloud. Alluxio v2.0 introduced Replicated Async Write to allow users to complete writes to Alluxio file system and return quickly with high application performance, while still providing users with peace of mind that data will be persisted to the chosen under storage like S3 in the background.
NetEase and Alluxio joint meetup
Hangzhou Meetup *
Joint meetup in Hangzhou discusses: An introduction to new features of big data storage system Alluxio and optimization of cache performance, Practice & exploration of Spark & Alluxio, and the Interactive query system Impala.