spark Archives | Alluxio

Advancing GPU Analytics with RAPIDS Accelerator for Spark and Alluxio

April 27, 2021

RAPIDS is a set of open source libraries enabling GPU aware scheduling and memory representation for analytics and AI. Spark 3.0 uses RAPIDS for GPU computing to accelerate various jobs including SQL and DataFrame. With compute acceleration from massive parallelism on GPUs, there is a need for accelerating data access and this is what Alluxio enables for compute in any cloud. In this talk, you will learn how to use Alluxio and Spark with RAPIDS Accelerator on NVIDIA GPUs without any application changes.

Tags: alluxio day, analytics, gpu, NVIDIA, RAPIDS, spark

Accelerating Analytics and AI with Alluxio and NVIDIA GPUs

March 23, 2021 By Dong Meng, Adit Madan and David Zhu

Data processing is increasingly making use of NVIDIA computing for massive parallelism. Advancements in accelerated compute mean that access to storage must also be quicker, whether in analytics, artificial intelligence (AI), or machine learning (ML) pipelines.

Accelerating Data Computation on Ceph Objects using Alluxio

December 13, 2020

In this talk, we will present how using Alluxio computation and storage ecosystems can better interact benefiting of the “bringing the data close to the code” approach. Moving away from the complete disaggregation of computation and storage, data locality can enhance the computation performance. During this talk, we will present our observations and testing results that will show important enhancements in accelerating Spark Data Analytics on Ceph Objects Storage using Alluxio.

Tags: ceph, compute, data orchestration, data orchestration summit, spark, storage

Speeding Up Spark Performance using Alluxio at China Unicom

December 13, 2020

Unicom’s traditional batch architecture consists mainly of IOE, Hive, and Greenplum systems. With the development of business, a large number of computing application modules based on diverse scenarios, chimney-like, decentralized applications have emerged. To solve the problem of resource fragmentation, we have introduced a unified computing platform for computing ecology with Spark and Alluxio as the core. Alluxio plays an important role in accelerating data processing and ensuring process stability.

Tags: data orchestration, data orchestration summit, spark

How to teach your data scientist to leverage an analytics cluster with Presto, Spark, and Alluxio

December 13, 2020

This is an open source community conference focused on the key data engineering challenges and solutions around building cloud-native data and AI platforms using latest technologies such as Alluxio, Apache Spark, Apache Airflow, Presto, Tensorflow, and Kubernetes.

Tags: analytics, data orchestration, data orchestration summit, presto, spark

Hybrid Data Lake Architecture with Presto & Spark in the cloud accessing on-prem storage

September 29, 2020

In this talk, we describe the architecture to migrate analytics workloads incrementally to any public cloud (AWS, Google Cloud Platform, or Microsoft Azure) directly on on-prem data without copying the data to cloud storage.

Tags: cloud, data analytics, data lake, hdfs, hybrid, on-prem, presto, spark, storage

Reducing large S3 API costs using Alluxio

July 30, 2020 By Juraj Pohanka (datasapiens), Koen Michiels (datasapiens) and Sam Gilbert (datasapiens)

This article described how engineers at datasapiens brought down S3 API costs by 200x by implementing Alluxio as a data orchestration layer between S3 and Presto.

Tag: spark