Reducing large S3 API costs using Alluxio
This article described how engineers at datasapiens brought down S3 API costs by 200x by implementing Alluxio as a data orchestration layer between S3 and Presto.
Tag: cloud object storage
Tech Talk: Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack
Join us for this tech talk where we’ll introduce the Starburst Presto, Alluxio, and cloud object store stack for building a highly-concurrent and low-latency analytics platform. This stack provides a strong solution to run fast SQL across multiple storage systems including HDFS, S3, and others in public cloud, hybrid cloud, and multi-cloud environments.
Tags: cloud object storage, cloud storage, presto, storage
Bay Area Meetup: Interactive Analytics in the Cloud with Presto and Alluxio
Bay Area Meetup which include presentations on the architecture of Presto, its separation of compute and storage, cloud-readiness, recent advancements in the project such as Cost-Based Optimizer and Kubernetes Support. Presto and Alluxio production use cases and more.
Tags: alluxio engineering, aws s3, cloud object storage, compute storage separation, meetup, presto
O’Reilly AI Conference Keynote: Data Orchestration for AI, Big Data, and Cloud
Haoyuan Li’s keynote at O’Reilly Beijing discusses open source data orchestration and the value of leveraging Alluxio with rising trends driving the need for a new architecture. Four big trends driving this need: Separation of compute & storage, hybrid-multi cloud environments, rise of object store and self-service data across the enterprise.
Tags: big data, cloud, cloud object storage, cloud storage, compute storage separation, conference, data, data orchestration, hybrid cloud, multi cloud, on-prem object storage, storage
How do you access data from both HDFS and cloud storage?
Problem Sometimes big data analytics need process input data from two different storage systems at the same time. For instance, a data scientists may need to join two tables one from a HDFS cluster and one from S3. Existing Solutions Certain computation frameworks may be able to connect to storage systems including HDFS and popular cloud … Continued
Evolution of big data stacks under computational and storage separation architecture
Shanghai *
A new generation of open source big data, represented by Alluxio, born at the University of California at Berkeley, looks at this issue. Different from systems such as designing storage tight coupling to achieve low-cost reliable storage HDFS, by providing a virtual data storage layer defined and implemented by software for data applications, abstracting and integrating cloudy, hybrid cloud, multi-data center and other environments The underlying files and objects, and through intelligent workload analysis and data management, make data close to computing and provide data locality, big data and machine learning applications can be achieved with the same performance and lower cost.
Accelerate and Scale Big Data Analytics and Machine Learning Pipelines with Disaggregated Compute and Storage
Alluxio | SwiftStack Tech Talk *
Enterprises are increasingly looking towards object stores to power their big data & machine learning workloads in a cost-effective way. The combination of SwiftStack and Alluxio together, enables users to seamlessly move towards a disaggregated architecture.
Alluxio for Hybrid Cloud | HDFS and AWS S3 demo
Alluxio Community Office Hour *
Alluxio can help data scientists and data engineers interact with different storage systems in a hybrid cloud environment. Using Alluxio as a data access layer for Big Data and Machine Learning applications, data processing pipelines can improve efficiency without explicit data ETL steps and the resulting data duplication across storage systems.
Spark Pipelines in the Cloud with Alluxio
Big Data Day LA *
In this talk, we discuss how Alluxio can be deployed and used with a Spark data processing pipeline in the cloud. We show how pipeline stages can share data with Alluxio memory for improved performance benefits, and how Alluxio can improves completion times and reduces performance variability for Spark pipelines in the cloud.