Deep Dive into the Implementation of Alluxio Metadata Storage
This article introduces the design and implementation of metadata storage in Alluxio Master, either on heap and off heap (based on RocksDB).
Category: Developer and Engineering
What’s New in Alluxio 2.8: Enhanced S3 API Functionality, Enterprise-grade Security and Data Migration With Better Usability and Low Cost
The Alluxio 2.8 version focuses on the S3 API, enterprise-grade security, scalability and observability in data migration. Enhanced S3 API makes managing Alluxio easier than ever. Features such as encryption at rest and policy-driven data management further improve Alluxio’s functionality to support enterprise customers.
From Zookeeper to Raft: How Alluxio Stores File System State with High Availability and Fault Tolerance
Raft is an algorithm for state machine replication as a way to ensure high availability (HA) and fault tolerance. This blog shares how Alluxio has moved to a Zookeeper-less, built-in Raft-based journal system as a HA implementation.
Recommendations to Level Up Your Machine Learning Platform
With machine learning (ML) and artificial intelligence (AI) applications becoming more business-critical, organizations are in the race to advance their AI/ML capabilities. To realize the full potential of AI/ML, having the right underlying machine learning platform is a prerequisite.
Orchestrating Data for Machine Learning Pipelines
This article will discuss a new solution to orchestrating data for end-to-end machine learning pipelines that addresses the above questions. I will outline common challenges and pitfalls, followed by proposing a new technique, data orchestration, to optimize the data pipeline for machine learning.
From Cache to Cash: Introducing NFT for Data Orchestration
Today, we are excited to announce the launch of Non-fungible token (NFT) as a new feature in our leading data orchestration platform.
Improving Presto Architectural Decisions with Alluxio Shadow Cache at Meta (Facebook)
With the collaboration between Meta (Facebook), Princeton University, and Alluxio, we have developed “Shadow Cache” – a lightweight Alluxio component to track the working set size and infinite cache hit ratio. Shadow cache can keep track of the working set size over the past window dynamically and is implemented by a series of bloom filters. Shadow cache is deployed in Meta (Facebook) Presto and is being leveraged to understand the system bottleneck and help with routing design decisions.
How to Become a Contributor to Alluxio Open Source Project
This is a tutorial to guide a newbie to complete a new-contributor task and become an open-source contributor of the Alluxio project.
Self-serve Data Architecture with Presto and Alluxio Across Clouds
This article highlights synergy between the two widely adopted open-source projects, Alluxio and Presto, and demonstrates how together they deliver a self-serve data architecture across clouds.