• Search
  • Sign In
  • Blog
  • Docs
  • EN
  • GitHub
  • Slack
Alluxio
  • Why Data Orchestration
  • Product
    • Alluxio Overview
    • Alluxio on AWS
    • Alluxio on GCP
    • Presto with Alluxio
    • Spark with Alluxio
    • Alluxio + Intel
    • Alluxio + NetApp
  • Use Cases
    • Zero-copy Hybrid Bursting
    • Zero-copy Burst Across Datacenters
    • Cloud Analytics Caching
    • Accelerated Workloads for Object Stores
  • Community
    • Alluxio Community
    • Powered by Alluxio
    • Data Orchestration Summit
    • Alluxio Day
    • Product School
    • Meetups & Conferences
    • Newsletter
  • Enterprise
    • Editions
    • Pricing
    • Contact Us
  • Resources
    • Downloads
    • Documentation
    • FAQ
    • Learning Center
    • Videos
    • Tech Talks
    • Slides from Talks
    • White Papers
    • Case Studies
    • Solution Briefs
    • Events
  • Try Alluxio

Scalable and Highly-available Distributed File System Metadata Service Using gRPC, RocksDB and RAFT

April 7, 2020

Tags: alluxio engineering, distributed systems, grpc, metadata service, office hour, raft, rocksdb

ALLUXIO COMMUNITY OFFICE HOUR

Alluxio (alluxio.io) is an open-source data orchestration system that provides a single namespace federating multiple external distributed storage systems. It is critical for Alluxio to be able to store and serve the metadata of all files and directories from all mounted external storage both at scale and at speed. 

This talk shares our design, implementation, and optimization of Alluxio metadata service (master node) to address the scalability challenges. Particularly, we will focus on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc. As a result of the combined above techniques, Alluxio 2.0 is able to store at least 1 billion files with a significantly reduced memory requirement, serving 3000 workers and 30000 clients concurrently.

In this Office Hour, we will go over how to:

  • Metadata storage challenges
  • How to combine different open source technologies as building blocks
  • The design, implementation, and optimization of Alluxio metadata service

Speaker:

Bin Fan is the founding engineer and VP of Open Source at Alluxio, Inc. Prior to Alluxio, he worked for Google to build the next-generation storage infrastructure. Bin received his Ph.D. in Computer Science from Carnegie Mellon University on the design and implementation of distributed systems.

Questions? Slack with the speakers, users, and many other community members!
Welcome to join Alluxio Global Online Meetup Group to attend online meetups like this!

Video:

Slides:

Scalable and High available Distributed File System Metadata Service Using gRPC, RocksDB and RAFT from Alluxio, Inc.

  • Resources
    • Blog
    • White Papers
    • Tech Talks
    • Case Studies
    • Events
    • Slides from talks
    • Videos
  • Open Source
    • Community
    • Download
    • Mailing List
    • Slack
    • Powered By Alluxio
    • Newsletter
  • Support
    • Documentation
    • Account Sign In
    • Pricing
    • Services & Support
    • Contact Us
  • Company
    • About
    • Careers
    • News & Press
    • Awards
    • Partners

© Copyright 2023 Alluxio, Inc. All rights reserved.
Alluxio is a trademark of Alluxio, Inc.
Terms of Service | Privacy Policy

Newsletter Signup