• Latest Blog:
  • Search
  • Sign In
  • Blog
  • Docs
  • EN
  • GitHub
  • Slack
Alluxio
  • Why Alluxio?
  • Product
    • Alluxio Overview
    • Alluxio on AWS
    • Alluxio on GCP
    • Trino with Alluxio
    • Presto with Alluxio
    • Spark with Alluxio
    • Alluxio + Intel
    • Alluxio + NetApp
  • Use Cases
    • Zero-copy Hybrid Bursting
    • Zero-copy Burst Across Datacenters
    • Cloud Analytics Caching
    • Accelerated Workloads for Object Stores
  • Community
    • Alluxio Community
    • Powered by Alluxio
    • Data Orchestration Summit
    • Alluxio Day
    • Product School
    • Meetups & Conferences
    • Newsletter
  • Resources
    • Downloads
    • Documentation
    • FAQ
    • Learning Center
    • Videos
    • Tech Talks
    • Slides from Talks
    • White Papers
    • Case Studies
    • Solution Briefs
    • Events
  • Company
    • About
    • Careers
    • News & Press
    • Awards
    • Partners
  • Enterprise
    • Editions
    • Pricing
    • Contact Us
  • Get Started
  • Contact Us
Tech Talk Slide Deck

Building fast and scalable big data and ML platforms at Pinterest and JD.com

June 21, 2019

By Calvin Jia & Yongsheng Wu [Pinterest]

Tags: aws s3, data, machine learning, meetup, metadata management, performance, scale, tiered storage

ALLUXIO BAY AREA MEETUP

Scalable Filesystem Metadata Services

This talk was presented by Alluxio’s top contributor and PMC Maintainer Calvin Jia at the Alluxio bay area Meetup. 

This talk shares our design, implementation and optimization of Alluxio metadata service to address the scalability challenges, focusing on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc.

Questions? Slack with the speakers, users, and many other community members!
Welcome to join Alluxio Bay Area Meetup Group to attend online meetups like this!
Alluxio – Scalable Filesystem Metadata Services from Alluxio, Inc.

Big Data Machine Learning Platform at Pinterest

This was presented by the Yongsheng Wu, head of big data and ML platform at Pinterest, at the Alluxio bay area meetup. 

Yongsheng shares Pinterest’s journey to build a fast and scalable big data and ML platform in AWS for Pinterest to handle the requests and complexity in data at scale. In this talk, he will cover different aspects from the requirements of the platform, the challenges encountered, the technologies chosen, and the tradeoffs that were made.

Pinterest – Big Data Machine Learning Platform at Pinterest from Alluxio, Inc.
  • Resources
    • Blog
    • White Papers
    • Tech Talks
    • Case Studies
    • Events
    • Slides from talks
    • Videos
  • Open Source
    • Community
    • Download
    • Mailing List
    • Slack
    • Powered By Alluxio
    • Newsletter
  • Support
    • Documentation
    • Account Sign In
    • Pricing
    • Services & Support
    • Contact Us
  • Company
    • About
    • Careers
    • News & Press
    • Awards
    • Partners

© Copyright 2023 Alluxio, Inc. All rights reserved.
Alluxio is a trademark of Alluxio, Inc.
Terms of Service | Privacy Policy

Newsletter Signup