Tag: scale

Testing Distributed System at Scale for the Cost of a Large Pizza on AWS

Building distributed systems is no small feat. Software testing is just one of many critical practices that engineers who build these systems need to utilize to ensure the quality and usability of their software. For distributed systems, scaling out testing frameworks to ensure that enterprises who run our in highly distributed environments is a complicated (and expensive task!)

Tags: , , , ,

Efficient Data Engineering with Apache Spark, Hive, and Alluxio on S3

Alluxio Meetup | Austin *

Welcome to the first event of the Cloud, Data, & Orchestration Austin Meetup! This meetup will feature two talks and an opportunity to engage with other data engineers, developers, and Alluxio users. Thanks to Bazaarvoice for hosting!

Building fast and scalable big data and ML platforms at Pinterest and JD.com

by Calvin Jia & Yongsheng Wu [Pinterest]

This talk shares our design, implementation and optimization of Alluxio metadata service to address the scalability challenges, focusing on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc.

Tags: , , , , , , ,

Summertime themed In-Memory Computing extravaganza! (cross-post)

New York Meetup *

[Talk 1] A “how-to” presentation for building a real-time alerting, analytics and reporting system (at scale). With Denis Magda, vice president of the Apache Ignite PMC and director of product management at GridGain Systems. And Viktor Gamov, developer advocate at Confluent.
[Talk 2] Using In-Memory technology for real time analytics. With Andy Rivenes is a Product Manager at Oracle for Database In-Memory.
[Talk 3] Feeding data to the Kubernetes beast: bringing data locality to your containerized big data workloads. With Bin Fan, founding engineer of Alluxio, Inc. and PMC member of Alluxio open source project.

Alluxio (formerly Tachyon): Open Source Memory Speed Virtual Distributed Storage System

Data by the Bay San Francisco *

The goal is to make Alluxio accessible to an even wider set of users through a focus on security, new language bindings, and further increased stability. In addition, the team is working on new APIs to allow applications to access data more efficiently and manage data across different under storage systems.

1st Beijing Alluxio (Formerly Tachyon) Meetup

Beijing Meetup *

In the active community development of the past year, Alluxio has greatly improved its read and write performance, scalability and user experience. In addition, in terms of functionality, Alluxio has added a number of new features, such as scalable tiered storage, transparent UFS data reading and writing, unified namespaces, and more. These features bring more value to Alluxio users and more efficient and convenient cluster storage management.