TWO SIGMA OPEN SOURCE MEETUP
TSOS meetups focus on the open source projects that Two Sigma cares most about, from projects we generated in-house then open sourced to large external open source projects that we depend on to do our work. This time, Wenbo Zhao (Two Sigma) and Bin Fan (Alluxio) will be presenting on how Two Sigma uses Alluxio to make data-intensive compute independent of the storage beneath.
The rise of computation-intensive workloads and the adoption of the cloud has driven organizations to adopt a decoupled architecture for modern workloads — one in which compute scales independently from storage. However, while enabling scaling elasticity, it introduces new problems — how do you co-locate data with compute, how do you unify data across multiple remote data centers, how do you keep storage and I/O egress costs down and many more.
In this meetup, the Wenbo and Bin will present a new approach to making data-intensive compute independent of the storage beneath using open source project Alluxio, an open-source distributed file system, which sits between compute and storage layer. Applications like Apache Spark or TensorFlow can then seamlessly access multiple disparate data sources with consistent performance without knowing where the data is actually persisted.
Wenbo will present why Two Sigma needed to disaggregate compute and storage and how they decided to adopt the Spark + Alluxio + HDFS architecture.
Bin will present a deep dive into the Alluxio open source project, a distributed file system, including the reference architecture, the data & metadata paths to serve requests from compute from remote understores and the compute API’s supported for accessing data from Alluxio.