In this talk, we will focus on Alluxio design, its architecture, data flow and metadata flow. We will dive into the choices in its design space and share the experiences when implementing features like data tiering, storage options and cache eviction policies. We will also share our lessons in design, implementation and operation when working to build an open source distributed storage systems with 900 contributors for 5+ years.
Enterprises are increasingly looking towards object stores to power their big data & machine learning workloads in a cost-effective way. The combination of SwiftStack and Alluxio together, enables users to seamlessly move towards a disaggregated architecture.
Alluxio can help data scientists and data engineers interact with different storage systems in a hybrid cloud environment. Using Alluxio as a data access layer for Big Data and Machine Learning applications, data processing pipelines can improve efficiency without explicit data ETL steps and the resulting data duplication across storage systems.
This technical salon will focus on big data, storage, database and Alluxio application practice, and invite Tencent technical experts and industry technical experts to share the basic principles of Alluxio system, big data system architecture, database application operation and maintenance, AI computer. Themes such as visual technology and landing practice bring rich practical content and experience exchange.
What’s Spark+AI Summit? It’s the world’s largest conference that is focused on Apache Spark – Alluxio’s older cousin open source project from the same lab (UC Berkeley’s AMPLab – now RISElab).
Open source software always plays critical role in software development. From Linux kernel to TensorFlow, it drives a lot of awesome projects which created trend and led direction of technology.
We are pleased to have several experts, Reynold Xin, Dongxu Huang, Qing Han, Bin Fan, Amelia Wong, etc. who will share the technology and stories on their successful open source project.
Tachyon is a memory-centric fault-tolerant distributed storage system, which enables reliable file sharing at memory-speed. It originated from AMPLab, UC Berkeley in 2012, the same lab produced Apache Mesos and Apache Spark. Soon later, it became an open source project and is deployed at many companies. Since then, Tachyon has attracted more than 200 contributors from over 50 institutions. In 2015, company Tachyon Nexus was founded to further accelerate the development of Tachyon. In this talk, we will review Tachyon’s new features, deployments, and developments in 2015, and look into 2016.
During the past several years, Spark has significantly changed the landscape of big data computing. It improves performance of various applications dramatically. However, in certain Spark use cases, the bottleneck is in the I/O stack. In this talk, we will introduce Tachyon, a distributed memory-centric storage system. In addition, we will talk about several production use cases where Tachyon further improves Spark applications’ performance by orders of magnitude.
In the presentation, we will explore several potential industry use cases enabled by the new features. One-click cluster deployment enables users to experiment and prototype with Tachyon on AWS, launching not only Tachyon but also the computation framework and storage system of their choice. Mounting of multiple under storage systems and transparent naming enables more exciting use cases for Tachyon users.