Testing distributed systems at scale is typically a costly yet necessary process. At Alluxio we take testing very seriously as organizations across the world rely on our technology, therefore, a problem we want to solve is how to test at scale without breaking the bank. In this blog we are going to show how the maintainers of the Alluxio open source project build and test our system at scale cost-effectively using public cloud infrastructure. We test with the most popular frameworks, such as Spark and Hive, and pervasive storage systems, such as HDFS and S3. Using Amazon AWS EC2, we are able to test 1000+ worker clusters, at a cost of about $16 per hour.
Tag: distributed systems
Strata Data Conference London 2017 – Learn about stream processing on Alluxio from real-world workloads at Qunar, as well as how to position Alluxio in the streaming architecture
Haoyuan Li offers an overview of Alluxio (formerly Tachyon), a memory-speed virtual distributed storage system.
Alluxio presents at Strata + Hadoop World Beijing 2016 with two talks: A keynote from founder Haoyuan Li and Alluxio’s latest use cases.
DataDriven NYC 2016 – In the past year, the Alluxio project experienced a tremendous improvement in performance and scalability and was extended with key new features including tiered storage, transparent naming, and unified namespace. At the same time, the Alluxio ecosystem has expanded to include support for more under storage systems and computation frameworks.
Strata+Hadoop World 2016 – Tachyon, a memory-centric fault-tolerant distributed storage system. An introduction of architecture, performance evaluation, and real world use cases.
ODSC West 2015 – Tachyon, a memory-centric fault-tolerant distributed storage system. An introduction of architecture, performance evaluation, and real world use cases.
Spark Summit 2014 – Haoyuan Li introduces Tachyon, a distributed in-memory storage system. Along with how Tachyon can further improve Spark’s performance and the integration between the two systems.