This blog is authored by Madan Kumar and Alex Ma originally posted on medium.
As the data ecosystem becomes massively complex and more and more disaggregated, data analysts and end users have trouble adapting and working with hybrid environments. The proliferation of compute applications along with storage mediums leads to a hybrid model that we are just not accustomed to.
With this disaggregated system data engineers now come across a multitude of problems that they must overcome in order to get meaningful insights.
- Enabling connections between the various computes and storage becomes increasingly complex.
- Often seeing low performance due to lack of data locality for compute, which is a new challenge that we did not have to face previously in collocated environments(storage & compute together).
- Ultimately having to deal with high costs, mainly due to creating multiple copies of data as and when they need it closer to compute which ultimately results in storage not being optimized and becoming increasingly saturated.
In this new 2.0 ecosystem data engineers need to find a way to be able to leverage and work with hybrid environments. While also being able to maintain minimal code changes for their applications and being able to leverage all storage systems available to their fullest.
Today I see data engineers when attempting to work in these hybrid environments have no easy and transparent way to deal with these issues. Many times we tend to make multiple copies across environments in the hopes of trying to achieve locality. While also not being able to use more efficient computes due to API incompatibility. We tend to inexplicably end up overloading storage and not fully leverage other cheaper solutions.
Handling these modern workloads requires a solution that solves a few different problems but most of all one that can serve as a virtualization layer between compute and storage. Similar to how we have orchestration frameworks for technologies like containers, there needs to be an orchestration framework for data. One such open source system is Alluxio (formerly Tachyon Nexus), Alluxio provides capabilities that allow it to function as a modern data orchestration solution.
Alluxio provides a few particular features that a data orchestration framework needs to be successful in Hybrid environments.
A framework that allows engineers to have unified access to data regardless of the storage system it may reside on. This becomes increasingly necessary when also using newer computes that may not natively integrate to a particular storage. Which allows you to not have to worry about using a common interface. Alluxio’s API translation allows users to continue bringing new technologies into their ecosystem while also ensuring a durable consistent way of ensuring that they can be connected. Alluxio’s tiering capability also helps solve the slow data access problem while letting you leverage lower cost storage.
While working in hybrid environments can be challenging, it is something that we must come to grips with in today’s rapidly involving data ecosystem. Modern data orchestration frameworks today while not solving the entire problem have come a long way in making the adaption to hybrid that much easier!
.png)
Blog
Coupang, a Fortune 200 technology company, manages a multi-cluster GPU architecture for their AI/ML model training. This architecture introduced significant challenges, including:
- Time-consuming data preparation and data copy/movement
- Difficulty utilizing GPU resources efficiently
- High and growing storage costs
- Excessive operational overhead maintaining storage for localized data silos
To resolve these challenges, Coupang’s AI platform team implemented a distributed caching system that automatically retrieves training data from their central data lake, improves data loading performance, unifies access paths for model developers, automates data lifecycle management, and extends easily across Kubernetes environments. The new distributed caching architecture has improved model training speed, reduced storage costs, increased GPU utilization across clusters, lowered operational overhead, enabled training workload portability, and delivered 40% better I/O performance compared to parallel file systems.
Suresh Kumar Veerapathiran and Anudeep Kumar, engineering leaders at Uptycs, recently shared their experience of evolving their data platform and analytics architecture to power analytics through a generative AI interface. In their post on Medium titled Cache Me If You Can: Building a Lightning-Fast Analytics Cache at Terabyte Scale, Veerapathiran and Kumar provide detailed insights into the challenges they faced (and how they solved them) scaling their analytics solution that collects and reports on terabytes of telemetry data per day as part of Uptycs Cloud-Native Application Protection Platform (CNAPP) solutions.
.png)