Hybrid and Multi-Cloud, AI and Deep Learning, Services, Data Sharing and New Table Formats
As we near the end of 2021, it’s a good time to take a deep breath, think about what we’ve learned as well as the trends we’re seeing, and share our thoughts around the top data predictions for 2022.
As more organizations advance their data revolution strategy, and run more diverse workloads on a wider variety of platforms across clouds and hybrid clouds, 2022 will see even more advances in AI, machine learning and analytic workloads and technologies and services to support them. There are five major trends I predict for 2022:
Hybrid Cloud a Reality & Multi-Cloud Strategy a No-Brainer
We’ve already seen a hybrid-cloud strategy with multiple data centers and public cloud providers emerge as the standard for large enterprises as the operational toolset continues to evolve and simplify cloud migrations. In 2022, we will see organizations grow their digital footprint by embracing the hybrid and multi-cloud model to enjoy elasticity and agility in the cloud, while maintaining tight control of the data they own. Cloud vendors will keep innovating and competing with differentiated capabilities in network connectivity and physical infrastructure improvements because organizations wouldn’t want being locked-in.
Mainstream AI and Deep Learning
As the toolset for AI applications continues to evolve, machine learning and deep learning platforms have entered the mainstream and will attain the same level of maturity as specialized data analytics. Just like we currently see a plethora of fully integrated managed services based on Apache Spark and Presto, in 2022 we will see vertical integrations emerging based on the likes of PyTorch and TensorFlow. MLOps for pipeline automation and management will become essential, further lowering the barriers and accelerating the adoption of AI and ML.
Services for Everything
Operational complexity was the demise of Hadoop on-premises. Cloud services offer the ease of elasticity of infrastructure provisioning with little operational hassle. In 2022, we will see the emergence of managed services not just for cloud environments but also hybrid-cloud and on-premises deployments to eliminate complexity from integrations of myriad components such as data catalog, data governance, computational frameworks, visualization and notebooks.
Data Sharing Across the Cloud
With SaaS and managed services in the cloud creating more data silos, improved governance and catalog with a data fabric spanning multiple services will come to the rescue in 2022. Sharing data across tenants and multiple service providers efficiently and securely will make data exchange easier than ever before.
Rise of Table Formats for Data Lakes
New stacks in both the storage and the compute layers keep evolving and innovating. Data Lakes are rising to prominence and structured data is transitioning to new formats. In 2022, open source projects like Apache Iceberg and Apache Hudi will replace more traditional Hive warehouses in cloud-native environments, enabling Presto and Spark workloads to run more efficiently on a large scale.
.png)
Blog
Coupang, a Fortune 200 technology company, manages a multi-cluster GPU architecture for their AI/ML model training. This architecture introduced significant challenges, including:
- Time-consuming data preparation and data copy/movement
- Difficulty utilizing GPU resources efficiently
- High and growing storage costs
- Excessive operational overhead maintaining storage for localized data silos
To resolve these challenges, Coupang’s AI platform team implemented a distributed caching system that automatically retrieves training data from their central data lake, improves data loading performance, unifies access paths for model developers, automates data lifecycle management, and extends easily across Kubernetes environments. The new distributed caching architecture has improved model training speed, reduced storage costs, increased GPU utilization across clusters, lowered operational overhead, enabled training workload portability, and delivered 40% better I/O performance compared to parallel file systems.
Suresh Kumar Veerapathiran and Anudeep Kumar, engineering leaders at Uptycs, recently shared their experience of evolving their data platform and analytics architecture to power analytics through a generative AI interface. In their post on Medium titled Cache Me If You Can: Building a Lightning-Fast Analytics Cache at Terabyte Scale, Veerapathiran and Kumar provide detailed insights into the challenges they faced (and how they solved them) scaling their analytics solution that collects and reports on terabytes of telemetry data per day as part of Uptycs Cloud-Native Application Protection Platform (CNAPP) solutions.
.png)