This is an open source community conference focused on the key data engineering challenges and solutions around building cloud-native data and AI platforms using latest technologies such as Alluxio, Apache Spark, Apache Airflow, Presto, Tensorflow, and Kubernetes.
Tags: analytics, data orchestration, data orchestration summit, presto, spark
Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows.
Tags: analytics, data orchestration, hybrid cloud, machine learning, overview, webinar
In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see.
Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows.
Tags: analytics, data lake, data management, data orchestration, hybrid cloud, machine learning, performance, webinar
In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see.
Join us for this webinar where Alex Ma of Alluxio, an open source data orchestration platform, will discuss how a data orchestration approach offers a solution for connecting traditional on-prem data centers with the cloud, data centers with other data centers, and clouds with other clouds. With Alluxio’s “zero-copy” burst solution, companies can bridge remote data centers with computing frameworks in other locations, enabling them to offload compute and leverage the flexibility, scalability, and power of the cloud for their remote data.
Adit Madan and Parviz Peiravi offer an overview of the Alluxio data orchestration layer that provides a unified data access layer for hybrid and multi cloud deployments, leveraging Intel® Optane™ Persistent Memory for higher performance caching at reduced cost. The data access layer enables distributed compute engines like Presto, TensorFlow, and PyTorch to transparently access data from various storage systems (including S3, HDFS, and Azure) while actively leveraging a multi-tier cache to accelerate data access.
In this office hour, we demonstrate how a “zero-copy burst” solution helps to speed up Spark and Presto queries in the public cloud while eliminating the process of manually copying and synchronizing data from the on-premise data lake to cloud storage. This approach allows compute frameworks to decouple from on-premise data sources and scale efficiently by leveraging Alluxio and public cloud resources such as AWS.
International Data Corporation (IDC) reported that the global datasphere will grow from 33 zettabytes in 2018 to 175 zettabytes by 20251. This trend becomes more and more complicated with the variety and velocity of data growth, and it continuously changes the ways data is collected, stored, processed, and analyzed. New analytics solutions, including machine learning, deep learning, and artificial intelligence (AI), and new architectures and tools are being developed to extract and deliver value from the huge datasphere.
Tags: analytics, big data, hybrid cloud, intel, open source, performance, persistent memory