We are in the early stage of the data revolution. Today’s conventional wisdom states that hybrid latency prevents you from running analytic and machine learning workloads in the cloud with the data on-prem. As a result, most companies either do not burst their workloads into the cloud or have to copy their data into a cloud environment and maintain that duplicate data—and even this can take days or weeks.
“Zero-copy” HYBRID Bursting With No ApP changes
FEATURED USE CASE
Is your compute capacity limited? Bursting to the cloud using data on-prem can bring the compute flexibility you need. Intelligently burst processing to cloud data services like EMR and Dataproc with Alluxio Data Orchestration.
what is zero-copy burst?
Hybrid Cloud is the concept that you want to take advantage of resources that are not local to your data. It may be a large data lakes or silos and you want to leverage some compute capacity in the cloud. Zero-Copy burst enables you to burst, or move, your remote data closer to compute in the cloud for these benefits:
Time to Production
Expand your cloud footprint with significantly lower lag
Reduce overload on existing infrastructure by moving ephemeral workloads to the cloud
One step closer to the cloud
Use zero-copy burst as the intermediate step before migrating fully to the cloud
spending too much time maintaining data copies?
Bursting your on-prem workloads to the cloud can mean slow performance and managing duplicate data/application changes.
Using S3 via HDFS leads to low performance due to network latency
Copying data via DistCP from on-prem to cloud means maintaining duplicate data
Using other storage systems like S3 means expensive application changes
intelligently burst processing to the cloud with alluxio
Alluxio’s data orchestration platform leaves your data on-prem and intelligently bursts processing to cloud data services like EMR and Dataproc.
Intelligently burst HDFS workloads to the cloud
Deploy Alluxio + compute on-prem and S3 in the cloud
Want help getting started on zero-copy hybrid bursting? Schedule a meeting with one of our solution engineers.
configuring alluxio + hdfs in the public cloud
You can zero-copy burst your workloads to AWS, GCP, and Azure with Alluxio. By bringing the data to the analytics and machine learning applications, the performance is the same as having the data co-located in the cloud. Plus, the on-prem data stores will have offloaded the computation and minimized the additional I/O overhead.
LATEST RELATED POSTS
As the data ecosystem becomes massively complex and more and more disaggregated, data analysts and end users have trouble adapting and working with hybrid environments. The proliferation of compute applications along with storage mediums leads to a hybrid model that we are just not accustomed to.
With this disaggregated system data engineers now come across a multitude of problems that they must overcome in order to get meaningful insights.