Tag: compute storage separation

Hybrid Environments for Data Analytics is a Possibility

By and

As the data ecosystem becomes massively complex and more and more disaggregated, data analysts and end users have trouble adapting and working with hybrid environments. The proliferation of compute applications along with storage mediums leads to a hybrid model that we are just not accustomed to.
With this disaggregated system data engineers now come across a multitude of problems that they must overcome in order to get meaningful insights.

Summertime themed In-Memory Computing extravaganza! (cross-post)

New York Meetup *

[Talk 1] A “how-to” presentation for building a real-time alerting, analytics and reporting system (at scale). With Denis Magda, vice president of the Apache Ignite PMC and director of product management at GridGain Systems. And Viktor Gamov, developer advocate at Confluent.
[Talk 2] Using In-Memory technology for real time analytics. With Andy Rivenes is a Product Manager at Oracle for Database In-Memory.
[Talk 3] Feeding data to the Kubernetes beast: bringing data locality to your containerized big data workloads. With Bin Fan, founding engineer of Alluxio, Inc. and PMC member of Alluxio open source project.

Running Spark & Alluxio in Kubernetes

Alluxio Community Office Hour *

The latest advances in container orchestration by Kubernetes bring cost savings and flexibility to compute workloads in public or hybrid cloud environments. On the other hand, it introduces new challenges such as how to move data to compute efficiently, how to unify data across multiple or remote clouds, how to co-locate data with compute and many more. Alluxio approaches these problems in a new way. It helps elastic compute workloads realize the true benefits of the cloud, while bringing data locality and data accessibility to workloads orchestrated by Kubernetes

How do you access data from both HDFS and cloud storage?

Problem Sometimes big data analytics need process input data from two different storage systems at the same time. For instance, a data scientists may need to join two tables one from a HDFS cluster and one from S3.  Existing Solutions Certain computation frameworks may be able to connect to storage systems including HDFS and popular cloud … Continued

Evolution of big data stacks under computational and storage separation architecture

Shanghai *

A new generation of open source big data, represented by Alluxio, born at the University of California at Berkeley, looks at this issue. Different from systems such as designing storage tight coupling to achieve low-cost reliable storage HDFS, by providing a virtual data storage layer defined and implemented by software for data applications, abstracting and integrating cloudy, hybrid cloud, multi-data center and other environments The underlying files and objects, and through intelligent workload analysis and data management, make data close to computing and provide data locality, big data and machine learning applications can be achieved with the same performance and lower cost.

Running Presto with Alluxio on Amazon EMR

Alluxio Community Office Hour - May *

Many organizations are leveraging EMR to run big data analytics on public cloud. However, reading and writing data to S3 directly can result in slow and inconsistent performance. Alluxio is a data orchestration layer for the cloud, and in this use case it caches data for S3, ensuring high and predictable performance as well as reduced network traffic.