memory Archives

Community Office Hour: Improving Memory Utilization of Spark Jobs Using Alluxio

November 26, 2019

Many Spark users may not be aware of the differences in memory utilization between caching data directly in-memory into the Spark JVM versus storing data off-heap via an in-memory storage service like Alluxio. In this office hour, I will highlight the two approaches with a demo and open up for discussions

Tags: caching, memory, office hour, spark

Improving Spark Memory Resource with Off-Heap In-Memory Storage

November 1, 2019 By Bin Fan and Adit Madan

In the previous tutorial ”Getting Started with Spark Caching using Alluxio in 5 Minutes”, we demonstrated how to get started with Spark and Alluxio. To share more thoughts and experiments on how Alluxio enhances Spark workloads, this article focuses on how Alluxio helps to optimize the memory utilization of Spark applications. For users who are … Continued

Tag: memory