Blog

Deep learning algorithms have traditionally been used in specific applications, most notably, computer vision, machine translation, text mining, and fraud detection. Deep learning truly shines when the model is big and trained on large-scale datasets. Meanwhile, distributed computing platforms like Spark are designed to handle big data and have been used extensively. Therefore, by having deep learning available on Spark, the application of deep learning is much broader, and now businesses can fully take advantage of deep learning capabilities using their existing Spark infrastructure.

Our mission at Alluxio is to unify data at memory speed. Today we’re excited to unveil our first products which enable organizations to turn data into value with unprecedented ease, flexibility, and speed. We believe our new products will substantially advance Alluxio for both the community and our enterprise customers. In this blog, I will share with you the big data challenges application developers and business line owners face today, and show how Alluxio addresses these challenges.

This is an excerpt from the Accelerating Data Analytics on Ceph Object Storage with Alluxio whitepaper. As the volume of data collected by enterprises has grown, there is a continual need to find efficient storage solutions. Owing to its simplicity, scalability and cost-efficiency object storage, including Ceph, has increasingly become a popular alternative to traditional file systems. In most cases the object storage system, on-premise or in the cloud, is decoupled from compute nodes where analytics is run. There are several benefits of this separation.

Alluxio is the world's first memory-speed virtual distributed storage system that bridges applications and underlying storage systems, providing unified data access orders of magnitudes faster than existing solutions. The Hadoop Distributed File System (HDFS) is a distributed file system for storing large volumes of data. HDFS popularized the paradigm of bringing computation to data and the co-located compute and storage architecture. In this blog, we highlight two key benefits Alluxio brings to a compute cluster co-located with HDFS.

Alluxio provides Spark with a reliable data sharing layer, enabling Spark to excel at performing application logic while Alluxio handles storage.