alluxio answers

Learn intermediate and advanced compute, storage, and cloud concepts.

Compute

How do you run Spark on On-Premise S3?

Introducing S3 and Spark S3 has become the de-facto standard API for digital business applications to store unstructured data chunks. To this end, several vendors have S3-API compatible offerings that allow app developers to standardize on the S3 API’s on-premise, and port these apps to run on other platforms when ready. So, what is S3 and … Continued

Tags: , , ,

Storage

Cloud

Why am I seeing AWS S3 500 / 501 errors?

Increasingly S3 is being used as a data store for analytical and machine learning workloads. This means that it is very easy to generate a massive amount of get operations and request data from S3. For example: a couple of commands can launch a 1000 node cluster of AWS EMR service with the Spark or … Continued

Tags: ,

Data Architecture

How does replication in Alluxio happen across worker nodes? Is the unit of replication a file or a block?

Alluxio is a data orchestration system which provides data locality with intelligent multi-tiering. The replication parameters are easily configured and once done, Alluxio handles replication transparently to the requesting compute framework. As always, there’s no changes required by the end user, it’s transparent: In the above diagram, data is stored in RAM, SSD, or HDD. … Continued

Tags: ,