Today when we create a Hive table, it is a common technique to partition the table across different values and ranges to improve query performance and reduce maintenance cost. However, Hive can not access a single table directly using a single query with the data of this Hive table across different mediums of storage and … Continued
Problem Sometimes big data analytics need process input data from two different storage systems at the same time. For instance, a data scientists may need to join two tables one from a HDFS cluster and one from S3. Existing Solutions Certain computation frameworks may be able to connect to storage systems including HDFS and popular cloud … Continued
Alluxio can help data scientists and data engineers interact with different storage systems in a hybrid cloud environment. Using Alluxio as a data access layer for Big Data and Machine Learning applications, data processing pipelines can improve efficiency without explicit data ETL steps and the resulting data duplication across storage systems.
Alluxio provides a distributed data access layer for applications like Spark or Presto to access different underlying file system (or UFS) through a single API in a unified file system namespace. If users only interact with the files in the UFS through Alluxio, since Alluxio has knowledge of any changes the client makes to the UFS, it will keep Alluxio namespace in sync with the UFS namespace.
Calvin Jia and Jiri Simsa explain how the current Alluxio tiered storage can be easily configured to use memory, SSDs, and hard drives in different tiers. Alluxio users and administrators do not have to manually migrate the data because data in Alluxio is managed transparently between all the configured tiers, similar to the way the CPU manages L1, L2, and lower-level caches. Meanwhile, Alluxio also provides users fine-grained control of manipulating data to plug in their own data-management strategies; users can also pin files in Alluxio to a specific storage or specify a TTL to files. Calvin and Jiri also describe the interface for managing heterogeneous data sources into the Alluxio namespace, which takes advantage of Alluxio’s ability to interoperate with different underlying storage systems such as HDFS, S3, GlusterFS, or Swift.
The Alluxio project has greatly improved system performance, Scalability and user experience, and added a series of new features, including scalable tiered storage, transparent UFS data reading and writing, unified namespaces, and more. Easy to use with Alluxio. At the same time, the Alluxio ecosystem has expanded to support different storage systems and computing frameworks. Alluxio now supports a variety of storage systems, including Amazon S3, Google Cloud Storage, Gluster, Ceph, HDFS, NFS and OpenStack Swift, as well as big data processing frameworks such as Spark, MapReduce, Flink and more. These integrations allow Alluxio to manage and help with more and more complex data.
The Alluxio project has greatly improved system performance, Scalability and user experience, and added a series of new features, including scalable tiered storage, transparent UFS data reading and writing, unified namespaces, and more. Easy to use with Alluxio. At the same time, the Alluxio ecosystem has expanded to support different storage systems and computing frameworks. Alluxio now supports a variety of storage systems, including Amazon S3, Google Cloud Storage, Gluster, Ceph, HDFS, NFS and OpenStack Swift, as well as big data processing frameworks such as Spark, MapReduce, Flink and more. These integrations allow Alluxio to manage and help with more and more complex data.
Using intermediate APIs means developers can learn just one framework and still access features offered by different technologies. It means writing job logic only once and being able to test it easily on a new underlying service with no effort. Not only is modularity a win for users but it means creators of execution frameworks and storage systems can focus on performance and capability without having to worry about API maintenance.
The goal is to make Alluxio accessible to an even wider set of users through a focus on security, new language bindings, and further increased stability. In addition, the team is working on new APIs to allow applications to access data more efficiently and manage data across different under storage systems.