Alluxio 2.6.0 Release

We are excited to announce the release of Alluxio 2.6.0! This is the first release on the Alluxio 2.6.X line.

Due to the improvements in the 2.5 release, a much larger number of users were able to leverage Alluxio’s ability to accelerate AI/ML workloads as a Data Orchestration framework. We have taken the feedback and contributions from the community and further improved the end-to-end AI/ML workflows with Alluxio. This has taken the form of simplified deployment, data management capabilities, performance optimizations, and enhanced visibility into system states.

The improvements in Alluxio 2.6 further enable the recommended architecture of running AI/ML workloads with Alluxio on any storage, and also improves the general maintainability and visibility of Alluxio clusters, especially for large scale deployments.

Highlights

Unified Alluxio Worker and FUSE Process

Alluxio 2.6 supports running FUSE as a part of a worker process (documentation). Coupling the two helps reduce interprocess communication, which is especially evident in AI/ML workloads where file sizes are small and RPC overheads make up a significant portion of the I/O time. In addition, containing both components in a single process greatly improves the deployability of the software in containerized environments, such as Kubernetes.

Improved Distributed Load

Alluxio 2.6 improves the efficiency of distributed load and its user experience (documentation), in addition to more traceability and metrics for easier operability. Distributed load command is a key portion commonly used for many users to pre-load data and speed up subsequent training or analytics jobs.

Instrumentation Framework

Alluxio 2.6 adds a large set of metrics and traceability features enabling users to drill into the system’s operating state (documentation). These range from aggregated throughput of the system to summarized metadata latency when serving client requests. The new metrics can be used to measure the current serving capacity of the system and identify potential resource bottlenecks. Request level tracing and timing information can also be obtained for deep performance analysis.

Other Improvements

Bug Fixes

  • Fix regression in AbsentUfsPathCache (55a3b28)
  • Open the source file before creating target file during migrate (c74924d)
  • Add a boundary check before fuse read (3aaf75b)
  • Add resource loading fallback logic to ExtensionClassLoader (342f17f)
  • Fix potential directBuffer leak (6231603)
  • Fix listStatus on open file stream will complete the stream unexpectedly (eeb4193)
  • Fix the rejection of legitimate async caching requests because of the fulfill of queue (1e7c2f7)
  • Cancel sync job in middle of ufs calls if client cancelled (a906c87)
  • Fix Fuse write then read problem in async release and support umount (b1b3f40)
  • Prevent NPE for AbstractFileSystemShellTest (4c4ec92)
  • Fix helm-chart fuse hostpath type from File to CharDevice (34eb7cb)
  • Fix master crash due to journal flush error (70d6d0b)
  • Fix the worker hang issue (c668ae8)
  • Add null check for bucket column in Glue UDB (3618aa2)
  • Fix alluxio-fuse stat for Alluxio-Worker-FUSE (386a9f4)
  • Improve Ceph InputStream to be thread safe (265c24f)
  • Handle client cache failure on “no space left on device” (89f8971)
  • Fix replicate job loading non-persisted files (ce249e3)
  • Fix a regression of uncleaned directories (6ae85e6)
  • Fix process local block in stream (d605580)
  • Fix packaging distributed load code into DistributedLoadUtils (40af661)
  • Fix instrumentation setup (837a41c)
  • Fix xstream security issue (78f1ec2)
  • Fix throw CCE when using Fuse while enable client cache bug (f34b615)
  • Fix possible NullPointerException when calling e.getCause (81fe34e)
  • Fix NPE in Allocator and add some UT for Reviewer (f8518d0)
  • Fix variable reference and exit status (8de3ea2)

Acknowledgements

We have a list of community contributors who help Alluxio 2.6.0. This release would not have been possible without your support! Especially, we would like to thank:

Arthur Jenoudet, Baolong Mao, Bing Zheng, Binyang Li, Ce Zhang, Chenliang Lu, cutiechi, Daniel Pham, Eugene Ma, Haifeng Wang, l-shen, ja725, Jieliang Li, Junfan Zhang, leewish,  Micah Zhao, Pan Liu, Peter Roelants, Pramesh Gupta, Thinking Chen, wuzhenwei-xxx, Xiang Li, Yantao Xue, Yue Shao, Yun Wang, Zac Blanco, Zhenwei Wu, Zhenyu Song.

Enjoy the new release and look forward to hearing your feedback at the community slack channel.