We are thrilled to announce the release of Alluxio 2.8.0! This is the first release on the Alluxio 2.8.X line.
Alluxio 2.8 further enhances the S3 API functionality, job service observability, and system scalability.
HIGHLIGHTS
Enhanced S3 API with Metadata Tagging
Alluxio 2.8 commences a slew of ongoing improvements to the S3-compatible REST API. Object and bucket tagging APIs have been added (f3d2477) which lay the groundwork for managing end-to-end file metadata through the Alluxio S3 API.
For example the “x-amz-tagging” header is now supported by the PutObject REST endpoint. You can specify user-defined tags to attach to the uploaded object through this request header as a query-parameterized string:
curl -H "x-amz-tagging: key1=value1&key2=value2&flag" -XPUT "http://<host>:<port>/api/v1/s3/<bucket>/<object>" --data="..."
Object tags can be retrieved via:
curl -XGET "http://<host>:<port>/api/v1/s3/<bucket>/<object>?tagging"
See the Alluxio S3 documentation for full details on supported tagging operations. Tags are currently limited to 10 user-defined tags per file in accordance with the S3 specifications.
This release aims to also further compatibility with S3 client applications through request header additions and improvements (e913945).
Stabilize and Scale Alluxio in Deep Training Workloads
Alluxio 2.8 improves the FUSE mount/unmount mechanism. FUSE can be mounted through the Alluxio CLI or via configuration properties (e5f53a9). FUSE unmount mechanism is improved to reduce the chance of leaving FUSE unmounted in the host machines (5393b6). FUSE3 (6f3fe6f) integration is newly supported, enabling future performance and scalability optimizations. Although FUSE2 is the default version, Alluxio will eventually phase out FUSE2 in favor of FUSE3.
Several critical issues were identified in production workloads and fixed to improve the stability of Alluxio when integrating with training workloads. Examples include a RocksDB core crash with high memory consumption (918e73), FUSE segment fault error (issue), and FUSE statfs potential OOM (fabcf47).
More optimizations were added to support a large number of small files and highly concurrent access, including supporting worker registration with millions of blocks (htf8e5e) and improving the performance by 10x and reducing memory overhead when preloading a large number of small files (bc104a9).
Exception Handling for Data Movement
In Alluxio 2.8, we improved the exception handling of data movement, specifically the distributedCp and distributedMv operations. The improvements include removing 0-byte files if the task failed or was canceled (2c83498,75210e0) and preserving the destination file for overwrites until the write operation succeeds, which previously would have been deleted at the start of the job (2fa2683).
Asynchronous Job Service Execution
Distributed commands for load and copy can now be submitted in asynchronous mode (cfbd06f, 01af5b1). The new mode can be turned on by specifying the –async flag in the distributedCp and distributedMv commands (82505a5).
This new feature simplifies the command submission and monitoring process for the end user. In previous versions, users needed to ensure their command console to remain active and not be interrupted while long-running commands were executing. After launching an asynchronous command, its progress can be reported by running the getCmdStatus CLI command. In the default sync mode, users will no longer see the progress information on files being loaded or copied as they complete. The output will now display the command JobControlId, successfully loaded or copied files, each separated on a new line, statistics of completed and failed files, and the failed file path information.
System Stability and Scalability
In 2.8, we improved the system scalability by fixing concurrency issues with RocksDB (918e73) and resolving leaks (2f2894).
Multiple improvements were made to reduce the general memory usage of Alluxio master. In particular, we improved the performance of recursive deletion and reduced the memory consumption, both by roughly 20%. Based on our performance tests, we have also updated recommendations on how recursive deletion can be done more efficiently. You can find the recommendations in our User-CLI doc.
System Observability
New metrics were added in various parts of the system to provide better observability into the system state, including but not limited to, metadata sync operations (f24c26), data I/O exceptions (250bb3), and distributed commands (696cb8).
Alluxio 2.8 also added the ability for standby masters to serve the Web UI and metrics (a10823).
StressBench Tool Usability
In 2.8 we improved the usability of the StressBench framework by making some parameters more user-friendly. We also added a quick start pipeline to batch run multiple stress tests.
A new workflow has been added under the name MaxFile. The goal of this benchmark is to evaluate the maximum number of files an Alluxio master can handle before becoming unresponsive. The benchmark can run for many hours depending on the metastore type and heap space allocated to the Alluxio master. You can find more information on this document.
IMPROVEMENTS AND FIXES SINCE 2.7.4
CLI
Add missing options in DistributedLoad command (4da1bcb73b)
Docker, CSI, K8s
Fix alluxio tarball owner in docker (46b20e997a)
Improve entrypoint.sh for fuse (f16c56f8da)
Make java version configurable in dev image (7fc1af3a99)
Unmount corrupted folder is case nodeserver restarted (d1d0628157)
Filesystem
Fix recursive delete not deleting children with valid permission (04761a9f19)
Fix root object store UFS directory not being listed (4ef4905056)
Journal
Close JournalServiceClient asynchronously to account for async calls (3506b4dfc7)
Make log compaction local in embedded journal (662c9c9e12)
Fix incorrect directory when mounting & formatting master journal volume (2b83d7872f)
Metrics
Fix client side leak caused by Gauge (1f54873a34)
Fix metric for ufs status cache can be negative (0c43ea1108)
Fix absent cache stats are always zero (f3f308c463)
Fix metric not being decremented in case of cancelled sync operation (8e021d6a32)
Support collecting worker metrics in collectMetrics command (21560be53f)
Web UI
Update Web UI query string utility functions (4101a3c362)
ACKNOWLEDGEMENTS
Thanks to the community for their invaluable contributions to the Alluxio 2.8 release. It would not have been possible without your help! We would especially like to thank:
Binyan2014, Junda Chen, Haoning Sun, JunLuo, Richard, XiChen, l-shen, qian0817, Weize Xu
We also want to especially thank Fei Peng from the Tencent team, who provided invaluable insights with release testing.
Enjoy the new release and let us know what you think on our community slack channel.