We are excited to announce the release of Alluxio v1.5.0, which comprises of over 3850 commits from 175 contributors. This release includes several major usability and performance improvements as well as enhanced integrations with projects in the ecosystem including for Azure Blob Store, Presto, and Docker.
For users upgrading from previous Alluxio versions, note that your Alluxio journal will need to be upgraded in order to migrate to the new version. See the documentation for step by step instructions.
Highlights for the release are listed below.
Downloads for this release can be found here: Downloads
Ecosystem Integrations
- Golang Client
- A native client has been developed to enable Golang applications to communicate with Alluxio. This allows applications in Golang to interact with Alluxio without needing to write custom code to invoke Alluxio’s REST interface or Java client. See the documentation in this repository for more details.
- Presto Integration
- Support for running Presto on top of Alluxio. Alluxio can provide up to 10x faster queries for Presto. See the documentation for more details.
- Docker Integration
- Support for deploying various Alluxio services in a Docker container. See the documentation for more details.
- Azure Blob Store Integration
- Support for running Alluxio with Azure Blob Store as an under storage. See the documentation for more details.
- Alternative Ceph Connector
- In 1.5.0, Alluxio can connect to Ceph under storages using the S3A connector which provides significant functionality and performance improvements. Users can expect up to 3x read performance improvement. See the documentation for more details.
- Improved Mesos Integration
- Improved support for deploying Alluxio in a Mesos and Marathon environment. This includes convenience functionality for deploying Alluxio servers as well as configuring Alluxio clients. See the documentation for more details.
Performance Improvements
- Periodic Journal Checkpointing
- Alluxio periodically compacts journal edit logs to update the journal checkpoint. This reduces restart and failover (if in HA mode) times, which were previously unbounded, to less than a minute. Periodic journal checkpointing is enabled in both HA and non HA modes.
- Packet Streaming Based I/O
- All network I/O now use the packet streaming protocol, providing up to 3x performance improvement. In addition, the metadata management between client and worker is greatly simplified, reducing the number of connections between the two components by about half.
- Domain Socket Based Short Circuit I/O
- Alluxio 1.5.0 introduces an option to use domain sockets for short circuit I/O to write data to local Alluxio storage through a worker process instead of the client. This mode is recommended for users running write heavy workloads in a container environment. Enable this by specifying the property
alluxio.worker.data.server.domain.socket.address. See the documentation for details on enabling and configuring this feature.
- Alluxio 1.5.0 introduces an option to use domain sockets for short circuit I/O to write data to local Alluxio storage through a worker process instead of the client. This mode is recommended for users running write heavy workloads in a container environment. Enable this by specifying the property
- Optimized Alluxio to Under Storage Metadata Operations
- The internal logic and concurrency mechanism of Alluxio is greatly improved for handling metadata heavy workloads with under storage systems, most notably object storages like Amazon S3, resulting in up to 10x performance improvement in metadata heavy stages of compute tasks.
Usability Improvements
- High and Low Watermark Space Reserver
- Asynchronous space reservation on Alluxio workers are now triggered at a specified high watermark and evict until the user specified low watermark is reached. It is highly recommended to use the space reserver in workloads with bursty writes. See the documentation for more details.
- Improved Object Store Directory Management
- The need to use dummy metadata files suffixed by
$_folder_$has been removed. This enables Alluxio to be more compatible with other systems interacting with the backing object store and allows users to mount buckets without having write permissions or the bucket set up with dummy files to work with Alluxio.
- The need to use dummy metadata files suffixed by
- Alluxio Client Module Isolation
- Alluxio 1.5.0 introduces a runtime module specifically designed for drop-in use of an Alluxio client. This client is equivalent to the core-client uber jar in previous versions. Alluxio’s other client interfaces such as HDFS and Alluxio’s native file system are in independent modules making it much easier to control dependencies for downstream projects. Best practices have been updated in the latest documentation.
- Mount Point Configuration Properties
- Richer configuration settings are exposed at a mount point granularity, enabling users to mount under storage systems with different properties. This enables use cases like using different credentials to connect to the same type of under storage. For example, mounting in Alluxio two Google Cloud Store buckets which require different credentials.
Acknowledgements
We would like to thank the following members of the community for their contributions to Alluxio 1.5.0. The release would not have been possible without your efforts!
Adit Madan, Albert Zhong, Amareesh Basanapalli, Andrew Audibert, Anjali Pemmaraju, Ankai Liang, Archie Yu, Arranius, Aseem Rastogi, Bin Fan, Bo Liang, Calvin Jia, Cao Yang, Cedric Xing, Chang Cheng, Chaomin Yu, Chaoya Jiang, Cheng Chang, Chongjie Li, Cody, Dabin Xu, Dan Johnson, Daniel Yu, Donghao Li, Dongyu Wang, Du Dingtian, Elsie Wang, Eric Zhang, Felix Xiao, Feng Xuan, Fengfeng Pan, Fengyun Cang, Fu Ziyu, Fuyi Zhang, Gene Pang, Guo Chen, Hang Cheng, Hanke Wang, Haoyuan Li, Hechen Gao, Henghui Li, Hong Wang, Hongji Cai, Huaming Yang, Huang Mengchen, Huang Zhi, Hui Chao, Jiadi Liang, Jianchao Wu, Jiang Song, Jieming Shi, Jin Xiao, Jingkai Shao, Jinhao Chen, Jinxuan Wu, Jiri Simsa, Jun Shi, Junfeng Ding, Junyao Zhang, Junyou Li, Kael Guo, Kaixuan Huang, Kaixuan Wu, Kang Fang, Kunlong Liu, Larry Mars, Liang Xu, Liangliang Yin, Liangyu Chen, Lifei Liu, Lili He, Liu Liu, Longxin Zheng, Lu Qiu, Maobao Long, Meng Sha, Mingwei Li, Minmei Wang, MoingLi, Neena Pemmaraju, Pei Sun, Pengfei Chen, Pengfei Li, Penny Wang, Peter Shen, Qi Yin, QianHao Dong, Qinghua Zhao, Qiu Hu, Rao Ma, Reid Chan, Rico Chiu, Rob Vesse, Roman Shtykh, Rongxun Guo, Ruiqi Zhang, Rulong Chen, Runze Cui, Ryan Walton, Sergey Petrenko, Shail Shah, Shaowu Tang, Shengguang Qiu, Shengxun Wu, Shiqi Lian, Shiqing Fan, Siyu Liu, Tao Han, Tao Zhong, TeaganZemel, Tiwari Nitish, Tong Zhou, Wang Lei, Wang Zhehan, Wanghong Ji, Wenhui Chen, Wenjia Yang, Wenjie Zhang, Wenming Zhang, Wenpu Cai, William Zapata, Wu Hang, Xianli Zhou, Xiao Yang, Xiaochang Wu, Xiaolei Jia, Xiaolei Wang, Xiaoqi Wu, Xiaoyao Jiang, Xinchen Guo, Xincheng He, Xingchen Yi, Xingxin Chen, Xinju Pu, Xinyue Chen, Xizao Wang, Xuewen Dong, Ya Zhou, Yahui Liu, Yanze Jiao, Ye Ji, Ye Qian, Yi Zhang, Yichen Zhou, Yifeng Chen, Yiming Liu, Yiran Wu, Yong Zhou, Yu Luo, Yu Shen, Yuchao Li, Yucheng Hu, Yue Wei, Yuening Su, Yufa Zhou, Yufeng Chen, Yuhan Zhang, Yuhao He, Yuhong Yang, Yun Zhao, Yunhong Yin, Yuntao Liu, Yupeng Fu, Yuting Wang, Zepeng Yu, Zhaokang Wang, Zhe Zhang, Zhehan Wang, Zhengzhao Chen, Zhenxing Li, Zhiqiang Gao, Zhiwen Zheng, cheng141220020, foreverunforget, tzattack