Alluxio 1.4.0 Release

We are excited to announce the release of Alluxio v1.4.0. This release includes several major usability and performance improvements as well as enhanced integrations with projects in the ecosystem including support for Apache Hive. Highlights for the release are listed below.

Downloads for this release can be found here: Downloads

Ecosystem Integrations

  • Optimized Integrations with Object Storages:
    • 10x speed up for file creation
    • 4x speed up for file deletion
    • Improved algorithm for discovering files in the under storage, resulting in 4x performance improvement
    • Streamlined programming interface, reducing the lines of code required for a new integration by half
  • More Flexible S3A Connector:
    • Enables connecting to other storage systems implementing S3-like interfaces, such as Minio
  • Support for Apache Hive:
    • Integration for running Apache Hive on top of Alluxio, see the documentation for more details
  • Improved integration with YARN based applications:
    • Support for user and permissions when running YARN applications which interact with Alluxio

Usability Improvements

  • Native REST File System API:
    • Interact with Alluxio with the native File System API through a RESTful interface
  • Tier Specific Writes:
    • Users may select a specific Alluxio storage tier to write data to instead of defaulting to the highest tier
  • Configurable Behavior for File TTL:
    • Allows for user defined behavior on TTL expiry, defaulting to deletion but can be configured to free the file from Alluxio instead
  • Alluxio – UFS Consistency Check:
    • A new shell command which provides a simple way for administrators to validate the consistency of their Alluxio metadata with the underlying storage system
    • An automatic check runs on Alluxio start up to validate the consistency of all persisted files with the mounted under storages

Performance

  • Improved Lock Management in Alluxio Master:
    • Allows higher concurrency for common metadata operation patterns in MapReduce based frameworks
    • Can provide order of magnitude performance improvements when using under storages with slow file system metadata operations (such as renames in cloud stores)
  • Packet Streaming Network Protocol:
    • A new network transfer protocol designed to fully utilize the available network bandwidth between Alluxio components
    • Up to 2x I/O performance improvement within a standard network, with better results in high latency-throughput product environments
    • Handles small reads and random reads optimally without configuration tuning
    • Currently an experimental feature, enabled by setting alluxio.user.packet.streaming.enabled to true in the client side configuration.

Acknowledgements

We would like to thank the following members of the community for their contributions to Alluxio 1.4.0. The release would not have been possible without your efforts!

Adit Madan, Alex Zhou, Andrew Audibert, Anil Nevase, Aseem Rastogi, Baole Ai, Bei Song, Bill Zhao, Bin Fan, Bingchuan Tian, Bo Li, Calvin Jia, Chang Cheng, Changwang Zhu, Chao Zhang, Chaomin Yu, Chenlin Zhang, Chenyu Zhang, Chongjie Li, Chunlong Zhang, Dan Johnson, Dong Xu, Dongxu Sun, Fan Shiqing, Feifei Xu, Felix Cheung, Fenggen Yu, Fengyu Qiu, Gang Chen, Gaojian Ge, Gary Gu, Gene Pang, Guo Jianhua, Haitao Wu, Hao Ren, Haonan Qiu, Haoyuan Li, Harsha Vardhana, Hongjun Ge, Huang Zhi, Jan Hentschel, Jia He, Jiacheng Huang, Jiakai Shen, Jiaxing Dong, Jie Chen, Jie Fang, Jie Yang, Jing Deng, Jingcheng Shi, Jinghao Feng, Jinquan Sun, Jipeng Huang, Jiri Simsa, Jun Li, Jun Shi, Kang Ye, Keyan Zhang, Kun Tao, Lei Fan, Lei Wang, Lei Xu, Liangliang Yin, Lingbing Guo, Longyi Chen, Luo Li, Malcolm Sun, Mao Baolong, Maojian Dong, Meng Li, Mengkui Yan, Min Chen, Minglin Ma, Pei Sun, Peng Wu, Peng Zhao, Pengfei Xu, Pengwei Chen, Qian Qu, Qilin Deng, Qinglong Wang, Qiu Hu, Qiufang Ma, Ramu Malur, Rico Chiu, Rui Zhang, Ryan Walton, Sergey Chernolyas, Shu Peng, Shunjie Ding, Siyu Tang, Songcheng Gao, Tang Bo, Tao Lu, Taotao Li, Tianhao Ning, Tianping Hu, Tiantian Wang, Tim Bisson, Tong Wei, Tongtong Xu, Wang Bingquan, Wang Pengyu, Wei Jing, Weiyang Qu, Weizhuo He, Wenhai Wang, Wenji Zhou, William Callaghan, Xi Jiang, Xiangbo Li, Xiaofei Zhang, Xiaoxue Liu, Xiaoyu Wang, Xingyu Chen, Xinyu Yan, Xu Chen, Xu Zhang, Xuan Huang, Yalin Zhang, Yan Meng, Yang Song, Yanjun Li, Ye Kang, Yi Gu, Yibo Jin, Yifeng Wu, Yingli Yan, Yiqun Wang, Yisheng Yue, Yize Li, Yu Gu, Yuan Wang, Yuanyu Wan, Yue Li, Yue Song, Yufa Zhou, Yuhang Zhou, Yujie Yu, Yulong Tian, Yun Li, Yunfeng Zhang, Yuntang Liu, Yupeng Fu, Zeyu Mi, Zhanshuai Meng, Zhaomin Chen, Zhen Wang, Zhen Wu, Zhenzhu Li, Zhicong Chen, Zhouyu Meng, Zixuan Wang, Ziyi Wang, Zuxian Huang