Tencent Case Study: Delivering Customized News to Over 100 Million Users per Month with Alluxio

Tencent is one of the largest technology companies in the world and a leader in multiple sectors such as social networking, gaming, e-commerce, mobile and web portal. Tencent News, one of Tencent’s many offerings, strives to create a rich, timely news application to provide users with an efficient, high-quality reading experience. To provide the best experience to more than 100 million monthly active users of Tencent News, we leverage Alluxio with Apache Spark to create a scalable, robust, and performant architecture.

MOMO: Accelerating Ad Hoc Analysis with Spark SQL and Alluxio

Alluxio clusters act as a data access accelerator for remote data in connected storage systems. Temporarily storing data in memory, or other media near compute, accelerates access and provides local performance from remote storage. This capability is even more critical with the movement of compute applications to the cloud and data being located in object stores separate from compute. Caching is transparent to users, using read/write buffering to maintain continuity with persistent storage. Intelligent cache management utilizes configurable policies for efficient data placement and supports tiered storage for both memory and disk (SSD/HDD).

Whitepaper: MOMO – Accelerating Ad Hoc Analysis with Spark SQL and Alluxio

From our friends at MOMO The hadoop ecosystem makes many distributed system/algorithms easier to use and generally lowers the cost of operations. However, enterprises and vendors are never satisfied with that, so higher performance becomes the next issue. We considered several options to address our performance needs and focused our efforts on Alluxio, which improves performance … Continued

Tags: , , , , ,

Hedge Fund Improves Machine Learning Model Performance 4X with Alluxio

Quantitative hedge funds process large data sets with sophisticated financial models to drive investment decisions. Machine Learning is used to continuously improve models and maximize financial return. One firm with billions ($US) of assets under management turned to Alluxio to address the performance and cost challenges of large scale data processing in a hybrid cloud environment. With Alluxio, the number of model runs per day increased by 4x and the cost of compute was reduced by 95%.

Tags: , , ,

Lenovo Analyzes Petabytes of Smartphone Data from Multiple Locations and Eliminates ETL with Alluxio

Lenovo is the world’s largest personal computer vendor and one of the world’s largest smartphone vendors. The company has invested extensively in global information technology infrastructure, including ten data centers worldwide collecting petabytes of smartphone data. Analyzing data located in multiple data centers world-wide is critical for Lenovo to understand and improve the usability and reliability of their products.

Tags: ,

Apache Spark DataFrame caching with Alluxio

Many organizations deploy Alluxio together with Spark for performance gains and data manageability benefits. Qunar recently deployed Alluxio in production, and their Spark streaming jobs sped up by 15x on average and up to 300x during peak times. They noticed that some Spark jobs would slow down or would not finish, but with Alluxio, those jobs could finish quickly. In this blog post, we investigate how Alluxio helps Spark be more effective. Alluxio increases performance of Spark jobs, helps Spark jobs perform more predictably, and enables multiple Spark jobs to share the same data from memory.

Tags: , , , ,