What do I do if Hadoop is slow?

So you have a Hadoop cluster that’s running fine and then you start to hear people saying that their jobs are running slow. This answer is meant to cover common reasons for slowness and look at some solutions to this problem.

Am I bottlenecking on S3, S3 throttling my requests?

When working with big data analytics and AI, you are likely reading and writing terabytes from S3, in some cases even at very high data transfer rates. There are several reasons why you may find S3 is slowing down your data-intensive job.

Hadoop overload: Reduce large performance variance in HDFS namenode

Some people experience serious performance issue in HDFS namenode (v2.7) response time. Particularly during peak traffic time, an HDFS namenode can become overloaded and some DFS operations (like listing a directory) can take a long time, which affects the query response time for Presto and other Hadoop applications. To solve for challenges in high latency … Continued

How do you offload workloads from Hadoop?

What is Apache Hadoop If you’re new to building big data applications, Apache Hadoop is a distributed framework for managing data processing and storage for big data applications running in clustered systems. It consists of 5 modules – a distributed file system (aka HDFS or Hadoop Distributed File System), MapReduce for parallel processing of datasets, … Continued

How do you orchestrate data between disparate storages?

As the data ecosystem within enterprises grow larger and larger, not only do we see an increase in total data volumes but also an increase in the disparate storage systems in which they are housed. The challenge then becomes how do different applications and teams have an efficient way of being able to access data … Continued

How do you run TensorFlow on a remote storage system?

Problem It becomes increasingly more popular among data scientists to train models based on frameworks like TensorFlow on a local server or cluster while using remote shared storages like S3 or Google Cloud Storage to store a massive amount of the input data. This stack provides high flexibility and cost efficiency, especially requires no dev-ops … Continued

How do you access data from both HDFS and cloud storage?

Problem Sometimes big data analytics need process input data from two different storage systems at the same time. For instance, a data scientists may need to join two tables one from a HDFS cluster and one from S3.  Existing Solutions Certain computation frameworks may be able to connect to storage systems including HDFS and popular cloud … Continued