Cloud has changed the dynamics of data engineering in many ways, from changing expectations of on-demand platform services to the popularity of the object store to the emergence of a flexible, separated data stack. And as a data engineer venturing into this cloudy world, the understanding of specific architectural approaches coupled with knowledge in some data stacks has proven useful.
Instead of being purely focused on data infrastructure, today's data engineer is now a full stack engineer. Compute, containers, storage, data movement, performance, network - skills are increasingly needed across the broader stack.
This white paper attempts to discuss some design principles as well as high priority elements of the stack that a data engineer should think about.