In the previous article, I have introduced about DataDog and how to deploy it to the Kubernetes cluster. Today I will share about the Autodiscovery feature in DataDog, and how we can send custom metrics to DogStastD socket. Requirements Basic knowledge about K8s, Docker, and DataDog. Administrator permission on a running…

When you are managing more than 10 Kubernetes (EKS) clusters for your team with some critical applications running on them, you will need to set up an observability stack with monitoring, logging, and tracing. In this blog, I will share how we build a monitoring system, set up a dashboard…

In previous article, I have introduced how we submit a Spark job to an EKS cluster. As long as we’re using other AWS components for our pipelines to interact, like S3/DynamoDB/etc., we need assign IAM policy to the Spark driver and executor pods. In this tutorial, I will show you how-to…

It took me 2 weeks to successfully submit a Spark job on Amazon EKS cluster, because lack of documentations, or most of them are about running on Kubernetes with kops or GKE, not Amazon EKS. …