Getting CloudWatch Metrics in Grafana

Many AWS customers use Grafana, an open-source tool, in their monitoring system to gather metrics from multiple data sources and do real time analytic.

Grafana ships with built-in support for AWS CloudWatch integration, which means in a few steps you can start to getting CloudWatch metrics in Grafana. The metrics captured can then be visualized in dashboards for real-time monitoring.

In this post, I’ll walk you through every step how to integrate Grafana with CloudWatch from scratch.

AWS IAM Policies:

There are many ways to install Grafana. Here I deploy Grafana as a Pod into my EKS cluster.

First Grafana needs IAM permission to call CloudWatch API and be able to fetch metrics.

In the context of EKS, it can be done either through creating an IAM role linked Kubernets service account, or by adding additional CloudWatch IAM policies to an existing EKS node IAM role. I used the latter.

CwPolicy="CloudWatchReadOnlyAccess"
export RolePolicy=$(aws iam list-policies \
| jq '.[]' | jq -r --arg CwPolicy "$CwPolicy" '.[] | select(.PolicyName==$CwPolicy).Arn')
RoleName="YOUR_CLUSTER_NODE_INSTANCE_ROLE_NAME"
aws iam attach-role-policy --policy-arn $RolePolicy --role-name $RoleName
unset RolePolicy
unset RoleName

Grafana Installation on EKS Cluster

# Create YAML file of Grafana configuration to set CloudWatch as the datasource. Replace defaultRegion with your AWS region:
mkdir ${HOME}/environment/grafana
cat << EoF > ${HOME}/environment/grafana/grafana.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: CloudWatch
      type: cloudwatch
      jsonData:
        authType: default
        defaultRegion: eu-west-1
EoF
# Install Grafana Helm chart
helm repo add grafana https://grafana.github.io/helm-charts
kubectl create namespace grafana
helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='admin123' \
    --values ${HOME}/environment/grafana/grafana.yaml \
    --set service.type=LoadBalancer
# Run the following command to check if Grafana is deployed properly:
kubectl get all -n grafana
# You can get Grafana ELB URL using this command. Copy & Paste the value into browser to access Grafana web UI:
export ELB=$(kubectl get svc -n grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "http://$ELB"
# When logging in, use the username admin and get the password hash by running the following:
kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Grafana Dashboard

When the Grafana pod is in running state, go test if Grafana can connect to your CloudWatch regional endpoint.

Now it’s time to create your own dashboards to visualize metrics under different CloudWatch namespaces.

Monitoring artist created some pre-defined dashboards for CloudWatch metrics and shared on the GitHub account.

First we need to create a new folder in Grafana to grouping those dashboards. Below you can see I created a folder called “AWS_CloudWatch”.

Importing all those dashboards in one go (example bash script, jq required)

#!/bin/bash
jq --version >/dev/null 2>&1 || { echo >&2 "I require jq but it's not installed.  Aborting."; exit 1; }
### Please edit grafana_* variables to match your Grafana setup:
grafana_host=$Grafana_HOST
grafana_cred=$USER_PASS
# Keep grafana_folder empty for adding the dashboards in "General" folder
grafana_folder=$Grafana_Folder
ds=(1516 677 139 674 590 659 758 623 617 551 653 969 650 644 607 593 707 575 1519 581 584 2969 8050 11099 11154 11155 12979 13018 13040 13104 13892);
folderId=$(curl -s -k -u "$grafana_cred" $grafana_host/api/folders | jq -r --arg grafana_folder  "$grafana_folder" '.[] | select(.title==$grafana_folder).id')
if [ -z "$folderId" ] ; then echo "Didn't get folderId" ; else echo "Got folderId $folderId" ; fi
for d in "${ds[@]}"; do
  echo -n "Processing $d: "
  j=$(curl -s -k -u "$grafana_cred" $grafana_host/api/gnet/dashboards/$d | jq .json)
  payload="{\"dashboard\":$j,\"overwrite\":true"
  if [ ! -z "$folderId" ] ; then payload="${payload}, \"folderId\": $folderId }";  else payload="${payload} }" ; fi
  curl -s -k -u "$grafana_cred" -XPOST -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -d "$payload" \
    $grafana_host/api/dashboards/import; echo ""
done

Congrats! Now you are able to monitor AWS metrics in your Grafana dashboards.

Ref:

https://grafana.com/docs/grafana/latest/datasources/cloudwatch/

https://github.com/monitoringartist/grafana-aws-cloudwatch-dashboards

https://grafana.com/docs/grafana/latest/dashboards/dashboard_folders/