Collecting and displaying metrics with Prometheus and Grafana

By Jon Colby

Getting monitoring up and running with Prometheus and Grafana deployed in a kubernetes cluster is surprisingly easy.

example metric

Tech Stack

Pre-requisites

These must already be installed and configured:

To be Configured

Install

Create a new git repository for monitoring

This assumes the repository has been created as an empty repo:

cd monitoring/
git init
touch README.md # Don't forget to document your stuff so you know what you've done!
mkdir prometheus
mkdir grafana

Install Prometheus

  1. Create a new file in the monitoring repo called values.yml.

  2. Add the unedited contents of the Prometheus default values.

  3. Modify values.yml to match your needs.

    These are some of the changes I’ve made to the default values. Most places insist on exposing the WebUI with kubectl port forwarding. This seems silly for just testing that the install came up. I chose to expose it through an ingress with tls certs. After verifying things look good, I disabled the ingress.

    Ingress is disabled because actual interactions with Prometheus will be done via Grafana for display and config alterations for metrics scraping.

    Modified ingress section of values.yml, with commented out sections removed for brevity:

    ingress:
    ## If true, Prometheus server Ingress will be created
    ##
    enabled: false # IMPORTANT: set to true to enable access via prometheus.domain_name.here # <---- CHANGED 

    annotations: # These annotations are assuming K3S and cert-manager setup specifics <---- CHANGED
      traefik.ingress.kubernetes.io/router.entrypoints: websecure # <---- CHANGED
      traefik.ingress.kubernetes.io/router.tls: "true" # <---- CHANGED
      cert-manager.io/issuer: letsencrypt-prod # letsencrypt-prod is the issuer I have configured, may be different in other environments  # <---- CHANGED

    ## Prometheus server Ingress additional labels
    ##
    extraLabels: {}

    ## Prometheus server Ingress hostnames with optional path
    ## Must be provided if Ingress is enabled
    ##
    hosts:
      - prometheus.domain_name.here # IMPORTANT: hostname setup # <---- CHANGED 

    path: /

    pathType: Prefix

    extraPaths: []

    ## Prometheus server Ingress TLS configuration
    ## 
    ## The default values file comment is a lie if you have cert-manager. Certs will be created automatically.
    ##
    tls:
    - secretName: prometheus-server-tls
      hosts:
      - prometheus.domain_name.here # IMPORTANT: set the hostname for cert-manager to create certs # <---- CHANGED

    Setting enabled: true in the above modifications will create an ingress and certificates automatically, allowing access to Prometheus at https://prometheus.domain_name.here. Assuming DNS is configured of course.

  4. Get the helm chart.

    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

  5. Install - from the prometheus directory we’ve been working in.

    helm install prometheus prometheus-community/prometheus -f values.yml

  6. Note down the value of scrape_interval in values.yml

server:
  < Lots of stuff left off >
  global:
    ## How frequently to scrape targets by default
    ##
    scrape_interval: 1m # <--- this is important for grafana

That’s all there is to it. Prometheus is installed and running. If the ingress above is enabled and DNS is configured, you should be able to poke around on https://prometheus.domain_name.here

Install Grafana

  1. Make a grafana directory in your monitoring repo.

  2. Add the unedited contents of the Grafana default values.

  3. Modify values.yml to match your needs.

    These are some of the changes I’ve made for my environment:

    Ingress section

    ingress:
  enabled: true
  annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: websecure # <-- changed
      traefik.ingress.kubernetes.io/router.tls: "true" # <-- changed
      cert-manager.io/issuer: letsencrypt-prod # <-- changed, allows cert-manager to get my certs
  labels: {}
  path: /

  # pathType is only for k8s >= 1.1=
  pathType: Prefix

  hosts:
    - grafana.domain_name.here # <-- changed
  ## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
  extraPaths: []

  tls: 
    - secretName: chart-example-tls
      hosts:
        - grafana.domain_name.here # <-- changed

    Persistence section, necessary to enable storage access to keep the data and configs.

    ## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  type: pvc
  enabled: true # <-- changed. Enables storage
  # storageClassName: default
  accessModes:
    - ReadWriteOnce
  size: 10Gi
  # annotations: {}
  finalizers:
    - kubernetes.io/pvc-protection
  extraPvcLabels: {}

  inMemory:
    enabled: false

  4. Get the helm chart.

    helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

  5. Install - from the grafana directory we’ve been working in.

    helm install grafana grafana/grafana -f values.yml

    Output should include instructions to get your initial admin password. Don’t forget to change this in the WebUI!

  6. Assuming DNS is configured, Grafana should now be reachable at https://grafana.domain_name.here

  7. Get your prometheus internal to the cluster IP - you’ll need this to setup Grafana.

    # kubectl get service | grep prometheus-server
    prometheus-server                     ClusterIP   10.43.198.203   <none>        80/TCP                                3h21m

    In this case, I needed to note down 10.43.198.203 as the Prometheus IP address. When you add your Prometheus data source, use this IP.

Complete documentation for adding data sources to Grafana.

NOTE: Scrape interval in Grafana must match the configured Prometheus scrape interval

scrape-interval