Managing Configurations for Prometheus and Grafana Helm Charts

Diagram

Managing Prometheus and Grafana via Helm (draw.io viewer)

Managing Prometheus and Grafana via Helm The Prometheus Operator manages the CRDs. The Values.yaml and CRDs impacts each object’s configuration

This blog post is in specific reference to the Kube Prometheus Stack as part of the Prometheus Community, prometheus-community/kube-prometheus-stack

When using the Helm Charts managed by the Prometheus Monitoring Community repo, certain considerations should be made when managing the Prometheus configuration options such as Alerting Rules and Scrape Configs. This post will attempt to break-down best practices using the Kubernetes Custom Resource Definitions (CRDs) created by the helm chart deployment.

Service Discovery Versus Static Configs

When using the Prometheus Monitoring Community’s Kube Prometheus Stack, Service Discovery for Kubernetes is configured. Discovery is dynamic upon the creation and destruction of objects within the cluster. Objects in Kubernetes are therefore ready for monitoring in Prometheus during their lifecycle which is accomplished using the Kubernetes API server. Service Discovery configs are used in lieu of Static Configs for dynamic metrics gathering. Refer to the Kubernetes Service Discovery configuration for more details.

When creating Scrape Configs, along with other Prometheus configuration options, using CRDs is the best approach when applying the Kube Prometheus Stack helm chart. The chart includes the Prometheus Controller Manager and its CRDs.

Using the Service Monitor CRD automatically configures the relabel_config based on spec.selector within the manifest. The relabel config enables service discovery of targets for the specified job being created.

Example:

apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    labels:
      app: ibkr-dash
      release: prometheus
    name: ibkr-dash
  spec:
    endpoints:
    - interval: 30s
      port: metrics
    namespaceSelector:
      matchNames:
      - default
    selector:
      matchLabels:
        app.kubernetes.io/managed-by: Helm

Will resolve to the following Configuration:

- job_name: serviceMonitor/default/ibkr-dash/0
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 30s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  http_headers: null
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_managed_by,
      __meta_kubernetes_service_labelpresent_app_kubernetes_io_managed_by]
    separator: ;
    regex: (Helm);true
    replacement: $1
    action: keep
...
kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    http_headers: null
    namespaces:
      own_namespace: false
      names:
      - default

The above YAML has been condensed as indicated by the ellipses (...)

Notice that the resolved scrape config includes the kubernetes_sd_configs for automated service discovery of endpoints created for IBKR-Dash app within the default namespace. You may review the resolved configuration with the Prometheus UI by selecting Status -> Configuration. The application will also be populated within the Status -> Service Discovery view.

More details about Service Monitor CRDs and why CRDs in general should be used when configuring Prometheus will be discussed further in this bog post.

Why CRDs?

Making changes the the CRDs objects or creating new CRDs as defined by the helm chart enables a more clean and consistent approach to managing the configuration options for Prometheus and Grafana. Changes or additions to the CRD will perform an automated config reload against the Prometheus or Grafana objects in Kubernetes. In addition, future updates the the helm charts will prevent your custom values from being overwritten.

This is in contrast to making changes by first pulling in the Values.yaml, editing then updating the helm charts using the -f flag. Using this method requires a manual reload by calling the service endpoints for the Prometheus deployment using CURL. These CRDs are defined and managed by the Prometheus Operator built into the aforementioned helm chart.

You may determine the reload endpoint by executing the following command against your Kubernetes cluster:

kubectl get sts <helm release>-prometheus-kube-prometheus-prometheus -n <namespace> -o yaml | grep reload-url

Available Prometheus CRDs

To check for available CRD object which may be editing or created, execute the following command:

kubectl get crds -n <namespace>

Each of the listed CRDs may then be called against the Kubernetes API like any other Kubernetes Object, such as a Deployments or StatefulSets. For example, for the alertmanagers.monitoring.coreos.com CRD, you may list its existing objects by executing

kubectl get alertmanagerconfigs -n <namespace>

New resources created as defined by each of the CRDs must contain the configured key/value CRD label selector when populating the metadata.labels object. This enables Prometheus Service Discovery to discover the endpoints as referenced by these new resources. For example, when creating a new ServiceMonitor object for scraping a custom metrics endpoint, the endpoints referenced by the new object are not discoverable by prometheus unless the metadata.labels also matches the underlying CRD config.

I will now delve into the key CRDs to create/update when managing the Helm Chart Configs for Prometheus and Grafana.

Alerting Rules

The alerting rules are managed by the prometheusrules.monitoring.coreos.com CRD which define alert conditions for Prometheus. Learn more at the official Prometheus Documentation - Alerting Rules

Create a yaml manifest as follows to add a new alert rule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    release: prometheus
  name: prometheus-example-rules
spec:
  groups:
  - name: ./example.rules
    rules:
    - alert: down
      expr: up==0
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: prometheus

The metadata.labels section must contain the key/value for what is configured for the underlying CRD. Check the existing CRD by running kubectl get prometheuses <helm release>-kube-prometheus-prometheus -n <namespace> -o yaml | grep -i matchlabels -A5

The output will show the matchLabels: selector.

Scrape Configs

Managed by the servicemonitor.monitoring.coreos.com CRD which specifies a set of targets and parameters describing how to scrape them. One scrape config specified a single job in prometheus. Learn more at the official Prometheus Documentation - Configuration/Scrape Config.

Note: When creating a scrape config, ensure a Kubernetes Service object exists exposing the service against the Pod/ReplicaSet/StatefulSet/daemonSet/Deployment.

Create a yaml manifest as follows to add a new scrape config:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: servicemonitor-app
  labels:
    app: servicemonitor-app
    release: prometheus
spec:
  endpoints:
  - port: http-metrics
    path: /url/paths
    interval: 30s
  namespaceSelector:
    matchNames:
      - service-monitor-namespace
  selector:
    matchLabels:
      instance: service

Note the spec.namespaceSelector and spec.selector work together to ensure the ServiceMonitor selects the Kubernetes objects containing the custom metrics URL path such as Deployments, Pods, or StatefulSets.

The spec.endpoints section must reference a port’s name inside of a Service which exposes its underlying objects such as NodePort or ClusterIP.

The metadata.labels section must contain the key/value for what is configured for the underlying CRD. Check the existing CRD by running kubectl get servicemonitors <helm release>-kube-prometheus-prometheus -n monitoring -o yaml | grep -i matchLabels -A

The output will show the matchLabels: selector.

Exporters

Scrape Configs are also required when adding or writing a Prometheus Exporter such as for Redis.These exporters act as a translation layer between the application and Prometheus, exposing an additional /metrics endpoint, sometimes using a sidecar container. These exporters also exist as Docker Images in Docker Hub.

It is recommended to add an Exporter, if not already built into the Helm chart maintained for the application in question, by first searching for an existing Exporter within Artifact Hub.

Redis Helm Chart for example contains a sidecar container for exposing Redis application metrics.

When installing an add-on Exporter via Helm, its Values.yaml should be updated to match the key/value label as specified in the metadata.labels above, in addition to the correct Service Name and Port as determined by the Service created to expose the application.

You may use the helm show values <repo/helm-chart> > values.yaml and helm install (or upgrade) <release> -f values.yaml method to accomplish this. See the chart’s documentation for more details.

The serviceMonitor section is what instructs the helm chart to spin-up a new ServiceMonitor CRD with the correct metadata.labels directive in lieu of the manual steps defined above. If a comparable serviceMonitor directive does not exist, manual steps must be taken to create each of the ServiceMonitor as defined above. In addition, if a Service is not created which exposes the Exporter pod, create one manually.

Example Exporter Match Label Config Snippet:

mongodb:
  uri: "mongodb://mongodb-service:27017" # points to the existing Service for MongoDB

serviceMonitor:
  additionalLabels:
    release: prometheus # must match what is configured for the ServiceMonitor CRD. 

Check the Service created by the Exporter’s helm chart via port forwarding or creating an additional NodePort, then attempt to browse to the exposed scrape URL path, usually /metrics. You should also see the new ServiceMonitor listed as a Target in the Prometheus UI with its detected endpoints.

Alert Manager Rules

Managed by the alertmanager.monitoring.coreos.com CRD which handles alerts sent by the client application. It takes care of grouping and routing alerts to the correct receiver. Learn more at the official Prometheus Documentation - AlertManger.

Create a yaml as follows to add a new alert manager rule:

apiVersion: monitoring.coreos.com/v1
kind: AlertManager
metadata:
  name: servicemonitor-app
  labels:
    app: servicemonitor-app
    release: prometheus
spec:
  route:
    groupBy: ["alertname"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: "webhook"
    routes:
    - matchers:
      - name: job
        value: kubernetes
      receiver: "infra"
      groupBy: ["severity"]

The metadata.labels section must contain the key/value for what is configured for the underlying CRD. Check the existing CRD by running kubectl get alertmanagers <helm release>-kube-prometheus-alertmanager -n <namespace> -o yaml | grep -i selector

If the alertmanagerConfigSelector value is empty, it must first be specified by the following steps:

Execute helm show values prometheus-community/kube-prometheus-stack > values.yaml
Edit the values.yaml by searching alertmanagerConfigSelector. Remove the value {} and replace with:
```
 alertmanagerConfigSelector:
  matchLabels:
    resource: prometheus
```
Execute helm upgrade <helm release> prometheus-community/kube-prometheus-stack -f values.yaml

Using Helm Values

You may also edit the aforementioned config values by editing the values.yaml. This is not recommended as further updates to the helm-chart may override these values.

Execute helm show values prometheus-community/kube-prometheus-stack > values.yaml
Edit the values.yaml for the following configuration options (examples to uncomment are provided):
- Scrape Config - additionalScrapeConfigs (example)
- Alerting Rules - additionalPrometheusRulesMap (example)
- Alert Manager Config - alertManagerConfiguration (example)
Execute helm upgrade <helm release> prometheus-community/kube-prometheus-stack -f values.yaml

Adam M. Lechnos