Configure observability in Platform Base

These tasks configure the observability stack described in Observability. Add the options to cluster/config.libsonnet.

Choose a profile

Set the profile before the first sync. It sizes Mimir, Loki, Alloy and kube-state-metrics together:

observability+: {
  profile: 'singleNode',  // or 'small' (default) or 'large'
},

Changing the profile later does not migrate data, see the warning. To change settings that no profile covers, use mimir.values, loki.values or alloy.values. They are merged into the Helm values of the chart.

Change the retention

Retention is shared by Mimir and Loki and defaults to 30 days:

observability+: {
  retention+: { period: '2160h' },
},

To give one of them its own retention, set mimir.retention or loki.retention to an object with enabled and period.

Use an S3 backend

By default Mimir and Loki store data in a bundled MinIO. To use an external S3-compatible endpoint for both, set the backend type and the endpoint:

observability+: {
  storage+: {
    backend+: {
      type: 's3',
      s3+: {
        endpoint: 's3.example.com',
        region: 'eu-central-1',
        accessKeyID: 'AKIAEXAMPLE',
        secretAccessKey+: {
          secretRef: { name: 's3-credentials', key: 'secretAccessKey' },
        },
      },
    },
  },
},

endpoint and accessKeyID are mandatory. Set s3.insecure to true for plain HTTP, or s3.insecureSkipVerify to true to skip certificate verification of a private CA. Mimir and Loki use their own bucket names.

The Secret must exist in both the mimir and the loki namespace, because the pods of each can only read Secrets of their own namespace. Both namespaces are created by the first sync, and the pods start when the Secret exists:

kubectl -n mimir create secret generic s3-credentials \
  --from-literal=secretAccessKey='<secret access key>'
kubectl -n loki create secret generic s3-credentials \
  --from-literal=secretAccessKey='<secret access key>'

To use different backends for Mimir and Loki, set mimir.storageBackend and loki.storageBackend instead. Set them as plain fields, not with +:, and give them the full shape of observability.storage.backend. They default to null.

Send email alerts

Alerts reach Mimir's Alertmanager. Configure the SMTP transport and a fallback recipient:

observability+: {
  alerting+: {
    email+: {
      enabled: true,
      smtp+: {
        smarthost: 'smtp.example.com:587',
        from: 'alerts@example.com',
        authUsername: 'alerts',
        authPassword+: {
          secretRef: { name: 'smtp-credentials', key: 'password' },
        },
      },
      defaultReceivers: ['oncall@example.com'],
    },
  },
},

Create the Secret in the grafana-alloy namespace, because Alloy reads the password. Without authUsername, no SMTP authentication is used. defaultReceivers receive every alert that no route matches.

This sets the transport only. Define the routes and receivers as AlertmanagerConfig resources in the cluster, and the alert rules as PrometheusRule resources. This only works in a cluster that runs the storage role.

Send data to an external target

The collector can send data to an endpoint outside the cluster, in addition to the local storage:

observability+: {
  collector+: {
    logs+: {
      externalTarget+: {
        enabled: true,
        url: 'https://loki.example.org/loki/api/v1/push',
      },
    },
    metrics+: {
      externalTarget+: {
        enabled: true,
        url: 'https://mimir.example.org/api/v1/push',
        auth+: {
          type: 'basicAuth',
          basicAuth+: {
            username: 'platform',
            password+: { secretRef: { name: 'remote-write', key: 'password' } },
          },
        },
      },
    },
  },
},

url is the full push URL. The data is sent with the tenant ID observability.clusterName.
The type of the logs target is loki and the type of the metrics target is prometheus (remote write). Both are the defaults.
auth.type is none (default), basicAuth or bearerToken. For bearerToken, set auth.bearerToken to a secretRef or a value.
Secrets must exist in the grafana-alloy namespace.
To stop sending data to the local Mimir or Loki, set disableInternalTarget to true on collector.logs or collector.metrics.

Run a collector-only cluster

To ship everything to another cluster, turn the storage role off and set at least one external target:

observability+: {
  storage+: { enabled: false },
  // plus the external targets above
},

Mimir, Loki and Grafana are then not deployed. On the receiving cluster, set mimir.ingress.enabled and loki.ingress.enabled to true. Their hostnames are mimir.<baseDomain> and loki.<baseDomain>, with the base domain from mimir.ingress.baseDomain and loki.ingress.baseDomain.

Collect more

Pod logs: set observability.collector.logs.allPods to true to collect the logs of all pods in all namespaces. This increases the ingest volume of Loki.
Control plane metrics: the API server is always scraped. Alloy scrapes the scheduler, the controller manager and kube-proxy on the node IP only if you set observability.collector.metrics.controlPlane.kubeScheduler, kubeControllerManager or kubeProxy to true. Most distributions bind these to localhost, so first bind them to the node address.
Self-monitoring: observability.selfMonitoring.enabled (default true) turns the scrapes, dashboards and recording rules for Mimir and Loki themselves on or off. Turn it off on small clusters to save storage.
Storage class: observability.storage.storageClass sets the class of every Mimir and Loki volume. null uses the default class.

Choose a profile​

Change the retention​

Use an S3 backend​

Send email alerts​

Send data to an external target​

Run a collector-only cluster​

Collect more​