Upgrade notes for Platform Base

How to upgrade

1. Change ref in your vendir.yml to the release you want to upgrade to.
2. Run vendir sync.
3. Read the sections below that lie between your current version and the target. They list the changes that need action on a running cluster.
4. Commit vendor/, vendir.lock.yml and any change to your configuration, then push. Argo CD syncs the new version.

Unreleased

Invalid configuration fails at render time (breaks configs that rendered before)

These configurations used to render and then broke the sync or a running component. Rendering now fails with a message that names the key:

• argocd.platformRepoCreds.enabled: true needs username and password, or sshPrivateKey.
• observability.alerting.email.enabled: true needs smtp.smarthost and smtp.from. Mimir rejected the whole Alertmanager config otherwise, which stopped all alert routing.
• argocd.configMaps.<name>.data has to be an object of strings, not a string or null.
• observability.collector.{logs,metrics}.enabled: true needs a target for that signal: observability.storage.{logs,metrics}.enabled (without disableInternalTarget) or externalTarget.enabled. Alloy accepted forward_to = [] and silently discarded the data. Otherwise set observability.collector.<signal>.enabled to false.
• tls.issuer.acme.solver: 'cloudDNS' needs cloudDNS.project, and 'webhook' needs webhook.solverName. type: 'acme' without a valid solver lists the valid solvers.
• Every identity provider in iam.realms.*.identityProviders needs oidc.clientSecret.value, or both oidc.clientSecret.secretRef.name and .key.

Collector signal toggles

observability.collector.metrics.enabled: false and observability.collector.logs.enabled: false now turn their pipeline off. They used to do almost nothing: Alloy kept scraping, discovering monitors, tailing pod logs and writing to Mimir and Loki. With metrics.enabled: false Alloy renders no scrape, monitor discovery or remote_write; with logs.enabled: false no pod log, event or loki.write component. The external target's credentials are no longer added to Alloy's environment for a disabled signal. The PrometheusRule and AlertmanagerConfig sync into Mimir follows observability.storage.metrics.enabled and keeps running. If you had set one of them to false and still rely on that data, set it back to true.

Loki keeps its volumes

The PVCs of Loki's backend (SimpleScalable) and single binary (singleNode profile) StatefulSets are no longer deleted with their StatefulSet. A scale-down or a deleted StatefulSet now leaves the PVCs behind; delete the orphaned ones by hand (kubectl -n loki get pvc) to free the storage. Mimir already worked this way.

IAM host and namespace

iam.ingress.host is honored now. It was ignored, so a custom value moved the issuer URLs of Argo CD and Grafana but not the Keycloak ingress. The iam app also takes its namespace from the keycloak-operator-edp app (both iam by default), since the operator reads the Keycloak admin Secret from its own namespace.

Argo CD SSO client secret (breaks argocd.sso.keycloakClient.enabled: false)

• Without the Keycloak client, argocd-cm now refers to argocd.sso.clientSecret.secretRef ($<name>:<key>). It used to refer to a generated copy of the Keycloak client's Secret, which does not exist then. Set secretRef, or value; rendering fails until you do. The Secret has to be in the argocd namespace and labelled app.kubernetes.io/part-of: argocd.
• SecretTemplate/argocd-client-secret is only rendered with the Keycloak client, and reads the key argocd.sso.keycloakClient.clientSecretSecretRef.key of its Secret (the generated Secret keeps the key clientSecret). With argocd.sso.enabled: false or an inline value it is no longer rendered.

Grafana SSO client secret (breaks grafana.sso.keycloakClient.enabled: false)

Grafana now reads the client secret from grafana.sso.clientSecret (secretRef, or the inline value, which wins) instead of always from the Keycloak client's Secret. Without the Keycloak client, set grafana.sso.clientSecret.secretRef.{name,key}; rendering fails until you do. The default with the Keycloak client is unchanged. With grafana.sso.enabled: false the Grafana pod no longer waits for the Keycloak client's Secret, which was never created in that case.

Inline secret values (nothing breaks)

Every inline value of a credential field is stored in plain text in the Application of its app and in Application/bootstrap, which every Argo CD user can read. Rendering now warns about each one, and the warnings no longer suggest that TLAs or dynamicConfig are a safe place for a secret. Move the credential into a Secret and set secretRef instead. Affected fields: argocd.platformRepoCreds, argocd.sso.clientSecret.value, the tls.issuer.acme solver secrets, iam identity provider oidc.clientSecret.value, observability.storage.backend.s3.secretAccessKey, observability.alerting.email.smtp.authPassword and the auth of observability.collector.{logs,metrics}.externalTarget. The README section "Secrets" lists where each value ends up and in which namespace its Secret has to exist. The S3, SMTP and Alloy values are also literal env vars in the Pod specs.

Pod Security labels

The namespaces iam, traefik (also used by tls), cert-manager, secretgen-controller and keycloak-operator get the labels pod-security.kubernetes.io/enforce: baseline and pod-security.kubernetes.io/warn and audit: restricted. Argo CD applies them to the existing namespaces at the next sync. The workloads this repository deploys there pass baseline. Pods you add to these namespaces, or Traefik settings in traefik.values that use the host network, host ports or capabilities beyond the baseline set, are rejected.

secretgen-controller namespace

The carvel-secret-gen app now deploys into the namespace secretgen-controller. The vendored release always created that namespace and put every resource in it, so Argo CD also created an empty carvel-secret-gen namespace. That empty namespace stays behind: delete it with kubectl delete namespace carvel-secret-gen. The workloads do not move. apps.carvelSecretGen.outputs.namespace and the Pod Security labels above follow the new namespace.

Keycloak operator permissions

• The view ClusterRole is bound to the operator's ServiceAccount only in the operator's namespace now, as upstream ships it (a RoleBinding instead of a ClusterRoleBinding). The cluster-wide read access to pods, services and other namespaced objects goes away. The operator's other permissions come from its own ClusterRoles, which stay bound cluster-wide.

Metric labels (can break your own queries, alerts and dashboards)

The new series start at the upgrade; the old ones stay queryable until the retention removes them.

• Alloy's node scrapes now use the kube-prometheus job names, and the kubelet series have metrics_path and node labels.

  Before	After
  job="prometheus.scrape.kubelet_scrape"	job="kubelet", metrics_path="/metrics"
  job="prometheus.scrape.kubelet_cadvisor_scrape"	job="kubelet", metrics_path="/metrics/cadvisor"
  job="prometheus.scrape.kubelet_probes_scrape"	job="kubelet", metrics_path="/metrics/probes"
  job="prometheus.scrape.api_server_scrape"	job="apiserver"
  job="prometheus.scrape.kube_scheduler_scrape"	job="kube-scheduler"
  job="prometheus.scrape.kube_controller_manager_scrape"	job="kube-controller-manager"
  job="prometheus.scrape.kube_proxy_scrape"	job="kube-proxy"

• Every series sent by Alloy gets cluster="<observability.clusterName>" (an external label of both remote_write blocks). A series that already has a cluster label keeps it, so CloudNativePG's cnpg_* metrics still carry the PostgreSQL cluster name. Mimir's own series changed from cluster="mimir-distributed" and Loki's from cluster="loki" to the same value; Loki's recording rules follow. The community dashboards select on this label.

• Every log stream sent by Alloy gets the same cluster label (an external label of both loki.write blocks). Pod logs had none before, only the Kubernetes events did. A stream that already has a cluster label keeps it.

• The kube-scheduler, kube-controller-manager and kube-proxy scrapes are now off by default: they listen on 127.0.0.1 on kubeadm and Talos, so they were always down. Set observability.collector.metrics.controlPlane.{kubeScheduler, kubeControllerManager, kubeProxy} to true after binding them to the node address.

Mimir and Loki self-monitoring

• observability.selfMonitoring.enabled (default true, as before) turns the scrapes of Mimir's and Loki's own metrics (10 ServiceMonitors for Mimir; Loki's every 15s) and the Mimir and Loki dashboards and recording rules on and off. Turn it off on small clusters to save storage. Alerts that read these series (discarded samples and logs, Kafka ingestion delay) then never fire; the pod, StatefulSet and kube-state-metrics alerts are not affected.

Grafana

• New dashboards from dotdc/grafana-dashboards-kubernetes 3.0.8: Kubernetes / Views / Global, Namespaces, Nodes and Pods, and System / API Server. They need the labels above (cluster, and node on the kubelet series), so they stay empty for data collected before the upgrade. The Nodes view also needs node_uname_info's nodename to equal the Kubernetes node name, which holds for the node-exporter DaemonSet.
• Nine more Mimir dashboards next to Overview and Writes: Reads, Queries, Compactor, Object Store, Alertmanager, Ruler, and the Overview, Writes and Reads resources ones. The mimir-recording-rules PrometheusRule gets the rule groups they read (mimir_querier_api_2, mimir_ingester_queries_1 and _2). Their disk panels stay empty: cAdvisor's instance is the node IP with port 10250, node-exporter's the one with port 9100.
• The Loki Writes dashboard now shows data in the singleNode profile too (it selected only the SimpleScalable job loki-write; Loki's single binary has the job loki), and there is a Loki Reads dashboard. Both are the Loki chart's own, with the job matcher made optional. Loki's operational, resources and logs dashboards are not shipped: they also select SimpleScalable pod names.

Resource requests of kube-state-metrics and node-exporter

Both had none. They now request CPU and memory (no limits), which restarts their pods once: node-exporter 20m and 32Mi on every node; kube-state-metrics 20m and 128Mi for singleNode and small, 50m and 256Mi per shard for large. These are conservative starting values, not measured.

Metrics of Grafana and grafana-operator

Both now have a ServiceMonitor, which Alloy scrapes into Mimir. They are on by default and off with observability.collector.enabled: false. The Prometheus Operator CRDs that prometheus-crds installs have to exist before they sync. The Grafana ServiceMonitor selects the Service the operator creates for the Grafana resource by the labels of that resource.

Loki usage reporting is off

Loki no longer sends anonymous usage statistics to Grafana Labs (analytics.reporting_enabled: false), as Mimir already did. The config change restarts the Loki pods once.

Mimir's Kafka log is capped

Mimir's Kafka kept 24 hours of writes with no size limit on a 5Gi volume, and a full volume stops the whole write path. Kafka now also gets KAFKA_LOG_RETENTION_BYTES and KAFKA_LOG_SEGMENT_BYTES (128MiB), so the partitions in use together stay below 60% of the volume: 2944MiB per partition for singleNode, 896MiB for small and large. Kafka deletes by whichever limit is reached first, so an ingester that is down while more than the cap is written loses what it has not read yet (as it already did after 24 hours). The change restarts the Kafka pod once. The volume size does not change. If you raise kafka.persistence.size or the number of ingesters through mimir.values, set both variables in kafka.env as well.

Volume sizes of Mimir, MinIO and Kafka (recommendation for fresh installs)

The profiles do not change any volume size: a StatefulSet cannot change its volumeClaimTemplates, so Argo CD would fail to sync with "field is immutable". What the chart and this repository set today:

Volume	singleNode, small	large	Chart preset small.yaml	Chart preset large.yaml
Mimir ingester	2Gi (chart default)	20Gi	50Gi	50Gi
Mimir store-gateway	2Gi (chart default)	20Gi	10Gi	50Gi
Mimir compactor	2Gi (chart default)	20Gi	20Gi	50Gi
Mimir alertmanager	1Gi (chart default)	1Gi	no size (1Gi)	no size (1Gi)
Mimir Kafka	5Gi (chart default)	5Gi	not set	not set
Mimir MinIO, per drive	10Gi, 2 drives	10Gi, 4 pods	MinIO disabled	MinIO disabled
Loki MinIO, per drive	10Gi, 2 drives	10Gi, 2 drives

The chart's presets are for about 1 million (small.yaml) and 10 million (large.yaml) series at a 15s scrape interval, and keep blocks in an external object store instead of MinIO. The sizes above are a recommendation for a fresh install with that load, not something measured here. This repository keeps blocks for 30 days (observability.retention), and a cluster with many more series than the defaults assume needs more local space than these defaults. Set the sizes before the first sync through mimir.values (ingester.persistentVolume.size, store_gateway.persistentVolume.size, compactor.persistentVolume.size, minio.persistence.size, kafka.persistence.size) and loki.values (minio.persistence.size).

To grow a volume of a running cluster:

1. Check that the StorageClass allows expansion: kubectl get storageclass shows ALLOWVOLUMEEXPANSION true.
2. Set the new size in mimir.values (or loki.values) and commit it. Argo CD cannot apply it to the existing StatefulSet until step 4.
3. Expand every PVC of the StatefulSet: kubectl -n mimir patch pvc storage-mimir-distributed-ingester-zone-a-0 -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'.
4. Delete the StatefulSet without its pods, so Argo CD recreates it with the new size in its template: kubectl -n mimir delete statefulset mimir-distributed-ingester-zone-a --cascade=orphan.

Volumes cannot shrink. After you raise kafka.persistence.size, adjust the Kafka log limits as described above.

Since v0.0.146 (0.1.x, 0.2.x)

Upstream dependency updates since v0.0.146 removed nothing this repository uses, but several change behaviour.

Grafana Mimir (chart 6.0.6 to 6.2.1, Mimir 3.0 to 3.2)

• Mimir 3.2 requires all queriers to run 3.1 first (remote execution is on by default). Going straight from 3.0 to 3.2 can cause query errors while old and new pods run side by side.
• The bundled MinIO server image is now pgsty/silo. Verify that existing MinIO volumes open correctly.
• rollout-operator goes from 0.37 to 0.50. Make sure the updated CRDs are applied by the sync.
• Kafka pods now also receive global.extraEnv, which carries the MinIO/S3 secret environment variables.

Keycloak

• 26.8: identity provider mappers can no longer grant admin roles or admin-carrying groups unless allowAdminRoleMapping is enabled on the provider. Set allowAdminRoleMapping: true under iam.realms.<realm>.identityProviders.<id> for providers that carry such a mapper (default false; the key is then left out of the provider config). Keep it set while the mapper exists, otherwise the operator disables the flag again and the next mapper write fails.
• 26.7: the alias of an identity provider can no longer be changed.
• To migrate later: ClusterKeycloak.spec.secret is deprecated in favour of auth, and the Keycloak resource k8s.keycloak.org/v2alpha1 in favour of v2beta1.

EDP keycloak-operator (1.28 to 1.35)

• Admission webhooks are enabled by default and require cert-manager. They use failurePolicy: Fail, so on a fresh bootstrap creating a KeycloakClient fails until the webhook certificate is issued. Argo CD retries until the cluster converges.

cert-manager (1.17 to 1.21)

• Certificate.spec.privateKey.rotationPolicy defaults to Always and can no longer be switched back: a new private key is generated at every renewal.
• The path label of the ACME client metrics was replaced by action.

Observability

• The bundled MinIO of Mimir and Loki no longer gets the charts' default users (console/console123 with consoleAdmin in Mimir, logs-user/supersecretpassword in Loki; minio.users: []). Users that an earlier sync already created stay in the MinIO volume: remove them by hand (mc admin user remove <alias> console, and logs-user in Loki). Until then any pod in the cluster can use them.
• The Traefik dashboard in Grafana now resolves its datasource. It was mapped to a datasource named mimir, which does not exist (the datasources are named mimir/<tenant>), so every panel failed. The dashboard now uses its own datasource dropdown, which defaults to the first Prometheus datasource. The Traefik ServiceMonitor also sets honorLabels: true, so Traefik's service label is kept on scrape instead of being replaced by service="traefik" (the original value went to exported_service). Series stored before the upgrade still carry the old labels until retention removes them.
• node-exporter now excludes some filesystems and mount points by default, so fewer node_filesystem_* series exist.
• Alloy collected no pod logs before: loki.source.podlogs only tails pods selected by PodLogs objects and none were created. Set observability.collector.logs.allPods: true (default false) to render one cluster-wide PodLogs (all-pods, empty selectors) so the logs of all pods in all namespaces are sent to Loki, labelled with namespace, pod and container. This raises Loki's ingest volume and stores the logs of every pod, including any secrets they print. Your own PodLogs objects keep working with or without it.
• Alloy now also sends Kubernetes events to Loki (the collection was defined but never enabled), and the pod label of pod logs no longer contains the container name (pod:container).
• Grafana: the viewer fallback of sso.rbac (viewer: 'true') was a field lookup, not a boolean, and never matched, so users only became Viewer through Grafana's default role. The defaults now use JMESPath literals (`true`, `false`), so every user without a higher role gets Viewer explicitly. To refuse the login of users that match no role, set sso.roleAttributeStrict: true (grafana.sso in the config, default false) together with a sso.rbac.viewer rule that does not match everybody, e.g. "contains(roles[*], 'viewer')".
• Alloy skips ServiceMonitors that point at local files (bearerTokenFile, tlsConfig.caFile, ...) unless allow_arbitrary_file_access is set. This only matters for ServiceMonitors created outside this repository.
• node-exporter no longer listens on all interfaces (service.listenOnAllInterfaces: false). It binds to the node's status.hostIP, which is the address the PodMonitor scrapes, so port 9100 is no longer open on the other addresses of the node. If the node's InternalIP is a public address (for example Talos nodes that only have public IPs), port 9100 stays reachable on it: restrict the addresses the kubelet registers (kubelet.nodeIP.validSubnets) or add a host firewall.
• node-exporter is scraped through a PodMonitor, and kube-state-metrics keeps the namespace, pod, container, service and endpoint labels of the objects it describes. Series that used to collide, or to lack those labels, are different from before.
• The bucket Job of the bundled MinIO (Mimir and Loki) reported success when MinIO never became reachable within its retry window, without creating any bucket. Loki and Mimir then failed with NoSuchBucket (404). The Job now waits up to 5 minutes and fails, so Kubernetes retries it, if MinIO stays unreachable. A cluster where a slow MinIO start already caused this still lacks the buckets: create them by hand (mc mb target/<bucket> in a pod with the alias configured, with chunks and ruler for Loki and mimir-tsdb and mimir-ruler for Mimir) or re-run the hook job by syncing the application.
• The Mimir ingesters of the small (default) and large profiles now request 700m CPU and 2Gi memory with a 3Gi memory limit (before: the shared 100m/256Mi with a 512Mi limit for small, 300m/ 512Mi with 1Gi for large). The old limits got every ingester OOMKilled right after the start, while it replayed the Kafka backlog. For the 3 ingesters this adds about 5.25Gi of memory requests and 1.8 CPU requests in small, and 4.5Gi and 1.2 CPU in large. Override it with mimir.values.ingester.resources if the cluster cannot spare that. The single singleNode ingester requests 700m CPU (before: 1500m) and keeps 6Gi memory with a 9Gi limit. Neither number has been measured on a single, unsharded ingester yet.
• New alerts on Mimir and Loki (restarting or OOMKilled pods, unready StatefulSets, discarded samples and logs, Kafka ingestion delay, missing kube-state-metrics data) are mailed to observability.alerting.email.defaultReceivers. They are PrometheusRules that Alloy loads into the Mimir ruler, which needs observability.storage.metrics.enabled. Nothing fires while Mimir's write path, ruler or Alertmanager is down, or while all ingesters are down at once (the ruler's queries need them, e.g. an OOM loop of every zone); only a dead man's switch outside the cluster covers that. Partial failures, such as one looping zone, are alerted. The recording rules of the Loki chart are now loaded into the ruler as well.
• Grafana gets the dashboards Mimir / Overview, Mimir / Writes and Loki / Writes, plus a mimir-recording-rules PrometheusRule with the Mimir mixin's recording rules that the two Mimir dashboards read.
• In the singleNode profile, Mimir's Kafka broker requests 100m CPU (before: the chart's 1 CPU), keeps 1Gi memory and has no limits. On a small node the old request left it Pending, and the distributor, ingester and ruler crash-looped because they could not create the mimir-ingest topic. small and large keep the chart default. Override with mimir.values.kafka.resources.
• New observability.storage.storageClass (default null: the cluster's default StorageClass) sets the class of every Mimir and Loki volume: Mimir's ingester, store-gateway, compactor, Alertmanager and Kafka, Loki's StatefulSets, and both bundled MinIOs. Kafka ignored per-component settings because its chart key is kafka.persistence.storageClassName. Set it before the first sync. On a running cluster the StatefulSets' volumeClaimTemplates are immutable, so they have to be recreated and their PVCs migrated or re-created. Settings in mimir.values/loki.values still take precedence.

Argo CD (3.3 to 3.5)

• The Argo CD dashboard's controller panels (memory, CPU, goroutines) now filter on job="argocd-application-controller-metrics". They stay empty if the controller is scraped under another job name.