Skip to content
antalos

Grafana and VictoriaMetrics · Operate

This runbook deploys the service from apps/victoriametrics/ and completes the configuration that Kubernetes cannot supply by itself. Start with the shared deployment workflow for repository rendering, credentials, and Argo CD ownership.

1. Prepare dependencies and inputs

  1. Prepare CNPG and OpenEBS, then set the monitoring storage variables, GRAFANA_HOST, and VICTORIAMETRICS_CHART_VERSION.

  2. Reseal grafana-db-app. Keep Grafana’s external PostgreSQL settings and startup permissions intact across chart updates.

  3. Account for the dedicated log-agent and node-exporter namespaces and their host-access requirements.

2. Reconcile the application

Publish the reviewed manifests and shared variables to the branch tracked by your Argo CD Applications. Local edits are not deployment inputs until that branch contains them. Let the root app-of-apps discover this service and retain its declared hook order; do not apply files containing unresolved variable placeholders directly.

From the repository root, inspect the Application and its namespace:

Inspect deployment state
/home/linuxbrew/.linuxbrew/bin/kubectl --kubeconfig kubeconfig \
  -n argocd get application victoriametrics


/home/linuxbrew/.linuxbrew/bin/kubectl --kubeconfig kubeconfig \
  -n monitoring get pods,svc,pvc

Wait for required controllers, database initialization, and migration Jobs before testing the user workflow. Synced configuration, ready workloads, and a successful user transaction are separate milestones.

3. Complete identity and first access

The repository supplies the Generic OAuth role mapping (grafana-admins becomes organization Admin, otherwise Viewer) and enables role synchronization. The provider connection, credentials and enablement remain separately configured. Use the generated Grafana administrator Secret through your private credential workflow, then create appropriate viewer/editor access. If adding Authentik later, configure Grafana Generic OAuth with its exact /login/generic_oauth callback and explicit role mapping, using a sealed client secret. Document whether UI changes or Git own each setting.

4. Connect and operate the service

  1. Confirm the provisioned data sources and dashboards can query real samples. Check scrape targets before interpreting missing metrics as zero.

  2. Keep availability queries scoped to each primary serving pod. Background Jobs and unrelated namespace pods should not drive a service’s UP/DOWN tile.

  3. Export durable dashboard changes into apps/victoriametrics/ and define Grafana database backup and metrics/log retention policies.

Availability before maintenance

Mixed availability: Grafana and metrics have replicated designs; the current log storage is sharded, not redundantly copied.

Grafana uses zero surge/one unavailable with a PDB. Metrics and logs have role-specific updates; two storage pods do not mean a log-storage rolling restart has complete-history query availability. The single operator may pause reconciliation during replacement.

Test Grafana, metric query completeness, degraded metric ingestion, and log completeness separately. Protect all storage shards and the Grafana database. Add a supported independent-copy log architecture if full log-history HA is required.

Use the component-by-component failure contract before a node drain, database promotion, or upgrade. Restoring a healthy replica count must include data resynchronization and restored voting capacity, not just Running pods.

5. Maintain and recover

Preserve Grafana PostgreSQL data for UI-created settings and dashboards that have not been exported to Git. Metrics and logs reside on node-local storage with configured retention. The legacy SQLite PVC is retained as migration-era data and is not the active Grafana database.

Before an upgrade, read the release notes for the pinned target and record a recovery point. A previous image tag is not a database rollback after a schema migration. Use routine operations and disaster recovery for the platform sequence.

Troubleshooting

For missing data, follow collection, ingestion, storage, then query layers. If Grafana works but a panel is empty, inspect the query labels and time range. If HPA resource metrics are missing, inspect metrics-server; Grafana’s historical metrics pipeline is a separate system.

Manifest and upstream reference

Official Grafana and VictoriaMetrics documentation.