Headscale and Headplane · Operate

This runbook deploys the service from apps/headscale/ and completes the configuration that Kubernetes cannot supply by itself. Start with the shared deployment workflow for repository rendering, credentials, and Argo CD ownership.

1. Prepare dependencies and inputs

Set the HEADSCALE_* and HEADPLANE_* variables, including DNS, issuer, allowed email domains, and storage. HEADSCALE_OIDC_ALLOWED_DOMAIN and HEADSCALE_OIDC_ADDITIONAL_ALLOWED_DOMAIN supply the enrollment domain allowlist. Both require verified email addresses. The pod template records these domains so changing either input restarts Headscale to load the updated configuration.
Prepare the NFS export and reseal headscale-oidc plus headplane; the first includes both the OIDC client secret and the Headscale API credential used by Headplane.

2. Reconcile the application

Publish the reviewed manifests and shared variables to the branch tracked by your Argo CD Applications. Local edits are not deployment inputs until that branch contains them. Let the root app-of-apps discover this service and retain its declared hook order; do not apply files containing unresolved variable placeholders directly.

From the repository root, inspect the Application and its namespace:

/home/linuxbrew/.linuxbrew/bin/kubectl --kubeconfig kubeconfig \
  -n argocd get application headscale

/home/linuxbrew/.linuxbrew/bin/kubectl --kubeconfig kubeconfig \
  -n headscale get pods,svc,pvc

Wait for required controllers, database initialization, and migration Jobs before testing the user workflow. Synced configuration, ready workloads, and a successful user transaction are separate milestones.

3. Complete identity and first access

Both applications share the configured Authentik OIDC client. Register Headscale’s /oidc/callback and Headplane’s /admin/oidc/callback as exact HTTPS redirect URIs on that provider. Match PKCE and the configured client_secret_basic token authentication method. Headplane disables API-key browser login and defaults new OIDC users to member; establish an explicit administrator assignment using the Headplane version’s role-management procedure. Do not confuse its backend API credential with a user login method.

4. Connect and operate the service

Generate an appropriate Headscale API key through the administrator workflow and seal it into the referenced Secret. Rotate the key before it expires.
Test client enrollment, private DNS, an allowed connection, and a denied connection. Test Headplane with a non-admin identity.
Rehearse restore of both databases and retain out-of-band cluster access before changing the network policy or identity provider.

Availability before maintenance

Not continuously HA: Headscale and Headplane each recover by restarting one process.

Single-replica updates interrupt the corresponding service; Headplane uses Recreate. Do not scale SQLite writers to two as an HA shortcut without a supported coordination/storage redesign.

State a measured recovery-time and recovery-point objective, protect NFS and sealing material, and rehearse both restores. Continuous service availability would require a supported application/storage topology beyond the current singleton design.

Use the component-by-component failure contract before a node drain, database promotion, or upgrade. Restoring a healthy replica count must include data resynchronization and restored voting capacity, not just Running pods.

5. Maintain and recover

Preserve both Litestream backup directories, Headscale configuration and key material included in the storage arrangement, the Headplane cookie secret, and sealed OIDC/API credentials. A surviving backup is needed when the disposable SQLite volume is recreated.

Before an upgrade, read the release notes for the pinned target and record a recovery point. A previous image tag is not a database rollback after a schema migration. Use routine operations and disaster recovery for the platform sequence.

Troubleshooting

If Headscale will not start, check issuer discovery and NFS restore first. If the VPN works but Headplane fails, check its backend API key and separate database. A policy denying a connection can be correct behavior even when both services are healthy.

An OIDC callback returning 401 with authenticated principal does not match any allowed domain means the authenticated email domain is outside the enrollment allowlist. An existing user record does not bypass this check. Update the domain inputs in apps/variables.yaml, publish the change, and confirm Argo CD reconciliation and the Headscale rollout before retrying enrollment. Start a fresh enrollment in the client if an old callback reports a missing state cookie.

Manifest and upstream reference

Official Headscale and Headplane documentation.

Headplane SSO and role mapping documents the callback and backend API-key requirements.