Service availability

antalos has mixed availability. Several serving tiers can use a healthy replica after one worker fails. Others must restart a single process or restore a database before serving again. Replicated applications can still lose functionality through shared storage, a singleton proxy, or an unavailable control-plane endpoint.

This reference describes the current repository design. It does not report live readiness, a completed node-loss test, an uptime percentage, or a measured recovery time. Follow each service link for its component counts, failure sequence, upgrade behavior, and remaining work.

Which applications are HA?

Non-HA scheduled jobs: Renovate runs one stateless pod at a time on any eligible node. Interrupted scans retry or wait for the next schedule; there is no replicated running instance or persistent local data.
HA serving designs: the documentation site and BentoPDF have interchangeable stateless replicas; Traefik has two serving proxies; Rancher has a replicated management web tier. These still depend on the shared network/control-plane paths. The docs rollout has a placement limitation described below.
Replicated stateful designs: LiteLLM, Grafana, and the VictoriaMetrics metrics tier include application/data redundancy and a client path to surviving members. They can have a failover interval and reduced durability/capacity during degradation. They are not guarantees against every whole-host or external-service failure.
Partially HA applications: Authentik, Nextcloud, Obsidian, Open WebUI, SuiteCRM, Stalwart, and Zammad have replicated important components but retain shared dependencies, singleton features, restart/rejoin concerns, or upgrade outages. The scope of each limitation matters more than the replica total.
Not continuously HA: Headscale, Headplane, Vaultwarden’s application, UrBackup, Uptime Kuma, CrowdSec’s singleton roles, and RustDesk rendezvous each run one primary application process. Kubernetes restart and persistent backups provide recovery, with an outage. RustDesk’s two native relays improve relay choice but do not make rendezvous HA.
Not replicated log storage: VictoriaLogs has two storage processes holding shards, not two complete copies. Grafana availability must not be used as evidence that all historical logs are available.
External or not established here: NFS/Garage server HA, public routing/DNS resilience, and the external Talk recorder require their own architecture and evidence.

What the labels mean

Term	Meaning in these guides
HA serving tier	Another ready process can serve new requests when one serving process/node fails, assuming its dependencies remain usable.
HA data design	Independent data copies, a supported promotion/election mechanism, and a client path to the surviving writer/reader. A transition can interrupt requests.
Partially HA	Some components meet those conditions, but an important workflow or dependency still has an outage path.
Recovery-based / not continuously HA	Service returns by restarting, rescheduling, remounting, or restoring one process/data copy; there is no hot alternative for that function.
Not established by manifests	Replica/default behavior or an external dependency has not been explicitly specified sufficiently to support the claim.

No service should be described as unconditionally end-to-end HA solely because it has two pods. Define the failure being tolerated and the user transaction that must still work.

Application-by-application classification

Service	Availability conclusion	Mechanism and limits
Mautic	Partially HA: two web/worker pods, two Galera members plus garbd, two Redis data pods and three Sentinels.	Shared NFS, API/operator dependencies, scheduled jobs and schema upgrades retain interruption paths; failover requires separate testing.
Authentik	Partially HA: replicated identity processing and database; external shared media and maintenance limits remain.	2; required anti-affinity; see the linked component table for the data, routing, and recovery path.
BentoPDF	HA static serving tier; access depends on the shared identity and ingress services.	2 stateless pods on separate nodes; see the linked component table for the data, routing, and recovery path.
Documentation	Replicated static serving tier with placement-compatible rolling updates.	2 stateless replicas; required host anti-affinity; see the linked component table for the data, routing, and recovery path.
Forgejo	Two serving pods and redundant database/Redis members; shared NFS and Garage paths limit whole-service HA.	Two anti-affined application pods; see the linked architecture page.
Headscale / Headplane	Not continuously HA: Headscale and Headplane each recover by restarting one process.	1 Recreate Deployment pod with a backup-restore migration gate; see the linked component table for the data, routing, and recovery path.
LiteLLM	HA design for a single data-worker loss, conditional on healthy control-plane, Sentinel communication, and upstream providers.	2 anti-affined replicas on the main workers; see the linked component table for the data, routing, and recovery path.
Medusa	Replicated backend and worker design for one data-worker loss; shared Garage and ingress remain dependencies.	Two servers, two workers, two CNPG instances, two Redis data members and three Sentinel voters; failover can interrupt work.
Nextcloud	Partially HA overall: replicated web and many companions, with shared storage, session, Redis recovery, and upgrade limits.	2 anti-affined web-sidecar pairs; see the linked component table for the data, routing, and recovery path.
Open WebUI	Partially HA: steady-state replicas exist, but Recreate upgrades and Redis/storage dependencies can interrupt all users.	2 anti-affined pods on main workers; see the linked component table for the data, routing, and recovery path.
Rancher	HA management web tier; it is not an independent control plane for recovering this cluster.	2 replicas with required host anti-affinity; see the linked component table for the data, routing, and recovery path.
RustDesk	Not continuously HA end to end: one recoverable rendezvous server, with two independent native relays.	1 Deployment replica; see the linked component table for the data, routing, and recovery path.
Stalwart Mail	Replicated mail/data tiers with two independent Redis proxies; shared storage and search allocation remain limits.	2 anti-affined Stalwart replicas; see the linked component table for the data, routing, and recovery path.
SuiteCRM	Partially HA: replicated web and Galera data, but background workers and shared files have separate outage paths.	2 anti-affined replicas; see the linked component table for the data, routing, and recovery path.
UrBackup	Not HA: one backup server, restored with two external NFS exports.	1 Deployment replica; see the linked component table for the data, routing, and recovery path.
Vaultwarden	Not HA at the application layer: one vault server; PostgreSQL alone is replicated.	1 Recreate replica; see the linked component table for the data, routing, and recovery path.
Grafana / metrics / logs	Mixed availability: Grafana and metrics have replicated designs; the current log storage is sharded, not redundantly copied.	2 anti-affined pods + 2 CNPG instances; see the linked component table for the data, routing, and recovery path.
Zammad	Partially HA: paired HTTP tiers and data services, with singleton real-time/background roles and Redis/search caveats.	2 NGINX + 2 Rails replicas; see the linked component table for the data, routing, and recovery path.
Obsidian LiveSync	Partial availability; two independent databases with asynchronous copying.	2 HAProxy replicas, primary/backup routing, 2 standalone CouchDB members, bidirectional vault replication.
Uptime Kuma	Recoverable singleton with a monitoring gap during replacement.	One process, pod-local SQLite, Litestream replica on external NFS.
Node-RED	Recoverable singleton with an automation gap during replacement.	One process, persistent `/data` on external NFS; in-memory context and in-flight messages are lost.
CrowdSec	Singleton LAPI, processor, and AppSec; enforcement impact depends on the consumer.	LAPI SQLite recovery through Litestream; VictoriaLogs and Traefik are separate dependencies.

Odoo has two serving replicas and two CNPG PostgreSQL instances separated across the left/right workers. Shared sessions and imported legacy attachments live in PostgreSQL; new attachments use Garage S3. Serving pods do not depend on a worker-local PVC. Online backups preserve availability, with consistency relying on retained immutable S3 attachment versions. Database promotion can interrupt requests, and shared ingress, API and Garage remain dependencies. See Odoo architecture.

How the HA mechanisms work

Ready replicas and client routing

A typical stateless path is client → ingress → Service → ready application endpoint. Required pod anti-affinity separates replicas across Kubernetes node names. If a member fails, the system must detect that condition and route new traffic to a surviving ready endpoint. Probe intervals, endpoint updates, network convergence, and client retry behavior all contribute to the interruption.

A readiness probe only establishes the condition it actually checks. A listening TCP port may not prove database writability or a successful object upload. New-request availability also says nothing about moving an in-flight stream, WebSocket session, mail connection, or TURN allocation between processes.

PostgreSQL promotion

CNPG manages a primary and a streaming standby for each two-instance cluster. They use separate OpenEBS volumes; the second copy comes from PostgreSQL replication, not from the StorageClass. If the primary fails, the surviving standby must become the writer and the read/write Service must identify it. Clients must reconnect. The operator and Kubernetes API are part of this failover path.

Application database	Explicit policy in this repository	Durability implication
Authentik, Nextcloud/Context Chat, Stalwart	No synchronous stanza	Default asynchronous replication; recent primary writes may not yet be on the standby.
LiteLLM, Open WebUI, Vaultwarden, Zammad, Grafana	any / 1 / preferred	Requests synchronous acknowledgement while possible, but permits degraded operation without the standby.

These settings do not promise zero loss under every failure sequence. See CNPG replication and durability for the mechanisms; use documentation matching the installed operator when administering it.

Redis election and rejoin

Most bespoke Redis layouts use two data members and three Sentinel voters, one voter in each worker failure domain. With all three initially healthy, losing one data worker leaves one data copy and two voters. Election still requires communicating voters and an eligible replica; a Sentinel-only pod cannot serve application data.

Consumer	Client path	Restart/rejoin behavior visible in its manifests
LiteLLM	Direct authenticated Sentinel discovery	Data-side Redis role and Sentinel topology persist; startup queries peers before choosing a role.
Open WebUI	Direct discovery; Sentinel listener auth follows its client contract	Data-side roles and Sentinel topology persist on the existing PVC; startup consults Sentinel before rejoining. Third voter uses stable identity and temporary state.
Zammad	Direct Sentinel discovery	Data-side roles and Sentinel topology persist on the existing PVC; startup consults Sentinel before rejoining. Third voter uses stable identity and temporary state.
Nextcloud	Two HAProxy replicas checking Redis ROLE with DNS refresh	Redis role changes and data-side Sentinel topology persist on existing data PVCs; the third voter has a stable identity and temporary state. Proxy replication does not elect or fence writers.
Stalwart	Two independent HAProxy replicas checking ROLE with DNS refresh	Data-side roles and Sentinel topology persist on existing PVCs; startup consults Sentinel. The third voter remains on RTX, independent of client proxies.

The migrated scripts no longer reset existing roles by ordinal. First migration of an existing dataset requires agreement from two voters, and persisted state alone does not prove partition safety. Redis replication remains asynchronous. A useful failure drill includes the old primary’s return and confirms that all clients converge on one writer.

Keep data, replica, voter, and client authentication consistent. Password-only Sentinel setups use matching requirepass values for peer authentication; a missing separate sentinel-pass directive alone does not prove a defect. Redis replication is asynchronous. See Redis Sentinel architecture and authentication.

Quorum is not a data copy

System	Third participant	What it protects	What it does not supply
SuiteCRM Galera	garbd on RTX	Membership majority with one surviving data node	A SQL-serving process or third database copy
Redis	Sentinel on RTX	Election voting	Another Redis dataset
Elasticsearch	Master-only node on RTX	Master-election majority	Replica shards for indices
Talk messaging	Third NATS Core member	Redundant messaging connectivity	Persisted call media or a durable user-data quorum

Galera’s connected majority can retain its Primary Component after a member loss; forced rebootstrap during a partition is a different recovery operation. See Galera quorum.

Elasticsearch also needs appropriate replica shards allocated across data nodes. Two data processes and a tiebreaker protect election, but do not by themselves establish a second copy of every index. See Elastic’s small-cluster resilience guidance.

Metrics replication versus log sharding

VictoriaMetrics declares a metrics replication factor of two and duplicate-scrape deduplication. A previously replicated sample can remain readable from a surviving copy. During degraded operation, fresh data may not receive two copies, and restoring a second storage process must not be assumed to backfill every missing historical sample automatically.

VictoriaLogs uses a different design: vlinsert distributes data across vlstorage shards. Losing one storage node removes access to that node’s history and can prevent complete queries. The remaining storage node is not its mirror. An explicit supported independent-copy ingestion/query architecture is needed for complete-history log HA. See VictoriaLogs replication and availability.

Restart and restore

Headscale, Headplane, and RustDesk hbbs keep SQLite on local pod storage and copy it with Litestream to NFS. Replacement requires a usable backup, the right identity/credentials, and a safe single-writer startup. Backup interval is not the maximum possible data-loss interval when storage is slow or unreachable.

Vaultwarden and UrBackup also have single application processes, even though their state persists elsewhere. They return through replacement/restart, not through a ready hot application replica. Document that outage instead of presenting rescheduling as uninterrupted failover.

The Debian VM layer has its own availability limits. Arc hosts recording, Immich Machine Learning, Jellyfin, LazyLibrarian, Storyteller, and Docling. RTX hosts llama-swap and Speaches. The left playbook configures a base Debian host without deploying Compose workloads. Each managed VM endpoint is a singleton; its loss interrupts its consumers’ corresponding features. Manually installed applications require a separate inventory.

Shared failure domains

Physical hosts and the API endpoint

The intended layout places one control-plane VM and one worker VM on each Proxmox host. Required anti-affinity uses Kubernetes node names; it only becomes physical-host separation because of that VM placement. If VMs are moved onto the same host later, node-name separation alone no longer protects against that host’s loss.

Three etcd members can retain a majority after one member fails. The legacy external client API address is 10.30.0.6 on the RTX control-plane VM. infrastructure/opentofu/talostofu/ now supports cluster_api_vip: an opt-in shared address elected by Talos across the control-plane nodes. Reserve an unused address outside DHCP on their shared layer-2 subnet before enabling it. The default remains unset for existing installations; source support is not evidence that a VIP is live.

An external API VIP does not supply worker capacity, database quorum, or redundant storage. In-cluster clients normally use the Kubernetes Service and Talos KubePrism/discovery rather than the external kubeconfig endpoint; inspect those paths separately. Retain direct node addresses for Talos recovery because its VIP election depends on etcd. See Talos virtual IP guidance.

Losing the RTX worker removes its application voters; losing the entire host also removes a control-plane member. A whole-host contract additionally requires physically separate VM placement, surviving storage/network services, and adequate remaining worker capacity.

External storage and network

Shared dependency	Affected paths	Why application replicas cannot replace it
NFS at the declared external endpoint	Authentik media, Nextcloud code/recording content, SuiteCRM files, Vaultwarden data, UrBackup exports, SQLite backups	Every replica or replacement mounts the same external service.
Garage S3 at the declared external endpoint	Nextcloud files, Stalwart blobs, Open WebUI uploads, Zammad attachments, SuiteCRM media/database backups	Separate application pods still address the same object service.
Router, DNS, subnet, switches	Public and internal client access	The repository does not declare independent replacements for every network component.
Authentik	New SSO/forward-auth access	Application replica counts do not duplicate the identity service’s database/storage path.
CNPG/operator/API	Controlled state changes and database recovery	A standby does not independently guarantee the required controller action.

The NFS and Garage defaults reference the same external host address. Its loss can affect several services at once. The repository does not establish whether an external implementation has its own failover; document that separately before assuming it.

Availability during maintenance

Configuration	Effect	Current examples
2 replicas, zero surge, one unavailable	Replaces one pod without needing a third anti-affined placement; leaves reduced capacity	LiteLLM proxies, BentoPDF, Traefik, Nextcloud web and Redis HAProxy, SuiteCRM web, Grafana
2 replicas, one surge, zero unavailable, hard anti-affinity	Needs a third eligible node with capacity for a replacement pod	Documentation serving pods
Recreate application	Can stop the full application during an upgrade regardless of steady-state replicas	Open WebUI, Vaultwarden; also single Headplane and Stalwart’s Redis proxy
PDB minimum 1 on a singleton	Blocks ordinary eviction; does not make a hot replacement or prevent its workload-controller update	RustDesk hbbs, UrBackup
Chart defaults without explicit overrides	Requires the actual chart policy before claiming a particular maintenance guarantee	Rancher and several platform controllers

A PDB limits voluntary evictions; node failure and workload-controller rolling changes have different rules. It is not a minimum healthy-pod guarantee under every event. See Kubernetes disruption behavior.

Database migrations, reindexing, cache elections, and stateful sessions can still need a maintenance interval. Follow the application administrator guide instead of treating a rolling Deployment as a zero-downtime schema upgrade.

Platform-service classification

Service	Availability conclusion	Detailed explanation
Argo CD	Partially HA overall: replicated API/rendering and Redis HA design; reconciliation and SSO are not proven by those counts.	Replica, failure, and maintenance contract
Traefik	HA ingress process tier for one serving-node loss; upstream routing and established connections remain separate.	Replica, failure, and maintenance contract
MetalLB	Partially HA: distributed address advertisement; controller allocation and the physical network remain separate.	Replica, failure, and maintenance contract
cert-manager	Partially HA: replicated admission webhook; certificate issuance is not fully replicated by the manifest.	Replica, failure, and maintenance contract
CloudNativePG operator	Redundant operator management with leader election; managed database availability remains service-specific.	Replica, failure, and maintenance contract
MariaDB operator	HA management design: replicated operator and webhook; database HA still belongs to each managed MariaDB.	Replica, failure, and maintenance contract
Metrics Server	Not explicitly HA: resource-metrics collection can pause until its chart workload recovers.	Replica, failure, and maintenance contract
NFS CSI	Not HA storage: the CSI path has node coverage, but all declared exports depend on one external endpoint.	Replica, failure, and maintenance contract
OpenEBS	Not replicated storage: individual LocalPV volumes cannot survive loss of their owning disk as live copies.	Replica, failure, and maintenance contract
Sealed Secrets	Not explicitly HA as a controller; existing generated Secrets remain available independently of its process.	Replica, failure, and maintenance contract

Argo CD’s controller replicas are configured with cluster sharding. They do not by themselves prove hot takeover of the one managed cluster’s work. Native OIDC removes the Dex singleton from SSO; Authentik, its database, ingress, DNS, and eligible spare capacity remain dependencies. Auxiliary chart components need their own replica assessment. See Argo CD HA and sharding.

Describe recovery objectives honestly

Recovery time (RTO) is the interruption until the required user workflow works again. Recovery point (RPO) describes the data that may be absent after recovery. Neither is established by a pod count, a five-second failure-detection threshold, or a one-second backup interval.

For antalos, the relevant intervals can include failure detection, voting, promotion, Service updates, rescheduling, image download, NFS locks, restore, replay, warm-up, and client reconnect. Correlated operator/API loss adds another dependency. Do not publish a numeric guarantee without measuring the complete path.

Replication is also not a backup: destructive changes or corruption can reach another live member. A recovery set needs independent database, object, file, and key material. See disaster recovery.

What would establish the claimed behavior

These are the evidence an administrator should capture in an authorized drill, not checks executed while editing this guide:

Identify the exact failure: one pod, one worker VM, one Proxmox host, a network partition, or the external storage host.
Confirm initial data copies, voter membership, physical separation, and enough surviving capacity; do not start already degraded.
Perform a real transaction continuously through the event: file upload/download, mail send/read, Git push/clone, model request, backup/restore, or a complete log query as appropriate.
Record the failed and elected writers, lost/interrupted sessions, time to useful service, and any missing data.
Return the old member safely and confirm a single correct writer, completed resynchronization, restored quorum, and complete data/query results.
Record Argo sync/health separately from application results and persist any repair in the source manifests.

Do not begin a second node’s maintenance just because replacement pods are Running. Full redundancy requires healthy copies, correct roles, and usable endpoints again.

Infrastructure map availability

Homelable is a recoverable singleton on Debian Arc. Its HTTPS path also depends on Debian Left Caddy and new sessions depend on Authentik. Running outside Kubernetes preserves the map during some cluster failures, but does not remove shared Proxmox/network dependencies. Local backup archives require a separate off-host copy for disaster recovery.

Scheduling capacity after restart

Both general workers declare max-pods=130 in infrastructure/opentofu/talostofu/main.tf. Keep this symmetric: a recovered worker at the default 110-pod ceiling can leave local-volume database members Pending even when CPU and memory requests fit. Compare scheduled nonterminal pod counts with each node’s allocatable pod count before maintenance. Raising the ceiling does not add memory or move local volumes; retain headroom for replacement and recovery pods.

Reserve the small RTX quorum worker for voters and explicitly assigned infrastructure. Invoice Ninja web, nginx, and queue workers are restricted to the two general workers. Argo CD Redis HAProxy has resource requests and a memory limit, and bounds its connection count. These controls reduce pressure and protect it from BestEffort victim selection; they do not create additional VM memory.

Roll out the node-failure improvements

These changes are source configuration, not a completed live migration. No node shutdown is needed to review or render them.

Restore a healthy baseline before changing topology: three control-plane members, both main workers, the third voter worker, and all application data copies. Investigate NotReady nodes, cordons, disk pressure, missing voters, and failed databases first. A PDB cannot compensate for multiple unavailable failure domains. Do not remove pressure taints to force scheduling.
Run python3 tools/check-node-failover.py from the repository root. This read-only gate reports cordons, pressure, incomplete controller/PostgreSQL replica counts, incomplete PostgreSQL primary transitions, and co-located ready replicas. It also checks requested CPU, memory and pod slots for the listed recoverable singleton replacements on each general worker, including Uptime Kuma, RustDesk rendezvous, CrowdSec, Headscale/Headplane, Talk, Context Chat, Node-RED, Vaultwarden, UrBackup, SuiteCRM messenger and Zammad’s singleton roles. It accounts for init-container peaks, native sidecars and pod overhead, and treats the declared low-priority Renovate batch reservation as reclaimable. This is a lower bound: other controllers/jobs, affinity, taints, volumes, actual resource use and scheduling order still require review. It does not establish database replication or successful user transactions.
Reserve N-1 CPU, memory, pod slots, and ephemeral disk headroom on the eligible workers. The small tainted RTX worker is a voter domain, not capacity for the whole stack. Required anti-affinity deliberately leaves a second replica Pending when only one eligible node remains; the surviving replica still needs healthy dependencies.
Review and publish application changes through the normal GitOps workflow. Argo dynamic distribution changes its application-controller kind and is upstream alpha. Verify both heartbeat-based reassignment and actual reconciliation, not just the UI. Stalwart proxy sync waves create healthy replacement endpoints before changing the Service selector.
Migrate Open WebUI, Zammad, and Stalwart Redis one member at a time. Existing data without agreement from two voters deliberately blocks initial migration. Keep the current writer and two communicating voters available. Data-side writable Redis/Sentinel configs persist on existing PVCs; validate replication and old-primary rejoin before continuing. Keep existing credentials and PVCs.
For external API failover, set cluster_api_vip only after reserving its address and reviewing the OpenTofu plan. Network and machine-configuration application needs explicit maintenance approval. Introduce and verify the VIP and certificate on control planes through direct node management addresses before moving clients. Reconfigure workers/client kubeconfigs in stages, retaining a direct API recovery path. Never use the VIP as the Talos recovery endpoint.
After full redundancy is restored, separately authorize a failure drill for one worker or physical host at a time. Confirm a new authenticated login, application read/write, database promotion, background work, and Argo reconciliation. Bring the node back, verify old-primary demotion and data resynchronization, and restore every voter before the next drill.

Remaining work for an all-services contract

Boundary	Required implementation or operational decision
Recoverable singletons: Headscale/Headplane, Uptime Kuma, Node-RED, Vaultwarden, UrBackup, CrowdSec, RustDesk rendezvous	Existing one-process restart/restore designs permit an outage. StatefulSet replacement on an unreachable node can wait for fencing or node recovery. Establish fencing before force deletion or automated takeover; two writers can corrupt SQLite/shared files or duplicate work.
Local OpenEBS PVCs	A bound local volume stays on its node. Replicated databases can promote another copy; a singleton local volume cannot follow a pod to another worker. Migration to replicated storage is a storage change requiring review, backup, and approval.
Galera databases	Two data members plus an independent arbitrator tolerate one failure only from a healthy communicating membership. Restore data-member health and quorum first; never automate forced bootstrap from replica counts.
VictoriaLogs	Two current shards are independent history, not replicas. Complete-history HA requires a supported replicated ingestion/query and retention design; adding replicas alone does not copy existing history.
Nextcloud Talk and external Debian workloads	Singleton media/recording/GPU endpoints need product-supported standby placement, device capacity, and recovery routing. Existing GPU passthrough and one VM per workload are not automatic failover.
NFS, Garage, NAS, router and DNS	External failure domains need independent storage/routing capacity or an approved failover design. Kubernetes application changes cannot supply that hardware.

Until those dependencies and the node-loss transactions are verified, the repository must retain its mixed-availability classification. Failover can include retries and a recovery interval; it does not transfer in-flight connections.

Read-only assessment on 2026-10-29

This separately collected live evidence is a readiness assessment, not a completed node-loss drill:

The initial snapshot had six Ready nodes and all 43 Argo CD Applications Synced/Healthy. All replicated Deployment/StatefulSet serving groups passed the gate’s ready-replica placement check, and CNPG instance readiness and primary-transition checks passed.
All five SQLite services (Uptime Kuma, CrowdSec LAPI, Headscale, Headplane and RustDesk rendezvous) successfully restored their current Litestream backup into a separate temporary file in their existing pod. Each output was nonempty with a SQLite header. This did not check full SQLite integrity, restored application behavior, cross-node NFS access or the age of the last replicated transaction. Scratch files were removed after testing; active databases and backups were preserved.
Each of the five configured Litestream replicas uses a 60-second sync interval and an NFS CSI ReadWriteMany backup claim without PV node affinity. Uptime Kuma’s live node-loss eviction tolerations were 300 seconds; the other four SQLite processes and Talk used 60 seconds. Neither interval is a recovery-time guarantee.
Talk’s signaling welcome endpoint and all three NATS health endpoints responded successfully. This did not exercise an authenticated call, TURN allocation or signaling coordination during a failure.
Thirty-one public ingress root probes returned HTTP responses: 29 success/redirect responses and two 404 responses (Talk and Medusa API roots). Subsequent probes of Talk’s /api/v1/welcome and Medusa’s /health returned 200. Nextcloud’s /status.php reported installed, maintenance disabled and no pending database upgrade. Root responses, login redirects and health endpoints do not prove authenticated application transactions.
Reassessing the initial snapshot with the expanded recovery list required 4,872 MiB on the surviving right worker versus 4,766 MiB available after accounting for the gate’s preemptible batch policy. The left worker had only three unused pod slots before reclaiming any batch work. Additional infrastructure replacements are outside this lower bound.
During subsequent checks, Rancher and Invoice Ninja entered rollouts with incomplete ready replica counts, and the right worker reported DiskPressure. These are additional reasons to defer failure injection until a fresh gate and the application transactions pass. Node filesystem statistics subsequently showed free disk space; that alone does not establish that the pressure condition or its cause has cleared.

No node was shut down, rebooted, drained or isolated during this assessment. A separately approved one-worker-at-a-time drill must measure continuous external requests, singleton recovery with preserved data/identity, database promotion, surviving NATS coordination, and full rejoin before testing another node. Do not infer uninterrupted service or a maximum data-loss interval from these baseline results.

The drill sequence is left worker loss and recovery, then right worker loss and recovery. Start only with a fresh passing gate, sufficient recovery headroom and a confirmed hypervisor recovery path. Monitor external application-specific requests continuously before, during and after failure, recording timestamps, errors and request latency. An abrupt worker VM stop tests failure detection; a drain or rolling restart does not reproduce that failure. Hypervisor confirmation that the VM is stopped fences its writers. Require the listed singleton processes to recover on the surviving general worker with their original state and identity, verify database writers and NATS peer connectivity, and exercise authenticated application read/write and a real external Talk call. After restarting the failed VM, restore all database replicas, voters and ready serving replicas, confirm former writers rejoin safely, and rerun the gate before testing the other worker. Stop further failure injection if any service fails its contract. This worker-only drill does not establish control-plane, whole-physical-host or external-storage failure tolerance.

Quorum-worker capacity boundary

RTX is reserved for its explicitly assigned voters and existing node infrastructure. General applications, identity processes, controllers, Redis client proxies and scheduled Renovate scans must not use it as overflow capacity. Argo Redis voters retain their existing three-domain placement; their client proxies have two replicas explicitly restricted to the general workers. Authentik reserves CPU/memory for each server and worker, rather than scheduling those processes as BestEffort.

After a general-worker loss, surviving application replicas continue serving where dependencies support it. A second anti-affined replica can remain Pending; that is preferable to overloading the quorum domain and losing election majorities. Recoverable singleton functions need spare CPU, memory and pod slots on the surviving general worker. Resource requests reserve capacity; they do not create it. No VM resizing or new storage capacity is implied by these manifest changes.

The coupled AIO Talk signaling/Janus singleton uses a Recreate Deployment for replacement after node loss; the stable TURN Service follows the replacement pod. Existing calls and allocations still require client reconnection. Increasing replica count alone does not provide supported cross-Janus routing. The failed worker drill must be repeated after publication, including Galera membership and authenticated transactions; these changes are not an all-services HA certification.

Replacement capacity on general workers

Talk and SuiteCRM messenger need spare requested memory on a surviving general worker. The readiness gate checks their replacement footprint against allocatable memory after accounting for bound pods, init containers, sidecars and pod overhead; reserved RTX capacity is excluded. This is a memory reservation check for those two replacement processes, not a full scheduler or application-capacity simulation.

The October 28 worker test exposed a messenger replacement blocked by memory reservations after Talk moved. Stalwart’s serving-pod memory request is 512Mi, with its 4Gi limit retained; the collected seven-day peak working set across its replicas was 157Mi. This releases 512Mi of reserved memory per general worker without adding RTX workloads. Recheck observed peaks and recovery headroom as workloads grow.

Renovate jobs use the negative renovate-batch PriorityClass with preemptionPolicy: Never. Application pods at the default priority may preempt this batch workload when recovery needs its requested memory. Batch execution can retry or wait for restored capacity; it cannot preempt serving workloads. The memory gate counts this explicitly preemptible batch reservation as reclaimable for the two default-priority replacement processes. Existing running Jobs retain their original priority until replaced.

SuiteCRM messenger uses sixty-second NotReady/unreachable tolerations, matching Talk’s replacement timeout, so a lost worker does not impose the default five-minute eviction wait before restarting this consumer.

The Cal.com Redis HAProxy resolves changing StatefulSet addresses at runtime and exposes /healthz only while an authenticated master backend is usable. A listening TCP socket alone must not mark an unroutable Redis proxy ready. Stalwart remains on the known-working 0.16 release line; 1.x requires a separately reviewed PostgreSQL storage migration rather than an automatic image replacement.

Redis rejoin must use the current Sentinel majority, not the first responding voter or a persisted former-writer role. Nextcloud now waits for two matching voter replies and announces its replica through stable DNS. The Cal.com, Forgejo, LiteLLM, Mautic, Medusa and Invoice Ninja startup helpers likewise require two voters before restarting existing Redis data. Their disposable Sentinel voters receive deterministic identities on fresh configuration. These safeguards complement replication and do not make asynchronous Redis writes durable during every failure.

CrowdSec LAPI, agent, and AppSec use 60-second NotReady/unreachable eviction tolerations. LAPI remains a single SQLite writer; its replacement and backup restore take time. Traefik retains fail-closed stream enforcement, so an unavailable LAPI feed can deny requests until recovery. These tolerations shorten recovery and do not establish continuous HA.

Authentik PostgreSQL uses TCP keepalives (30-second idle, 10-second interval, three retries) to detect lost clients and release their session locks. A dead client holding the startup advisory lock can otherwise delay replacement application processes even when the primary database is available. See PostgreSQL TCP settings.