Invoice Ninja · Operate

Invoice Ninja is owned by apps/invoiceninja/. Shared hostnames, image versions, and storage sizes live in apps/variables.yaml; infrastructure credentials are sealed in the application’s manifest tree. Configure company email in Invoice Ninja’s UI. Follow the application deployment workflow to publish changes to the branch tracked by Argo CD. Local edits are not deployment inputs.

Deployment and dependencies

The local Helm chart deploys two PHP application replicas and two nginx replicas. The supporting manifests provide two queue workers, a minute-by-minute scheduler, two MariaDB Galera data members with a separate arbitrator, and Redis with Sentinel. Web, nginx, and queue worker replicas are restricted to the two general workers, leaving the small RTX worker for quorum services. Their zero-surge rollouts do not need a third worker. Application files use the configured Garage S3 bucket. Local application volumes are ephemeral; they are not backups. Replication does not establish a tested recovery procedure or prove failure tolerance for the full service.

The Application explicitly supplies both the Ingress hostname and its / Prefix path. Helm parameters that set an array element can replace chart-default array content; supplying only ingress.hosts[0].host can render an invalid Ingress with no paths and block subsequent sync waves.

Workers and the scheduler invoke /usr/local/bin/init.sh before runuser and their Artisan command. This preserves the image’s architecture-specific Chrome path for SnapPDF. These commands do not request the supervisor startup branch, so they do not run the web container’s migration and initialization sequence.

Verify a deployment

From the repository root:

/home/linuxbrew/.linuxbrew/bin/kubectl --kubeconfig kubeconfig \
  -n argocd get application invoiceninja
/home/linuxbrew/.linuxbrew/bin/kubectl --kubeconfig kubeconfig \
  -n invoiceninja get pods,deployments,statefulsets,cronjobs,jobs,pvc,certificate

Check Argo’s operation result and resource drift separately from pod readiness. The public /health endpoint returns a fixed API response; it does not test the database, mail delivery, PDFs, or object storage.

Verify authenticated login and API reads, pending migrations, Galera primary component membership and synchronization, Redis queue depth and failed jobs, recent scheduler completion, and actual worker job completion. Test PDF rendering in both web and worker processes using the application’s configured PDF engine. An explicitly overridden Chrome path can hide a broken worker configuration.

Test S3 writes and reads with a unique diagnostic object and remove only that object afterward. Fetch the frontend assets as well as the landing page. A full acceptance test also needs an authorized disposable invoice, client portal access, PDF download, and any configured payment gateway’s sandbox workflow.

Authentik OIDC

The shared environment supplies OIDC_ISSUER, OIDC_WELL_KNOWN, OIDC_CLIENT_ID, OIDC_SCOPES (openid profile email), and OIDC_PROVIDER_LABEL (antID, displayed as Sign in with antID). The issuer, client ID, provider label, and JIT target company are owned by apps/variables.yaml and passed through the Application to the local chart. OIDC_CLIENT_SECRET is sealed in invoiceninja-app alongside the existing application credentials. Web replicas, queue workers, and scheduler Jobs consume the same ConfigMap and Secret.

The issuer is https://auth.antblu.net/application/o/invoice-ninja/. Discovery supplies the authorization, token, userinfo, logout, and JWKS endpoints; these do not need separate environment variables. OIDC_WELL_KNOWN also supplies the discovery URL for releases using the older configuration name. Register https://invoice.antblu.net/auth/oidc as the exact redirect URI in Authentik; Invoice Ninja derives it from APP_URL. See the upstream OIDC configuration.

The checked-in image tag is 5.13.50, which includes generic OIDC support. Upstream only signs in existing users. The local chart installs the antID JIT extension from chart/files/oidc/ before the normal image initialization and Laravel configuration cache. Its installer checks the upstream callback shape and refuses to start on incompatible changes; review it when upgrading the image. The extension is included in the web pod’s rollout checksum. The nginx asset-seeding init container also fills missing English OIDC strings in this UI build, including Sign in with {{provider}}. Its extension checksum triggers a rollout when that fix changes. Hard-refresh the login page after the update if the browser retains the older JavaScript asset.

INVOICENINJA_OIDC_JIT_COMPANY_ID selects the existing company by its internal numeric database ID (currently 3). A blank value disables JIT. New users must have a valid email and a literal boolean email_verified: true assertion from antID. Provisioning serializes on the target company inside a database transaction; repeat callbacks reuse the same subject. New members receive no owner/admin flags or additional permissions. Administrators can grant the required permissions under the company’s user settings after first login. Deleted users, existing conflicting email identities, and different OIDC subjects cannot be restored or rebound by JIT. Existing unlinked accounts can still use upstream’s verified-email linking. Authentik’s application access policy controls who may reach this flow; there is no local email-domain allowlist. No new company or account is created, and no invitation email is sent.

Run chart/tests/oidc-jit.php inside the matching application image with the extension helper path and company ID as arguments to exercise first/repeat provisioning, permissions, and denial cases. It rolls back its fixture writes. This test does not replace a real signed Authentik browser login.

If the callback and code exchange succeed but the UI reports a sign-in failure, check /api/v1/refresh_react in the browser’s network log. A 403 with Traefik OriginStatus: 0 can be CrowdSec’s Langflow rule crowdsecurity/vpatch-CVE-2025-34291, which matches the /api/v1/refresh substring when the request has an HTTPS Origin header. The owning CrowdSec Application carries a rule-specific exception for POST to this exact path on the Invoice Ninja hostname; see CrowdSec operations.

The web Deployment includes a ConfigMap checksum in its pod template so chart configuration changes trigger a rollout and rebuild Laravel’s cached settings. Secret changes require a rolling restart of the web Deployment; restart the queue workers when their shared environment changes as well. New scheduler Jobs inherit the current ConfigMap and Secret automatically.

After reconciliation, verify /api/v1/oidc/config reports oidc_enabled: true and /auth/oidc redirects to Authentik with the expected client ID, callback, and scopes. Complete a browser login with a new permitted antID identity and repeat it to verify provisioning, callback, and session; readiness and a successful redirect alone do not establish authenticated login.

Mail and payment setup

Configure the company’s SMTP provider, host, port, encryption, credentials, and sender under Settings → Email Settings in Invoice Ninja. Select the SMTP provider instead of Default to use the company’s configuration. The repository does not supply MAIL_* environment variables or a sealed SMTP password. See the official Invoice Ninja configuration reference.

When removing an existing environment-based mail configuration, publish and reconcile the changes, then roll the web and worker Deployments so their processes and cached configuration no longer retain the old settings. Subsequent scheduler Jobs inherit the updated configuration. Existing company settings remain in the database; this source change does not populate or clear them.

SMTP connection and authentication tests can run without sending a message. Delivery requires a separately authorized test recipient and evidence of receipt, not merely acceptance into a queue. If Stalwart reports Unsupported credentials type for OIDC backend, its selected authentication backend cannot handle the submitted password. Use a supported service-account authentication arrangement; do not change shared mail authentication merely to clear an Invoice Ninja check. See Stalwart’s OIDC documentation.

Configure the intended payment gateway and use its sandbox before testing payment collection. An empty gateway list is not evidence of working payments.

Recovery

Protect the MariaDB database, S3 bucket, and existing application encryption key together. Preserve integration credentials. Establish and test independent backups before relying on the service for billing; do not treat two database replicas as a backup. Restoring requires compatible application code and database migrations.

Diagnostic snapshot: 22 September 2026

Live checks found working HTTPS and frontend assets, authenticated API reads, synchronized Galera membership, no failed jobs or queue backlog, completing scheduler jobs, and successful S3 diagnostic write/read/cleanup. The account had no clients, invoices, or payment gateways. No MariaDB operator Backup resources were present in the namespace; external backups were not audited.

Argo initially reported an invalid Ingress with no paths. Worker PDF rendering failed because the Chrome path was absent. Both corrections reached main in commit 611eb77 through a concurrent update, then reconciled successfully after an Argo refresh. The worker rollout completed, the scheduler completed with the corrected startup command, and Argo reported Synced and Healthy. A graceful php artisan queue:restart allowed old workers to exit during the rollout. Both replacement workers rendered a valid diagnostic PDF using the Chrome path from their running worker process environment. SMTP authentication still failed with Stalwart’s OIDC backend error. No email was sent and no payment was attempted. Browser interaction, invoice lifecycle, client portal transactions, failover, and restore remain outside this diagnostic snapshot.

Arbitrator address refresh and health

The RTX arbitrator re-resolves both data-member names. If their addresses change, it reconnects only after both data nodes report connected, Synced Primary components with matching state UUIDs and at least two members. This health query uses the existing application account through a Secret reference and performs no writes. It does not force-bootstrap a component, change voting weights, or restart merely because a member disappears. Readiness requires the latest logged view to be Primary, not just a running PID. Confirm three members before maintenance, then check surviving SQL writes and two-member Primary membership during a single data-worker loss.