Skip to content

Operations: Endpoint Healthchecks

Endpoint healthchecks probe storage endpoints and persist status, latency, and incident history for Admin, Manager, Portal, and Ceph Admin status views.

Availability model

  • The effective feature switch is endpoint_status_enabled in app settings.
  • FEATURE_ENDPOINT_STATUS_ENABLED can force that switch on or off from the backend environment.
  • Each storage endpoint can select a healthcheck mode and optional probe URL from Admin Storage Backends.

Manual trigger

curl -X POST "http://localhost:8000/api/internal/healthchecks/run" \
  -H "X-Internal-Token: <INTERNAL_CRON_TOKEN>"

The endpoint returns skipped when another backend replica already holds the healthcheck operation lease.

Scheduler integration

  • Compose scheduler calls the internal endpoint periodically.
  • QuickStart starts the scheduler automatically; its default interval is five minutes.
  • Helm supports healthcheckCronJob values.

The scheduler or CronJob must use the same INTERNAL_CRON_TOKEN as the backend. Keep the internal route on a trusted network.

Initial checks

Saving Admin settings with Endpoint Status changing from disabled to enabled requests a check of all endpoints. Creating an endpoint through Admin requests a check of that endpoint only, provided the feature remains enabled.

These checks run after the HTTP response with a separate database session and share the operation lease used by scheduled and manual checks. If another check is running, the initial check is deferred to the next scheduled run. Failures are logged without undoing successful configuration changes; the scheduler retries at its next interval. Environment feature locks continue to apply.

Administrator notifications

Active ui_admin and ui_superadmin users receive an in-app notification for the first abnormal result and for every later status transition to degraded or down. An unchanged state does not generate another notification. A transition from either abnormal state back to up generates an informational recovery notification.

degraded maps to warning severity and down maps to error severity. The notification payload contains the endpoint identifier and name, check mode, latency, status and a sanitized error summary when one is available. Health telemetry does not create application audit events.

Per-endpoint mode

Admin Storage Backends exposes:

  • HTTP probe: checks the endpoint or a configured healthcheck URL.
  • S3 signed probe: signs a lightweight S3 request with Supervision Ops credentials. On a Ceph endpoint in external service-identity mode, supplying a complete Supervision pair automatically enables this mode. Without Supervision, use the HTTP probe instead.
  • Optional healthcheck URL override. Empty value uses the endpoint URL.

Relevant backend settings

  • HEALTHCHECK_TIMEOUT_SECONDS
  • HEALTHCHECK_INTERVAL_SECONDS
  • HEALTHCHECK_RETENTION_DAYS
  • HEALTHCHECK_DEGRADED_LATENCY_MS
  • HEALTHCHECK_VERIFY_SSL
  • HEALTHCHECK_LATENCY_BASELINE_WINDOW_DAYS
  • HEALTHCHECK_BASELINE_SAMPLE_SIZE
  • HEALTHCHECK_RELATIVE_DEGRADED_RATIO
  • HEALTHCHECK_RELATIVE_DEGRADED_MIN_DELTA_MS
  • HEALTHCHECK_INCIDENT_RECENT_MINUTES

UI dependency

Endpoint Status pages and dashboard health widgets require endpoint_status_enabled in app settings. When the feature is disabled, the authenticated internal trigger returns HTTP 200 with status: skipped and reason: feature_disabled, without probing endpoints or writing healthcheck rows. The manual Admin trigger remains unavailable. Authentication failures and actual execution errors are not converted into successful skipped runs.