Skip to content

Monitoring and Observability

Status: Operational baseline. Verify all monitoring capabilities against the deployed application before treating them as implemented.

Monitoring and observability help operators detect service failures, investigate incidents, identify capacity constraints, and maintain reliable AI-assisted workflows without unnecessarily exposing sensitive business information.

This guide defines the recommended monitoring approach for WhiteLotusAI across its application services, AI orchestration, integrations, background workers, and infrastructure.

Implementation status

This document describes monitoring requirements and recommended practices. A documented metric, health check, dashboard, or alert is not evidence that the corresponding capability has been implemented or verified.

1. Objectives

Monitoring should provide actionable information about the availability, reliability, security, and performance of the platform.

  • Service health

    Detect unavailable services, failed dependencies, and unhealthy workers.

  • Performance

    Track request latency, error rates, queue delays, and resource utilization.

  • Security

    Identify authentication failures, authorization denials, and suspicious tool-dispatch activity.

  • Operational visibility

    Investigate incidents using correlated logs and bounded diagnostic information.

2. Monitoring scope

Monitor each component according to its operational responsibilities. The following inventory is a recommended starting point, not confirmation that all components currently expose telemetry.

Component or layer Signals to monitor Operational purpose
REST API gateway Request volume, latency, HTTP status codes, rate-limit events Detect API failures and access pressure
Chat and AI orchestration Workflow failures, execution duration, model errors, tool-dispatch outcomes Diagnose degraded AI responses
Session and user services Request or worker failures, processing latency, dependency errors Detect identity and session-management problems
RAG service Retrieval failures, search latency, embedding errors, cache behavior Diagnose document retrieval quality and availability
Integrations Provider latency, API errors, OAuth refresh failures Identify third-party service disruptions
Notification service Delivery failures, processing latency, queue backlog where applicable Detect delayed or failed notifications
PostgreSQL / pgvector Connection failures, query latency, connection saturation Identify persistence and retrieval bottlenecks
Redis Connection failures, latency, memory pressure, cache errors Diagnose caching and rate-limiting problems
RabbitMQ Queue depth, consumer availability, unacknowledged messages Identify worker backlogs and stalled processing
Host or container CPU, memory, disk, restarts, resource limits Detect infrastructure pressure

Verify the actual execution model

Confirm whether each service runs as an HTTP server, a message-queue worker, or both. Do not configure HTTP probes or request metrics for a component that does not expose an HTTP interface. Worker health should be measured through process state, message consumption, and processing progress where supported.

Use metrics for trends and alerting, structured logs for diagnostics, and traces or correlation identifiers to connect related operations.

Signal Recommended measurements Useful dimensions
Request traffic Requests per second or minute Service, route template, method
Request latency p50, p95, p99 Service, route template
Error rate Failed requests and normalized error categories Service, status class
Provider health Request duration, timeout count, provider errors Provider, operation category
OAuth reliability Refresh failures, expired credentials, reauthorization requirements Provider, error category
AI workflow health Execution duration, node failures, model timeouts Workflow stage, error category
Tool authorization Allowed and denied operations Tool category, policy reason
Queue health Queue depth, oldest-message age, consumer count Queue, worker
Resource utilization CPU, memory, disk, open connections Host, container, dependency
Retrieval performance Search duration, empty-result rate, retrieval errors Retrieval stage, outcome

Treat these signals as planned until their collection and export paths have been verified.

Metric design principles

  • Prefer bounded labels such as service, route template, provider, and normalized error category.
  • Avoid raw URLs containing identifiers, user IDs, email addresses, document names, prompts, or message contents as metric labels.
  • Avoid unbounded dimensions such as arbitrary exception text, query strings, and user-supplied tool names.
  • Define units, aggregation windows, ownership, and expected behavior for every operational metric.
  • Record the monitoring backend and collection method in the deployment inventory.

4. Logging and correlation

Application logs should be structured, consistent, searchable, and safe to retain.

Required logging practices

  • Include a timestamp, severity, service name, and event name.
  • Propagate a request or correlation ID across service boundaries where supported.
  • Record normalized error codes and concise diagnostic context.
  • Redact authorization headers, cookies, access tokens, refresh tokens, passwords, API keys, and other secrets.
  • Avoid logging full prompts, email bodies, chat messages, documents, attachments, or unrestricted provider responses by default.
  • Restrict log access according to operational responsibilities.
  • Define retention periods and deletion procedures according to applicable organizational requirements.
  • Test redaction behavior automatically and review logs during controlled failure tests.

Sensitive data must not enter diagnostic logs

Redaction should occur before sensitive values reach persistent log storage. Do not rely solely on dashboard masking or access controls to protect secrets that should never have been recorded.

Example structured log

The following JSON illustrates a recommended event format. It is an example schema, not a claim about the application's current logger.

{
  "timestamp": "2026-10-16T10:15:30Z",
  "level": "ERROR",
  "service": "integrations",
  "event": "provider_request_failed",
  "correlation_id": "example-correlation-id",
  "provider": "example-provider",
  "error_code": "PROVIDER_TIMEOUT",
  "duration_ms": 3000
}

Use generated or sanitized correlation identifiers. Do not put access tokens, user content, or credentials into the event.

Correlation across services

flowchart TD
    A["Client request"] --> B["API gateway"]
    B --> C["Chat orchestration"]
    C --> D["Integration or RAG service"]
    D --> E["External provider or data store"]

    A -. "Correlation ID" .-> F["Structured logs"]
    B -. "Service events" .-> F
    C -. "Workflow events" .-> F
    D -. "Dependency events" .-> F
    E -. "Sanitized outcome" .-> F

    F --> G["Search and incident diagnosis"]

    classDef app fill:#e8eaf6,stroke:#5c6bc0,color:#1a237e
    classDef data fill:#e0f2f1,stroke:#00897b,color:#004d40
    classDef ops fill:#fff3e0,stroke:#ef6c00,color:#7f3b00

    class A,B,C app
    class D,E data
    class F,G ops

The diagram represents a recommended logical flow. Actual correlation propagation depends on the service implementations and instrumentation.

5. Health checks and dependency monitoring

Health checks should distinguish process availability from the ability to accept useful work.

Check Purpose Recommended behavior
Liveness Determines whether a process is running correctly Avoid failing solely because an external provider is unavailable
Readiness Determines whether a service can accept work Check required local dependencies using bounded timeouts
Dependency health Measures the availability of databases, caches, brokers, or providers Report dependency-specific failures without confusing them with process failure
Worker progress Detects stalled message processing Monitor consumer state, processing progress, and backlog where applicable

Avoid cascading restarts

An external provider outage should not automatically make every application process appear dead. Separate application liveness from dependency readiness, and use explicit recovery policies for workers and dependencies.

Document the following for each implemented health check:

  • Endpoint, command, or monitoring mechanism.
  • Expected response and timeout.
  • Dependencies checked.
  • Startup grace period and failure thresholds.
  • Whether the check is used by a load balancer, container orchestrator, or alerting system.
  • Recovery action when the check fails.

Do not assume that a root URL returning HTTP 200 is a meaningful health check. Verify what the endpoint actually tests.

6. Alerting and incident response

Every production alert should identify a condition that requires action. Prefer symptoms affecting users over isolated low-level signals, while retaining infrastructure alerts for risks that need early intervention.

Alert category Example trigger to define Initial investigation
Service availability Sustained failed health checks Inspect process state, recent deployments, and service logs
API reliability Error rate exceeds an agreed threshold Group errors by service, route, and normalized code
High latency p95 latency remains above the agreed target Identify slow dependencies, queries, or workflow stages
Provider outage Repeated provider timeouts or elevated errors Check provider status, credentials, rate limits, and network access
OAuth failures Repeated refresh failures or authorization loss Inspect sanitized error categories and connection status
Worker backlog Queue age or depth exceeds an agreed limit Check consumers, retries, dead-letter queues, and dependency health
Resource exhaustion CPU, memory, disk, or connection usage approaches a defined limit Inspect resource trends and workload changes
Security anomaly Unusual authorization denials or tool-policy violations Verify event context and investigate potential misuse

The thresholds must be established from measured workload, service objectives, and recovery requirements. The examples above do not define universal production thresholds.

Alert ownership requirements

Every actionable alert must document:

  1. Severity — impact and urgency.
  2. Trigger — exact metric, condition, evaluation window, and threshold.
  3. Impact — affected service or user-facing capability.
  4. Owner — responsible team or operator.
  5. Diagnostics — first checks and relevant log queries.
  6. Escalation — next contact if the issue persists.
  7. Recovery — safe remediation and rollback steps.
  8. Runbook — link to the applicable incident procedure.

Do not enable an alert until its trigger has been tested and an owner can act on it.

7. Operational workflows

The following workflows describe recommended operator actions, not automated capabilities guaranteed by the application.

  1. Confirm the alert and identify the affected service.
  2. Inspect process state, restart history, and recent deployments.
  3. Check required local dependencies and resource pressure.
  4. Review correlated logs for the first failure.
  5. Restore service using the approved recovery procedure.
  6. Verify recovery and document the incident.
  1. Identify the provider and affected operation category.
  2. Check provider status, request timeouts, rate limits, and network connectivity.
  3. Review sanitized OAuth and provider error events.
  4. Determine whether retrying is safe and respects provider limits.
  5. Restore access or apply the documented fallback.
  6. Verify the affected integration without exposing user content.
  1. Identify whether the failure occurred during orchestration, model execution, tool dispatch, embedding, or retrieval.
  2. Inspect workflow-stage latency and normalized errors.
  3. Check model availability, retrieval dependencies, and queue health where applicable.
  4. Confirm that tool authorization and read-only restrictions remain enforced.
  5. Apply the approved recovery or rollback procedure.
  6. Run a controlled functional test and record the outcome.

8. Verification and operational readiness

Before declaring monitoring production-ready, verify the complete path from event generation to operator action.

  • Inventory all deployed services and background workers.
  • Record which metrics, logs, traces, and health checks are implemented.
  • Verify that telemetry reaches the intended backend.
  • Confirm correlation IDs propagate across supported service boundaries.
  • Test redaction for credentials, prompts, messages, and document content.
  • Validate alert thresholds against observed workload.
  • Run controlled tests for service failure, provider timeout, worker backlog, and resource pressure where applicable.
  • Confirm that alerts reach the correct owner.
  • Exercise the linked incident and rollback procedures.
  • Document monitoring access controls and retention settings.
  • Record unresolved gaps and assign owners.

Monitoring verification record

Maintain a record for each critical signal or alert.

Field Record
Signal or alert Metric, log event, health check, or alert name
Implementation status Implemented, partially implemented, planned, or not verified
Source Service, exporter, agent, or monitoring configuration
Destination Dashboard, log platform, or alerting system
Threshold and window Defined condition, if applicable
Owner Responsible operator or team
Last verified Date and test evidence
Runbook Link to recovery instructions

Completion criterion

Monitoring is operationally ready when critical signals are collected, sensitive data is protected, actionable alerts have owners, and controlled tests demonstrate that failures can be detected and investigated.

Link these entries to the corresponding pages in the documentation navigation when available.

Operational principle: collect enough evidence to detect and resolve failures, but never collect more sensitive information than the investigation requires.