Troubleshooting Guide¶

This guide provides a consistent, evidence-driven approach to diagnosing and resolving failures across WhiteLotusAI. It covers application startup, database connectivity, OAuth integrations, AI orchestration, HTTP errors, and deployment infrastructure.

The objective is to identify the underlying cause, restore expected behavior safely, and document the resolution so that the issue can be prevented from recurring.

1. Scope and diagnostic principles¶

Use this guide when investigating:

• Application startup failures or unexpected service termination.
• Database connectivity, migration, or query failures.
• OAuth authorization and third-party integration errors.
• AI orchestration, tool selection, or authorization failures.
• Unexpected HTTP responses or API timeouts.
• Docker, networking, configuration, and deployment problems.

Follow these principles throughout the investigation:

• Evidence first: Capture the exact error before changing configuration.
• Reproducibility: Establish the conditions under which the failure occurs.
• Least privilege: Use the minimum access required for diagnosis.
• Controlled changes: Make one targeted change at a time whenever practical.
• Verified recovery: Confirm that the original failure is resolved and related functionality still works.

2. Standard diagnostic workflow¶

Use the following workflow for incidents that do not have an established, service-specific runbook.

flowchart TD
    A["Capture error and timestamp"] --> B["Identify affected service and environment"]
    B --> C["Inspect logs and configuration"]
    C --> D["Check dependencies and connectivity"]
    D --> E{"Root cause identified?"}
    E -- "No" --> F["Collect additional evidence"]
    F --> C
    E -- "Yes" --> G["Apply a controlled fix"]
    G --> H["Run regression checks"]
    H --> I{"Recovery verified?"}
    I -- "No" --> F
    I -- "Yes" --> J["Document resolution and prevention"]

    classDef start fill:#dbeafe,stroke:#2563eb,color:#172554
    classDef process fill:#e0f2fe,stroke:#0284c7,color:#0c4a6e
    classDef decision fill:#fef3c7,stroke:#d97706,color:#78350f
    classDef success fill:#dcfce7,stroke:#16a34a,color:#14532d

    class A,J start
    class B,C,F,G,H process
    class E,I decision
    class J success

The diagram uses explicit text and node colors to improve readability in supported Mermaid renderers. If your documentation theme overrides diagram colors, verify legibility in both light and dark modes.

Diagnostic checklist¶

• Record the exact error message and timestamp, including the timezone.
• Identify the environment, affected service, application version, and recent changes.
• Inspect application and dependency logs around the failure.
• Verify required configuration keys without printing their secret values.
• Check dependency availability, network connectivity, and service health.
• Reproduce the problem in a safe environment where possible.
• Identify the root cause before implementing a permanent fix.
• Apply the change through the appropriate review and deployment process.
• Run relevant regression tests and confirm recovery.
• Record the cause, resolution, verification evidence, and preventive actions.

3. Common issues at a glance¶

Use this table to identify the first diagnostic area to investigate. These are troubleshooting starting points, not confirmed root causes.

4. Application startup failures¶

Startup failures may originate from configuration, dependency incompatibilities, invalid startup commands, unavailable infrastructure, or resource constraints.

Investigation checklist¶

1. Read the earliest relevant error in the service logs. Later exceptions may be consequences of the original failure.
2. Confirm that required environment variables exist and use the expected formats.
3. Check the runtime and dependency versions against the project's declared requirements.
4. Verify that the required ports are available and mapped correctly.
5. Confirm that dependent services, such as the database, Redis, or RabbitMQ, are reachable when needed.
6. Inspect database migration failures and initialization errors.
7. Check container exit codes, health checks, and memory or CPU pressure.
8. Compare the failing deployment with the last known working version.

Example diagnostic commands¶

Run commands from the appropriate project or deployment directory. Replace placeholders with the actual service names and port numbers.

# List services and their current state
docker compose ps

# Inspect recent logs for a specific service
docker compose logs --tail=200 <service_name>

# Follow logs while reproducing the issue
docker compose logs -f <service_name>

# Inspect service status
systemctl status <service_name> --no-pager

# Inspect recent service logs
journalctl -u <service_name> -n 200 --no-pager

# Follow service logs
journalctl -u <service_name> -f

These commands are diagnostic examples. Use the actual service manager and deployment configuration for your environment.

5. Database failures¶

Investigate database errors in the context of the service that initiates the connection. A database can be healthy while remaining unreachable from a particular container or host.

Recommended checks¶

• Verify the configured hostname, port, database name, and credential source.
• Confirm that the database accepts connections from the application environment.
• Check migration history and schema compatibility with the running application.
• Inspect connection pool utilization and database resource pressure.
• Review database logs for authentication failures, connection limits, locks, and query errors.

6. OAuth and integration failures¶

OAuth issues commonly arise from mismatched redirect URIs, missing permissions, expired or revoked credentials, incomplete callback handling, or provider API changes.

Investigation procedure¶

1. Identify the integration and the exact stage at which the failure occurs.
2. Confirm that the configured client ID and redirect URI correspond to the intended environment.
3. Compare the requested OAuth scopes with the permissions actually granted.
4. Inspect callback and token-refresh errors without logging token values.
5. Verify whether the user has disconnected the integration or revoked access.
6. Check provider rate limits, API availability, and relevant error responses.
7. Confirm that the requested operation is permitted by the application's authorization policy.

7. AI orchestration and tool execution failures¶

AI workflow issues require tracing the request through the orchestration pipeline rather than assuming that every failure originates in the language model.

Diagnostic sequence¶

1. Input validation: Check whether the request was parsed and validated correctly.
2. Intent resolution: Determine whether the request was mapped to the intended operation.
3. Workflow routing: Inspect conditional branches, state transitions, and error-handling paths.
4. Tool registry: Confirm that the selected tool is registered and its declared arguments match the execution request.
5. Authorization: Verify the user's identity, permissions, and applicable operation restrictions.
6. Argument validation: Check required fields, types, and provider-specific constraints.
7. Provider execution: Inspect connectivity, timeouts, rate limits, and upstream responses.
8. Result handling: Confirm that provider output is validated and errors are propagated correctly.

Request-to-tool diagnostic model¶

flowchart TD
    A["User request"] --> B["Validate input"]
    B --> C["Resolve intent"]
    C --> D["Select registered tool"]
    D --> E{"Authorized operation?"}
    E -- "No" --> F["Reject safely"]
    E -- "Yes" --> G["Validate arguments"]
    G --> H["Execute provider read"]
    H --> I["Validate result"]
    I --> J["Return response"]
    F --> K["Record safe diagnostic context"]
    J --> K

    classDef input fill:#dbeafe,stroke:#2563eb,color:#172554
    classDef processing fill:#e0f2fe,stroke:#0284c7,color:#0c4a6e
    classDef decision fill:#fef3c7,stroke:#d97706,color:#78350f
    classDef rejected fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
    classDef output fill:#dcfce7,stroke:#16a34a,color:#14532d

    class A,B input
    class C,D,G,H,I processing
    class E decision
    class F rejected
    class J,K output

The diagram represents the intended diagnostic model, not proof that every runtime path currently implements these exact stages.

Evidence to collect¶

Capture the request correlation ID, workflow stage, selected tool identifier, validation outcome, authorization result, execution duration, and sanitized provider error where available.

Do not record raw access tokens, sensitive document contents, private email bodies, or other confidential payloads merely to make debugging easier.

8. HTTP errors and API behavior¶

Troubleshoot HTTP errors by identifying the specific route and its actual response contract. Do not assume that all services return the same error schema.

These are general HTTP troubleshooting guidelines. The observed status code and response body must be confirmed against the actual implementation.

Reproduce a request safely¶

# Inspect an endpoint response without printing credentials
curl -sS -D - -o /dev/null \
  "http://localhost:<port>/<health_or_test_path>"

Replace the path with a known, appropriate endpoint in your environment. A 404 from / does not, by itself, prove that the service is unavailable; the application may not define a root route.

Avoid placing tokens directly in shell commands, where they may be captured in shell history or process information.

9. Docker and deployment failures¶

Inspect the complete path between the client, gateway, application, and dependencies before concluding that a deployment is unavailable.

Deployment checklist¶

• Confirm the intended image tag or application revision.
• Inspect container state, restart count, and recent logs.
• Verify environment-variable injection and required configuration.
• Check exposed ports, host mappings, and container networking.
• Confirm that health checks target valid application endpoints.
• Verify dependency availability and startup ordering.
• Inspect CPU, memory, disk, and connection limits.
• Confirm reverse-proxy and TLS configuration where applicable.
• Compare the deployed configuration with the expected environment.
• Re-run the relevant health and regression checks after recovery.

Docker commands¶

# Check running and stopped containers
docker ps -a

# Inspect a container's recent output
docker logs --tail=200 <container_name>

# Inspect exit state and health information
docker inspect <container_name>

# Check resource consumption
docker stats --no-stream

Use these commands with the appropriate permissions. Container inspection can expose environment variables and other sensitive metadata; review output before sharing it.

10. Incident escalation¶

Escalate promptly through the organization's incident-response process when any of the following occurs:

• Suspected exposure of customer data, credentials, or confidential documents.
• An unauthorized operation or unexpected write operation.
• Suspected credential compromise or unexplained token activity.
• Persistent production unavailability or widespread service degradation.
• Repeated failures of authentication, authorization, or other security controls.
• Evidence of data corruption or unexplained changes to stored records.

For potential security incidents, prioritize containment and preservation of relevant evidence over routine debugging. Follow the established incident-response and credential-rotation procedures; do not independently delete logs or make destructive changes.

Incident handoff checklist¶

Include the following information where available:

11. Resolution and preventive follow-up¶

An incident should not be considered resolved solely because an error disappears temporarily.

Before closing an issue:

1. Confirm that the original failure can no longer be reproduced under the same conditions.
2. Run relevant regression tests and verify affected service interactions.
3. Confirm that authentication, authorization, and read-only restrictions remain enforced.
4. Record the root cause, corrective action, and verification evidence.
5. Add a regression test or monitoring improvement where appropriate.
6. Update this guide when the investigation reveals a recurring failure mode or a change to the supported architecture.

12. Related documentation¶

Consult the relevant project documentation for additional context:

• Architecture Overview — system components and service boundaries.
• System Design — service responsibilities and communication patterns.
• Threat Model — security risks and required controls.
• Read-Only Integration Test Plan — integration verification scenarios.
• Testing Strategy — quality controls and verification expectations.
• Documentation Audit and Governance — documentation accuracy and verification requirements.

Verify each relative link against the actual documentation tree before merging.

Maintenance note: Update this guide when verified incidents, architecture changes, or regression findings materially change the recommended diagnostic procedure. Documentation alone does not establish that a control or recovery procedure has been implemented or tested.