Incident Response¶

Status: Operational baseline. This document defines the recommended incident-handling process. Assign incident owners, verify escalation channels, and approve severity thresholds before relying on it during a production incident.

1. Purpose and objectives¶

This runbook establishes a consistent process for identifying, assessing, containing, investigating, and resolving incidents affecting WhiteLotusAI.

The objectives are to:

• Protect users, credentials, enterprise documents, and connected-provider data.
• Limit service disruption and prevent further unauthorized access.
• Preserve reliable evidence for investigation and applicable compliance obligations.
• Restore service only after relevant security and operational controls are verified.
• Communicate impact and recovery status through approved channels.
• Prevent recurrence through documented corrective actions and regression testing.

2. Scope¶

This procedure applies to incidents involving:

• Frontend, REST API gateway, authentication, and authorization.
• Chat orchestration, AI model execution, tool dispatch, and agent workflows.
• Document ingestion, retrieval-augmented generation (RAG), and vector storage.
• User and session processing, databases, caches, message queues, and service communication.
• OAuth credentials, integration tokens, and connected enterprise providers.
• Deployment configuration, infrastructure, dependencies, and CI/CD pipelines.
• Confidentiality, integrity, availability, or privacy of enterprise data.

3. Severity classification¶

Assign severity according to the actual impact, scope, exposure, and urgency. When evidence is incomplete, use the higher plausible severity until the risk is understood.

Severity should be reassessed whenever new evidence changes the estimated impact.

4. Roles and responsibilities¶

Assign named people and verified contact channels in the organization's operational directory. One person may perform multiple roles in a small team, provided that responsibilities remain explicit.

For a critical incident, designate an incident commander and a technical responder as early as practical. Involve security and privacy stakeholders when sensitive data, credentials, or authorization controls may be affected.

5. Response workflow¶

The response follows a controlled sequence from detection through learning. Containment and investigation may proceed in parallel when doing so is safe and evidence is preserved.

flowchart TD
    A["Detect and record incident"] --> B["Assess severity and scope"]
    B --> C["Assign incident owner"]
    C --> D["Contain immediate risk"]
    D --> E["Preserve evidence and investigate"]
    E --> F{"Root cause understood and risk controlled?"}
    F -- "No" --> D
    F -- "Yes" --> G["Apply reviewed fix and recover"]
    G --> H{"Security and service checks pass?"}
    H -- "No" --> D
    H -- "Yes" --> I["Communicate resolution"]
    I --> J["Review incident and track actions"]

    classDef process fill:#64748b,stroke:#475569,color:#ffffff
    classDef decision fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef response fill:#d1fae5,stroke:#059669,color:#064e3b

    class A,C,D,E,G,I,J process
    class B,F,H decision

6. Incident response procedure¶

6.1 Detect and record¶

Create an incident record in the approved tracking system as soon as an incident is suspected.

Capture:

• Incident identifier and detection timestamp, including timezone.
• Reporter, assigned owner, and current severity.
• Affected services, environments, integrations, and approximate start time.
• Observable symptoms, error codes, request IDs, and relevant deployment changes.
• Known or suspected impact on users, documents, credentials, and provider data.
• Actions already taken and the outcome of each action.

Use verified observations and clearly distinguish confirmed facts from hypotheses.

6.2 Triage and assess impact¶

Determine what is affected, what remains operational, and whether the incident could extend to other users or services.

Check the following areas where relevant:

• Authentication, session validation, and role-based authorization.
• Tenant and user boundaries for document retrieval and tool results.
• AI planner decisions, tool selection, and authorization enforcement.
• OAuth grants, token storage, credential exposure, and provider access.
• Database integrity, queue health, cache behavior, and service availability.
• Recent releases, configuration changes, dependency updates, and infrastructure events.
• Data confidentiality, integrity, availability, and potential reporting obligations.

Reassess severity when the impact becomes clearer. If a security boundary may have been crossed, do not wait for complete root-cause analysis before containing the risk.

6.3 Contain the incident¶

Choose the narrowest containment action that reliably prevents further harm. Escalate to broader isolation when the affected path cannot be safely restricted.

Document each action, its authorization, its timestamp, and its observed result.

6.4 Investigate and preserve evidence¶

Build a timeline using trustworthy, access-controlled evidence. Correlate application logs, infrastructure telemetry, identity events, provider audit records, and deployment history where available.

Investigation should answer:

1. What happened, and when did it begin?
2. Which users, services, integrations, or data categories were affected?
3. What vulnerability, failure, configuration error, or unsafe behavior enabled the incident?
4. Was the issue exploited, and is there evidence of continued access?
5. Which preventive or detective controls failed or were missing?
6. What evidence supports the findings, and what uncertainty remains?

Preserve original records where possible. Record the source, collection time, collector, relevant request or event identifiers, and any transformations applied to investigative copies.

Restrict access to evidence according to its sensitivity. Follow approved retention, legal-hold, and privacy requirements; do not assume that retaining all application content indefinitely is appropriate.

6.5 Recover safely¶

Recovery requires more than a successful health check. Validate the security controls and the user-facing workflows affected by the incident.

• The root cause has been addressed or the remaining risk is explicitly accepted by an authorized owner.
• The remediation has been reviewed and tested.
• Credentials have been rotated or revoked where necessary.
• Authentication, authorization, and tenant-isolation checks pass.
• Prohibited tool operations remain blocked by server-side controls.
• Affected data and persistent state have been checked for integrity.
• Critical dependencies and essential application workflows operate correctly.
• Monitoring and alerting are adequate to detect recurrence, where implemented.
• The rollback or renewed-containment procedure is available.

Restore service gradually when practical. Monitor the affected path for recurrence and retain the ability to disable it again.

Do not re-enable a connector or tool solely because the original error has disappeared.

6.6 Communicate¶

Communications must be accurate, proportionate, and approved through the appropriate organizational process.

Provide an initial notification when required by organizational policy. Share updates when impact, containment, or recovery status materially changes.

Do not speculate about the cause, promise an unverified recovery time, or disclose investigative details that could create additional risk.

6.7 Close and review¶

An incident may be closed when:

• The immediate threat is contained.
• Service is recovered or a documented, approved alternative is in place.
• Required security and functional checks have passed.
• Stakeholder communications and applicable reporting obligations have been addressed.
• Evidence has been preserved and access appropriately restricted.
• Corrective actions have owners, due dates, and acceptance criteria.

Prepare a blameless post-incident review that explains system conditions and control failures rather than assigning personal blame. Track corrective work until it is completed and verified.

7. AI and integration-specific incidents¶

WhiteLotusAI combines AI-generated decisions with authenticated enterprise data and connected-provider capabilities. Treat failures at the tool authorization boundary as security-relevant, even when the model's response appears normal.

AI-specific investigation checklist¶

• Identify the user, session, request, and tool invocation involved.
• Establish which server-side identity and authorization checks were applied.
• Verify that the requested action was allowed independently of model-generated intent or confirmation claims.
• Determine whether retrieved content contained untrusted instructions or sensitive data.
• Inspect tool arguments and results using access-controlled evidence handling.
• Check whether retries, queued work, cached responses, or parallel requests expanded the impact.
• Add a regression test for the exact failure and relevant neighboring authorization cases.

Security principle: The model may propose an action, but the server-side authorization layer must decide whether that action is permitted.

8. Incident record template¶

Use this template when creating an incident ticket. Fill in only information that is known; mark unresolved fields as pending rather than guessing.

Incident ID:
Title:
Status:
Severity:
Detected at (timestamp and timezone):
Incident owner:
Technical responder:
Security contact:

Affected environment and services:
Affected integrations:
Potentially affected users or data:
Confirmed facts:
Current hypotheses:
Detection source:
Relevant request/event IDs:

Containment actions and timestamps:
Evidence locations and access restrictions:
Investigation findings:
Root cause:
Recovery and validation results:

Communications and approvals:
Outstanding risks:
Corrective actions and owners:
Target completion dates:
Closure decision and approver:
Post-incident review reference:

9. Post-incident action tracking¶

Track each corrective action in the approved work-management system.

Prioritize systemic improvements, such as stronger authorization boundaries, better alerting, safer credential handling, reproducible recovery procedures, and automated regression tests.

10. Readiness and maintenance¶

Before treating this runbook as operationally ready, verify the following:

• Incident commander and security escalation contacts are assigned.
• Severity thresholds and response-time targets are approved.
• Incident tracking and restricted evidence storage are available.
• Credential revocation and connector-disable procedures are documented.
• Service isolation, rollback, and recovery procedures are tested.
• Relevant security and privacy notification requirements are documented.
• Incident responders know where to find this runbook.
• A tabletop exercise or controlled incident simulation has tested the process.

Review this document after significant architecture changes, security incidents, changes to integration permissions, or changes to applicable organizational requirements.

Related operational documentation: Link this section to the approved monitoring, backup and recovery, deployment, security, and troubleshooting pages once their canonical documentation paths and cross-references have been verified.