Test Cases and Regression Matrix¶

This document defines the core regression scenarios used to evaluate the correctness, security, reliability, and operational behavior of WhiteLotusAI.

The catalogue provides a consistent reference for designing automated tests, conducting manual verification, investigating defects, and assessing release readiness.

1. Objectives¶

The regression matrix aims to:

• Validate authentication, authorization, and user-data isolation.
• Verify that AI planning and tool dispatch reject malformed or unauthorized requests.
• Enforce the read-only integration policy across supported providers.
• Test resilience against provider failures, timeouts, and rate limits.
• Detect prompt injection, unsafe tool execution, and sensitive-data leakage.
• Validate API contracts, configuration handling, and documentation build requirements.
• Establish evidence-based release criteria.

2. Test case lifecycle¶

Each test case should progress through a defined lifecycle. Its status must reflect the available implementation and verification evidence rather than the intended behavior alone.

flowchart TD
    A["Define test scenario"] --> B["Identify requirements and risks"]
    B --> C["Implement automated test or manual procedure"]
    C --> D["Execute in a recorded environment"]
    D --> E{"Expected result achieved?"}
    E -- "Yes" --> F["Review and retain evidence"]
    F --> G["Mark verified"]
    E -- "No" --> H["Record defect and impact"]
    H --> I["Fix and rerun"]
    I --> D

    classDef work fill:#DCEBFA,stroke:#3975B7,color:#17365D,stroke-width:1.3px
    classDef decision fill:#FCE8CF,stroke:#B7791F,color:#704214,stroke-width:1.3px
    classDef done fill:#D9F2E6,stroke:#23835B,color:#174B37,stroke-width:1.3px
    classDef defect fill:#F8DEDE,stroke:#B94A48,color:#762D2D,stroke-width:1.3px

    class A,B,C,D,F,H,I work
    class E decision
    class G done

    linkStyle default stroke:#7B8794,stroke-width:1.3px

2.1 Required test case fields¶

3. Priority definitions¶

Priority describes risk and release impact, not execution order. A P0 test should receive appropriate attention even if it is difficult to automate.

4. Core regression matrix¶

The following scenarios are the baseline regression catalogue. They must be mapped to actual test functions, test-management records, or reproducible manual procedures as implementation progresses.

Interpretation: These are expected outcomes, not claims that the corresponding tests already exist or pass. Confirm each scenario against the current source, test suite, and runtime behavior.

5. Integration-specific regression tests¶

WhiteLotusAI's initial integration scope is read-only. The following baseline applies independently to Gmail, Google Calendar, Microsoft Teams, Jira, Google Drive, and Slack.

5.1 Provider coverage matrix¶

Use this matrix to track test implementation across the supported integrations. Update a cell only when its status is supported by test records or retained evidence.

The word Track indicates a coverage item that must be assessed; it does not indicate a passing result.

6. AI orchestration and agent safety¶

AI-related tests must validate the complete execution boundary, not just the language model's response. In particular, structured planner output must not bypass application-level validation, authorization, or tool-registry restrictions.

flowchart TD
    A["User request"] --> B["Authenticate and authorize"]
    B --> C["Resolve intent and planner output"]
    C --> D{"Schema and policy valid?"}

    D -- "No" --> E["Reject safely"]
    D -- "Yes" --> F["Validate registered tool"]

    F --> G{"Authorized read-only operation?"}
    G -- "No" --> E
    G -- "Yes" --> H["Execute provider read"]

    H --> I["Validate and sanitize result"]
    I --> J["Return permitted response"]

    classDef work fill:#DCEBFA,stroke:#3975B7,color:#17365D,stroke-width:1.3px
    classDef decision fill:#FCE8CF,stroke:#B7791F,color:#704214,stroke-width:1.3px
    classDef success fill:#D9F2E6,stroke:#23835B,color:#174B37,stroke-width:1.3px
    classDef rejected fill:#F8DEDE,stroke:#B94A48,color:#762D2D,stroke-width:1.3px

    class A,B,C,F,H,I work
    class D,G decision
    class J success
    class E rejected

    linkStyle default stroke:#7B8794,stroke-width:1.3px

6.1 Required AI regression scenarios¶

• Intent handling: ambiguous, contradictory, unsupported, and out-of-scope requests.
• Planner validation: malformed output, unexpected fields, missing fields, and incorrect data types.
• Tool selection: unknown tools, unavailable integrations, and unauthorized tool requests.
• Confirmation integrity: model-generated or planner-generated confirmation flags must not substitute for a trusted user-confirmation mechanism when confirmation is required.
• Prompt injection: malicious instructions in user messages, retrieved documents, and provider content.
• Tool-output trust: misleading or adversarial tool output must not change authorization policy or grant additional permissions.
• Authorization ordering: no provider operation should occur after authentication or authorization has failed.
• Failure handling: retry exhaustion, graph-node exceptions, provider failures, and malformed tool responses.
• State isolation: one user's conversation, retrieved content, credentials, or tool results must not leak into another user's execution.
• Read-only enforcement: planner output and model instructions must not enable write-capable provider operations.

6.2 Security invariants¶

The following invariants should be tested at the relevant execution boundaries:

1. Deny by default: unregistered or unauthorized tools cannot execute.
2. Trusted identity: authorization is derived from trusted application context, not untrusted planner fields.
3. No implicit confirmation: a model-generated flag is not equivalent to an authenticated user action.
4. Provider-side safety: rejected operations must not cause external mutations.
5. Data isolation: results and cached data must remain scoped to the authorized user and conversation.
6. Untrusted content: retrieved content is data, not a source of application policy.

7. API, data handling, and observability¶

Tests should inspect both the returned response and relevant side effects. For example, a denied mutation request should be accompanied by evidence that no provider mutation was issued.

8. Automation and evidence management¶

8.1 Automation status¶

Use consistent labels in the test suite or test-management system.

An automated test is not automatically verified. Verification depends on actual execution results and the quality of the assertions.

8.2 Example evidence record¶

The following YAML illustrates a proposed evidence-record format. It is a template, not a record of a test that has run.

test_id: AUTH-002
title: Prevent cross-user resource access
priority: P0

automation:
  status: planned
  test_reference: null

execution:
  result: not_run
  environment: null
  executed_at: null
  reviewer: null

evidence:
  report: null
  logs: null
  linked_defect: null

When a test is implemented, replace the placeholder values with actual references. Avoid including access tokens, credentials, private messages, or unnecessary production data in evidence artifacts.

8.3 Evidence quality requirements¶

Evidence should make the result independently reviewable:

• Identify the exact test case and test implementation.
• Record the test command, environment, and relevant configuration.
• Include assertions or observations demonstrating the expected behavior.
• Capture failure details and linked defects when applicable.
• Redact secrets and sensitive user or provider data.
• Record the reviewer and execution date for manual tests.
• Preserve evidence in the project's approved test-reporting or CI artifact system.

9. Release gate¶

A release should not be considered ready solely because the test catalogue is complete or the CI pipeline has finished.

9.1 Required release checks¶

• All required P0 authentication, authorization, data-isolation, and read-only enforcement tests pass.
• AI planning and tool dispatch reject malformed, unknown, and unauthorized operations.
• Prompt-injection tests cover both user-provided and retrieved provider content.
• Sensitive credentials are absent from client-facing errors and relevant logs.
• Required P1 tests pass, or each exception has an explicit owner, rationale, and approval under the project's release policy.
• Integration failures, timeouts, rate limits, and revoked credentials have been exercised.
• Test reports and relevant evidence are retained and reviewable.
• Documentation builds and configured CI validation checks pass.
• Known defects and accepted risks are documented before release approval.

9.2 Release decision¶

10. Maintenance and traceability¶

Review the regression matrix when any of the following changes:

• Authentication, authorization, session handling, or tenant/user isolation.
• AI orchestration, planner schemas, tool registry, or dispatch behavior.
• Provider APIs, OAuth scopes, credentials, pagination, or retry policies.
• Read-only policy or integration permissions.
• Caching, logging, error handling, or sensitive-data processing.
• Deployment configuration, CI workflows, or documentation build rules.

Every new security-relevant defect should be assessed for a regression test. Where practical, add the test before or alongside the fix so that the defect cannot recur unnoticed.

The catalogue should remain aligned with the source code and the actual CI test suite. Remove obsolete cases only when their underlying requirement has been retired or replaced and the decision is documented.

Related documentation: API Error Handling · Testing Strategy · Threat Model · Read-Only Integration Security Policy