Carapace

Channel Setup / Channel Smoke Testing

Channel Smoke Validation

This playbook defines reproducible, real-world smoke checks for channel integrations. Use it to produce concrete pass/fail evidence before upgrading channel claims in docs/feature-status.yaml and docs/feature-evidence.yaml.

Scope

Priority for current validation wave:

The smoke report template also supports Telegram and Discord; use the same evidence format.

Preflight

  1. Record version and OS:
    • cara version
    • platform details (uname -a or Windows equivalent)
  2. Start Carapace with target channel configured.
  3. Keep logs open:
cara logs -n 200

cara logs prints a recent tail (not a live follow stream). Re-run it at key steps and after failures to capture relevant evidence.

Pass Criteria

A channel smoke run is considered pass when all are true:

  1. Carapace health is good (cara status healthy).
  2. Channel registration succeeds at startup (no repeated auth/signature errors).
  3. One inbound message is received and produces an agent run.
  4. One outbound reply is delivered back to the same channel.
  5. If optional channel-activity features are enabled for the target channel, those behaviors also match the configured policy.

If any step fails, capture the first failing step and logs.

Signal Smoke

Assumes signal-cli-rest-api is running and configured in carapace.json5 (see Signal Channel Setup).

  1. Start services and verify health:
    • cara status --port 18789
  2. Send one test message from another Signal device/account to the configured Signal number.
  3. Confirm logs show inbound parsing + agent run dispatch from signal_receive.
  4. Confirm reply is delivered in Signal.
  5. If channels.signal.features.typing.enabled is true, confirm the sending device/account sees a typing indicator while Carapace is generating the reply.
  6. If channels.signal.features.readReceipts.enabled is true, confirm the inbound message stays unread until Carapace durably appends it to the session/history store, then transitions to read before the assistant reply is generated or delivered.

Common failure indicators:

Slack Smoke

Assumes Slack bot token and signing secret are configured and Events API request URL points to /channels/slack/events (see Slack Channel Setup).

  1. Verify Slack Events URL challenge succeeds.
  2. Send one message in a subscribed Slack channel.
  3. Confirm logs show inbound event parsing and agent run.
  4. Confirm outbound reply appears in Slack.

Common failure indicators:

Matrix Smoke

Assumes Matrix credentials and encrypted store state are configured (see Matrix / Element).

Run the encrypted-room portions on Unix/macOS. On Windows, matrix.encrypted=true intentionally fails closed until Carapace implements owner-only ACL enforcement for encrypted Matrix state files; Windows smoke can only cover matrix.encrypted=false unencrypted-room behavior for this PR.

Executable evidence harness:

scripts/smoke/matrix-smoke.sh

The harness skips with a clear missing-env list unless these are set: MATRIX_SMOKE_HOMESERVER_URL, MATRIX_SMOKE_USER_ID, MATRIX_SMOKE_ACCESS_TOKEN or MATRIX_SMOKE_PASSWORD, MATRIX_SMOKE_DEVICE_ID, MATRIX_SMOKE_STORE_PASSPHRASE, CARAPACE_CONFIG_PASSWORD, MATRIX_SMOKE_ENCRYPTED_ROOM_ID, MATRIX_SMOKE_UNENCRYPTED_ROOM_ID, MATRIX_SMOKE_ALLOWLIST_USER, MATRIX_SMOKE_VERIFICATION_USER_ID, and MATRIX_SMOKE_VERIFICATION_DEVICE_ID. Optional overrides: CARA_BIN, CARAPACE_CONTROL_URL, CARAPACE_GATEWAY_TOKEN (or CARA_CONTROL_TOKEN), and MATRIX_SMOKE_REPORT_DIR.

  1. Start Carapace and verify runtime wiring:
    • cara status --port 18789
    • cara verify --outcome matrix --port 18789 --matrix-to "<room_id>"
    • cara verify confirms config, runtime registration, control-API reachability, encrypted-store prerequisites, and sends a real Matrix test message to --matrix-to through the daemon-owned Matrix runtime.
  2. Confirm password login persists matrix.accessToken, then restart and confirm token restore works without MATRIX_PASSWORD.
  3. Send one message in an unencrypted room and confirm an agent run is created.
  4. Confirm the assistant reply appears in the same Matrix room. This is the normal conversation-path smoke; record the event ID returned in the agent run as evidence of delivery.
  5. Repeat receive/send in an encrypted room when matrix.encrypted=true.
  6. Invite Carapace from an allowed user/server and confirm auto-join succeeds.
  7. Invite Carapace from a user/server outside the allowlist and confirm the invite is rejected.
  8. Run a SAS verification flow:
    • cara matrix devices
    • cara matrix verify <user> [device]
    • cara matrix accept <flow>
    • read the returned verification.sas emoji or decimals, or rerun cara matrix verifications until SAS data appears
    • compare the SAS values with the other Matrix device out-of-band
    • cara matrix confirm <flow> --match
  9. Restart Carapace and confirm the encrypted Matrix store opens successfully.
  10. Recovery key flow:
    • scripts/smoke/matrix-smoke.sh records only whether cara matrix recovery-key show succeeded; it must not capture the plaintext recovery key in report artifacts.
    • If a manual restore test is needed, display the key directly to the operator and store it outside the smoke report.
    • Stop the daemon. Move {state_dir}/matrix/recovery_key aside and keep the file until the restore test passes.
    • Run cara matrix recovery-key restore --key-file <operator-held-file>, or run cara matrix recovery-key restore and paste the key into the non-echoing prompt.
    • If restore exits non-zero after writing the key because stale cleanup failed, keep the daemon stopped and clear the stale rotation artifacts only after confirming the restored key is current.
    • Restart and confirm cara matrix devices shows the prior trust state preserved (the restored recovery key unlocked cross-signing).
    • Do not expect the daemon to mint a fresh recovery key when the homeserver already has secret-storage recovery configured; missing local key material is a fail-closed state that requires restore.
  11. Store rekey:
    • With the daemon stopped, run cara matrix rekey-store --new. The command rotates SQLite store ciphers AND re-encrypts the inbound DLQ in the same transaction.
    • Restart and confirm the daemon opens the encrypted store under the new pinned passphrase.
    • Confirm any pending DLQ records dispatch on the next replay tick.

Required evidence for #234 sign-off

Every step above must produce one of: a captured cara status JSON snapshot, a journald excerpt, or a Matrix client screenshot showing the expected state. File the artifacts under .local/reports/matrix-smoke-<date>/ and link them in the PR / sign-off issue. The artifacts must demonstrate:

Recorded Local Synapse + Element Run (2026-05-22)

The issue #447 sign-off run used local Synapse + Element Web on macOS with OrbStack. The per-run raw report directory stayed local to the runner because it can contain host-specific paths, screenshots, and log excerpts, so the committed audit record is this summary plus the reusable harness and checklist above.

Stable result:

The harness's first summary.json is a preliminary machine summary and can record manual-required markers before browser/manual supplemental coverage is folded in. For this run, the supplemental issue #447 summary was the sign-off artifact and mapped those manual-required markers to the passing Element, invite, recovery, rekey, and SAS checks above.

Common failure indicators:

Evidence Capture

Open a smoke report with:

Template:

Open a smoke report