Spec: Kill switch confined to the caller’s tenant

Status: APPROVED and BUILT 2026-10-16. Branch fix/killswitch-tenant-scope. Found while building the agent identity POC (docs/specs/agent-identity-poc.md).

1. Problem & outcome

api/routes_killswitch.py serves three routes, mounted on both planes (core/app.py, and admin_app.py when the module is present):

Route	Tenant comes from	Checked against the caller?
`POST /v1/shield/tools/{tool}/disable`	body `tenant_id`	No
`POST /v1/shield/tools/{tool}/enable`	body `tenant_id`	No
`GET /v1/shield/tools/disabled`	query `tenant_id`	No

Reproduced on main (f63c511) with the shipped configuration (auth.enabled: false) and again with SHIELD_GUARD_REQUIRE_KEY=enforce:

tenant A’s API key disables a tool for tenant B: 200, and B’s tool is disabled (B’s /v1/shield/tool/check calls are blocked by tool_killswitch when the kill switch is enabled);
a request with no credentials does the same: 200;
either can list B’s disabled tools, and re-enable them.

Impact: any party that can reach Shield can switch off any tenant’s tools (a cross-tenant denial of service), switch back on a tool a tenant killed during an incident (undoing its response), and read which tools a tenant has under incident. SHIELD_GUARD_REQUIRE_KEY does not help: it covers the guard endpoints, not these routes.

Not affected (checked): api/routes_mcp_admin.py (/tools/{tool}/disable|enable resolve the tenant with _require_tenant), and the MCP server tools shield_disable_tool / shield_enable_tool (api/routes_mcp_server.py, tenant from the caller’s resolved identity).

Outcome. The three routes act only for the caller’s own tenant, except for an admin identity, which may act for any tenant it names. A request with no credentials is refused.

Non-goals.

Whether an agent holding a tenant key should be able to use the MCP server’s shield_disable_tool (a design question, noted for a separate spec).
The kill switch’s storage or enforcement (storage/tool_killswitch.py, is_tool_disabled on the guard path): unchanged.

2. Plane & latency contract

Both planes mount the router; both change. These routes are management endpoints, off the guard path. is_tool_disabled, which /v1/shield/tool/check reads, is untouched: no guarded-traffic impact.

3. Data model

No change. Keys in storage/tool_killswitch.py stay as they are.

4. API / interface

Request and response shapes unchanged; tenant_id becomes optional.

Who the caller is, in order (the same rule api/routes_agent_identity.py already applies to certificate registration, _authorized_tenant):

Admin identity (core.workload_identity.resolve_workload_identity: the admin key, SPIFFE or mTLS, whichever providers are enabled): may act for any tenant; tenant_id (body or query) is required (400 without it).
Tenant API key (X-API-Key, or the portal session that resolves to it): acts for its own tenant. tenant_id may be omitted (the key’s tenant is used); if given and different, 403.
Neither: 401.

That rule moves to one helper, core/auth.py: authorized_tenant_scope(request, claimed_tenant_id) -> tenant_id, used by these routes and by api/routes_agent_identity.py (replacing its private copy), so the two cannot drift.

Writes and the registry write gate. Disable and enable also go through core.auth.require_registry_write: with a tenant’s write gate enforced, only an admin-scoped key may flip the kill switch, as for other registry writes. With the gate off (the default) nothing changes for a tenant key. Rationale: the kill switch changes what every agent of the tenant can do; it is a registry-class write. Incident responders use the portal (an SSO session with an admin role) or an admin-scoped key.

Audit. log_admin_action keeps its rows and adds scope: "admin" | "tenant" and the caller’s tenant, so a cross-tenant action by an admin is visible as such.

5. Security & backward compatibility

Caller today	After
The portal (sends its own tenant)	Unchanged
Scripts with `X-Admin-Key` naming a tenant	Unchanged
A tenant key naming its own tenant, or omitting it	Unchanged (omitting is new and works)
A tenant key naming another tenant	403 (the bug)
No credentials	401 (the bug)
A tenant key with the write gate enforced	403 for disable/enable unless admin-scoped (consistent with other registry writes)

Escape hatch, as the repo requires for behavior changes: SHIELD_KILLSWITCH_TRUST_BODY_TENANT=true, for one release, lets a tenant credential act for whichever tenant it names, as before. A credential is still required (no credentials stays 401), the registry write gate still applies, every cross-tenant call logs a warning, and its audit row has scope: "cross_tenant" with the caller’s tenant. Documented as temporary. Default off. (The first draft restored the old behavior for every caller, including anonymous ones; the Pepper review on PR #497 rated that critical, and anonymous callers have no legitimate use for it.) Migration note in the release notes: callers that act for other tenants must use an admin identity.

6. Packaging & deploy

No new module (core/auth.py and api/routes_killswitch.py are already in Dockerfile.admin). No new dependency. New env flag above. Rebuild both images.

7. Failure modes & edge cases

Case	Behavior
Tenant store unreachable while resolving a key	503, never “act for the named tenant”
Admin identity, no `tenant_id`	400
Admin identity naming a tenant that does not exist	As today (the write lands; nothing reads it). Not made worse
Portal session for tenant A, body names A	200
`tool_name` path parameter	Unchanged
Escape hatch on	Tenant credentials may name another tenant; still 401 without one; warning per cross-tenant call

Fail mode: fail closed. Anything that cannot establish the caller’s tenant is refused.

8. Test plan (Definition of Done)

New tests/test_killswitch_tenant_scope.py, through the app:

The reproduction, as regression tests: tenant A’s key disabling, enabling and listing tenant B’s tools is 403; no credentials is 401; B’s tools are untouched.
A tenant key on its own tenant: disable, list, enable work, with and without tenant_id.
Admin key: acts for any named tenant; 400 without tenant_id.
Write gate enforced: a runtime-scoped tenant key gets 403 on disable/enable, 200 on list; an admin-scoped key works.
Escape hatch: a tenant key may act for another tenant, logged and audited; no credentials is still 401; the write gate still applies.
Audit rows carry the scope and caller tenant.
api/routes_agent_identity.py behaves exactly as before on the shared helper (its existing tests: tests/test_agent_identity_authz.py, tests/test_cert_identity.py).

Existing tests/test_killswitch.py updated where it relies on the bug (its unauthenticated GET /disabled). Full suite green; CI pytest gate passes.

Task (one PR)

authorized_tenant_scope in core/auth.py; the three routes use it; write gate on disable/enable; escape hatch; audit fields; routes_agent_identity.py on the shared helper; tests 1 to 7.

As built

core/auth.py: caller_tenant_scope (who is calling) and authorized_tenant_scope (which tenant they may act for). Two additions found while building, both in the direction of the spec:
- a global master key (auth.api_keys / SHIELD_API_KEYS) counts as an admin identity, as require_tenant_access already treats it, so scripts using one keep working;
- a workload identity that carries a tenant (an mTLS agent, the tenant_key provider) is confined to that tenant rather than treated as an operator. This also tightens api/routes_agent_identity.py: with mtls in SHIELD_WORKLOAD_IDENTITY_PROVIDERS, an agent’s certificate could previously register certificates in any tenant.
api/routes_agent_identity.py keeps its status codes (403 with no credentials, 500 when the admin key provider is on and SHIELD_ADMIN_KEY is unset) through parameters of the shared helper; it now also accepts a portal session and X-Tenant-Key, both confined to their tenant.
The registry write gate applies to tenant callers only: an admin identity has no tenant key scope to check.
The 503 path applies when the route resolves the key itself; when a middleware already resolved the tenant, that tenant is used.
Docs: docs/enterprise-features.md (who may use the kill switch, migration).
Tests: tests/test_killswitch_tenant_scope.py (22); the seven reproduction tests fail on the previous routes and pass now.