Policy Lifecycle — Templates, Monitor, Enforce

The safe path to a live guardrail policy, for security and platform operators. You never have to author a policy from a blank page, and you never have to flip a brand-new policy straight to “block” in production and hope.

Table of contents
  1. The three moves
  2. Step 1 — Pick a template
  3. Step 2 — Apply it (lands in monitor mode)
  4. Step 3 — Watch what would block
  5. Step 4 — Flip to enforce
  6. Mental model
  7. Notes
  8. Tuning a custom policy that missed
    1. A redact policy on output returns redacted text
  9. Where to next

The three moves

  1. Start from a template — pick an industry preset instead of an empty config.
  2. Run in monitor mode — the policy evaluates every request and records what it would block, but does not block anything. Review it against real traffic.
  3. Flip to enforce — once the would-block list looks right, the same policy starts blocking. One field changes; nothing else moves.

All endpoints below are admin routes — send your admin key as X-Admin-Key.

Step 1 — Pick a template 

curl -s http://SHIELD/v1/admin/policy-templates -H "X-Admin-Key: $ADMIN_KEY"

{ "templates": [
  { "id": "general",            "name": "General Baseline",                 "recommended_mode": "monitor", ... },
  { "id": "healthcare",         "name": "Healthcare (HIPAA)",               "recommended_mode": "monitor", ... },
  { "id": "financial_services", "name": "Financial Services (PCI DSS / SOX)","recommended_mode": "monitor", ... }
]}

Inspect the full guardrail set before applying:

curl -s http://SHIELD/v1/admin/policy-templates/healthcare -H "X-Admin-Key: $ADMIN_KEY"

Step 2 — Apply it (lands in monitor mode) 

curl -s -X POST http://SHIELD/v1/admin/tenants/acme/apply-template \
  -H "X-Admin-Key: $ADMIN_KEY" -H "Content-Type: application/json" \
  -d '{"template_id": "healthcare"}'

{
  "status": "applied",
  "tenant_id": "acme",
  "template_id": "healthcare",
  "policy_mode": "monitor",
  "input_guardrail_count": 5,
  "output_guardrail_count": 4,
  "note": "Policy is in MONITOR mode — it records what would block but does not enforce. Review, then PUT policy-mode 'enforce'."
}

The template replaces the tenant’s input/output guardrails and sets policy_mode: monitor (its recommended mode). Traffic is not affected yet.

Want to apply but go straight to enforcing? Pass {"template_id": "...", "mode": "enforce"}.

Step 3 — Watch what would block

In monitor mode, every decision still runs. A request that the policy would block comes back allowed, but tagged so you (and your SIEM) can see it:

Tool / agent check (POST /v1/shield/tool/check) and input check (POST /guardrails/input) both return:

{
  "allowed": true,
  "action": "monitor",
  "would_block": ["toxicity", "pii_detection"],
  "guardrail_results": [
    { "guardrail": "toxicity", "passed": false, "action": "block", "enforced": false, ... }
  ]
}
  • action: "monitor" — a would-be block was suppressed (vs "block" when enforcing).
  • would_block — exactly which guardrails would have fired. This is your review list.
  • enforced: false on a result — that finding was observed, not acted on.

These flow into the audit log and SIEM events too, so you can review in the telemetry dashboard or your own tooling: “what would this policy have blocked over the last day?”

Check the current mode any time:

curl -s http://SHIELD/v1/admin/tenants/acme/policy-mode -H "X-Admin-Key: $ADMIN_KEY"
# {"tenant_id": "acme", "policy_mode": "monitor"}

Step 4 — Flip to enforce

When the would-block list looks right — no surprising false positives — enforce it:

curl -s -X PUT http://SHIELD/v1/admin/tenants/acme/policy-mode \
  -H "X-Admin-Key: $ADMIN_KEY" -H "Content-Type: application/json" \
  -d '{"mode": "enforce"}'

Now the same policy blocks: allowed: false, action: "block". Nothing about the guardrails changed — only whether their findings are enforced. You can flip back to monitor at any time with the same endpoint.

Every apply and every mode flip is written to the admin audit log (apply_policy_template, set_policy_mode) with the actor and timestamp.

Mental model

  Monitor (dry-run) Enforce
Guardrails evaluate yes yes
Findings recorded / audited yes yes
Request actually blocked no yes (on block)
Top-level action on a would-be block monitor block
would_block list populated yes yes

Safety guarantee. Enforce mode is the original behavior, untouched — the monitor logic only ever relaxes a decision, and only for a tenant that explicitly opted into monitor. A tenant with no policy_mode set enforces, exactly as before.

Notes

  • Propagation. Tenant config is cached per process (short TTL), so a mode flip can take a few seconds to take effect on every worker.
  • warn / log are unchanged. Monitor mode only suppresses blocks. Guardrails set to warn or log behave identically in both modes.
  • Per-guardrail monitor (running only some guardrails in dry-run while others enforce) is a natural extension of resolve_mode() in core/policy_mode.py — not wired yet.
  • Managed admin credentials need the scope. A managed credential (shadm_...) issued with an explicit scope needs tenants:write on the tenant to apply a template or change its mode, and tenants:read to read the mode or the template catalog. The root SHIELD_ADMIN_KEY, SSO operators and credentials issued without a scope are unaffected. Before this check, any managed credential could switch any tenant to monitor mode. If a scoped credential now gets 403 here, issue one with tenants:write for the tenants it manages (POST /v1/admin/credentials) and revoke the old one; a credential’s scope cannot be edited in place.

Tuning a custom policy that missed

A custom policy that returns "violations": 0 is not always a clean scan. The custom_policy_input / custom_policy_output response carries two extra fields whenever they apply:

Field Meaning What to do
suppressed_by_threshold The model DID flag the content, but below the policy’s confidence_threshold (default 0.8), so it was not acted on. Each entry has the policy, the confidence, and the threshold it missed. Lower that policy’s threshold, or sharpen its prompt so the model commits.
errors The policy could not be evaluated at all (LLM error, unparseable verdict). policies_evaluated drops below policies_checked. Investigate the upstream. These are NOT passes.

Long, multi-topic messages are the common cause of a low-confidence miss: the evaluator is told to scan every sentence rather than summarise the message, but a violating clause buried in a long unrelated request still scores lower than the same clause on its own. Watch suppressed_by_threshold before assuming a policy simply does not fire.

SHIELD_CUSTOM_POLICY_FAIL_OPEN (default 1) controls what happens to a policy that failed to evaluate. The default lets it pass, which is the historic behavior. Set it to 0 when your custom policies are the control rather than defense in depth, and an unevaluated policy will block instead.

A redact policy on output returns redacted text

When an output policy’s action is redact, the guardrail makes one extra model call to reproduce the response with only the violating spans replaced by [REDACTED], and the gateway, the OpenAI-compatible route, and agent chat return that text instead of the original. The response carries redacted_text and redacted: true. If the model cannot produce a usable redaction (nothing returned, text unchanged, cut off, or the call failed) the response is withheld with action: block and redaction_failed naming the reason. A redaction is never silently replaced by the original.

Before this change a redact verdict returned the original response under a redact label. Tenants who relied on that for detection only should set the policy action to warn. SHIELD_CHAT_REDACTION=off restores the old verdict-only behavior for rollback; it is not a supported mode.

The same rule holds for pii_leakage with auto_redact: true, which now writes its masked text to redacted_text (it used to write redacted_output, which nothing read).

Where to next