Troubleshoot an instance

Each entry gives the symptom, a check you can run, and the fix. For the errors of the first environment track, see Troubleshoot your first environment.

Set two values. The blocks marked as checks run in order in one shell, and each one fails if the answer is not the one this page describes.

export POLICYARC_URL="https://your-environment.example"
export POLICYARC_ADMIN_KEY="paste the Admin API key"

A request was denied

Symptom. A token request returns HTTP 400 with {"error":"access_denied","error_description":"Access denied"}. The response does not say which check failed.

Check. The reason is in the request's audit row, in policyDecision.reason. These blocks make a denied request and read its reason. A client that registered itself has no grant for client_credentials, so policy denies its token request.

reg=$(curl -fsS -X POST "$POLICYARC_URL/register" -H 'Content-Type: application/json' \
  -d '{"client_name": "why-denied", "redirect_uris": ["https://why-denied.example/cb"], "token_endpoint_auth_method": "client_secret_basic"}')
export DENIED_ID=$(jq -er .client_id <<<"$reg")
export DENIED_SECRET=$(jq -er .client_secret <<<"$reg")
resp=$(curl -sS -w '\n%{http_code}' -u "$DENIED_ID:$DENIED_SECRET" "$POLICYARC_URL/token" \
  -d grant_type=client_credentials -d scope=orders:read -d resource_types=urn:example:orders)
echo "$resp"
[ "$(tail -n1 <<<"$resp")" = 400 ] && grep -qF '"error":"access_denied"' <<<"$resp" ||
  { echo 'Expected HTTP 400 access_denied' >&2; false; }

row=$(curl -fsS "$POLICYARC_URL/admin/api/audit-logs?clientId=$DENIED_ID&endpoint=/token&outcome=deny&size=1" \
  -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" | jq -c '.content[0]')
jq '{outcome, reason: .policyDecision.reason, deny_reasons: .policyDecision.deny_reasons}' <<<"$row"
jq -e '.policyDecision.reason == "grant_type_not_allowed"' <<<"$row" >/dev/null

The reason is grant_type_not_allowed. policyDecision.deny_reasons lists the reasons that custom controls gave, and is empty here. policyInput on the same row is what the policy engine evaluated.

Fix. Change what the reason names. For grant_type_not_allowed, grant the client the grant type and its resources. See Clients and access. For a denial at the gateway, the gateway row carries pdp_reason. See Troubleshoot your first environment.

Symptom. An MCP client opens the browser, and the sign-in ends with one of these.

What the browser or client shows	Cause
A JSON page with `"error_description":"redirect_uri not registered for client"`	The client sent a redirect URI it did not register
The client reports `login_required` with `No active session; authentication required`	The instance has no default identity provider to send the user to

Check. List the client's /authorize rows. An error row with no policyDecision and a requestSummary.redirect_uri the client never registered is the first cause. Compare it with the client's redirectUris.

export CLIENT_ID="the client id from the row"
curl -fsS "$POLICYARC_URL/admin/api/audit-logs?clientId=$CLIENT_ID&endpoint=/authorize&size=5" \
  -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" | jq '.content[] | {createdAt, outcome, redirect_uri: .requestSummary.redirect_uri}'
curl -fsS "$POLICYARC_URL/admin/api/clients/$CLIENT_ID" -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" | jq .redirectUris

The second cause writes no audit row. Check that the instance has an identity provider and a default one.

curl -fsS "$POLICYARC_URL/admin/api/identity-providers" -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" | jq length
curl -fsS "$POLICYARC_URL/admin/api/policy-data" -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" |
  jq '[.[] | select(.dataKey == "pbac.operator.controls.idp_selection") | .payload | fromjson][0].default_idp'

The default lives in the pbac.operator.controls.idp_selection row, which a new instance does not have. We ran both on an instance with no identity provider. The first printed 0 and the second printed null, and /authorize answered login_required. After we set default_idp in that row, the second printed the provider's key.

Fix. For an unregistered redirect URI, add it to the client with PUT /admin/api/clients/{clientId}, sending the client as GET returns it with the URI added to redirectUris. We did this, and /authorize then accepted the URI. For a missing identity provider, add one. An identity provider becomes the default only when you create it enabled on an instance with no default. Enabling one you created disabled does not set it. We have not run this fix through a real identity provider sign-in. See Identity providers.

A policy change does not take effect

Symptom. You changed a rule or a data row, and decisions still follow the old policy.

Check. Each cause has its own check.

The deploy was refused. POST /admin/api/policy-bundle/deploy returns HTTP 422 with "error": "bundle_not_activated" and "rolledBack": true when the policy engine cannot load the bundle. Nothing changed.
The policy engine has not loaded it yet. A data row write is not confirmed the way a deploy is, and the policy engine loads it a few seconds later.
The instance was upgraded and keeps old bundled rules. See Update the bundled policy rules.
A deploy used "mode": "replace", which deletes the rules and data rows the request leaves out, apart from the bundled ones, and so removes the denylist. See Deploy and roll back policy.

This block deploys a rule with a missing closing brace and checks the refusal. The policy engine takes about 10 seconds to refuse it.

export PROBE="$(mktemp -d)/troubleshoot_probe.rego"
cat > "$PROBE" <<'EOF'
package pbac.controls.troubleshoot_probe

import rego.v1

deny contains reason if {
	input.resource.type == "urn:example:never"
	reason := {"code": "troubleshoot_probe"}
EOF
resp=$(jq -n --rawfile rego "$PROBE" '{mode: "merge", rules: [{path: "pbac/controls/troubleshoot_probe.rego", content: $rego}]}' |
  curl -sS -w '\n%{http_code}' -X POST "$POLICYARC_URL/admin/api/policy-bundle/deploy" \
    -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" -H 'Content-Type: application/json' -d @-)
echo "$resp"
[ "$(tail -n1 <<<"$resp")" = 422 ] || { echo 'Expected HTTP 422' >&2; false; }
sed '$d' <<<"$resp" | jq -e '.error == "bundle_not_activated" and .rolledBack == true' >/dev/null

Fix. Run opa check on the file before you deploy, to see the parse error and its line. Fix the rule and deploy it again. This block adds the brace and expects the policy engine to confirm the deploy.

echo '}' >> "$PROBE"
jq -n --rawfile rego "$PROBE" '{mode: "merge", rules: [{path: "pbac/controls/troubleshoot_probe.rego", content: $rego}]}' |
  curl -fsS -X POST "$POLICYARC_URL/admin/api/policy-bundle/deploy" \
    -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" -H 'Content-Type: application/json' -d @- |
  jq -e '.activation == "confirmed"'

Remove the probe rule when you are done.

id=$(curl -fsS "$POLICYARC_URL/admin/api/policy-rules" -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" |
  jq -er '.[] | select(.path == "pbac/controls/troubleshoot_probe.rego") | .id')
code=$(curl -sS -o /dev/null -w '%{http_code}' -X DELETE "$POLICYARC_URL/admin/api/policy-rules/$id" -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY")
[ "$code" = 204 ] || { echo "Expected HTTP 204, got $code" >&2; false; }

Token, register and admin calls return HTTP 500

Symptom. /token, /register and admin changes return HTTP 500.

Check. Search the server's log for Failed to persist audit log. The server returns 500 when it cannot write the audit row. See When an audit write fails. Check that the database is up, that its disk has room, and that nothing changed the audit tables.

Fix. Fix the database, then check the state of the change before you retry it. An admin change that returned 500 can still have been saved.

The server does not start

Symptom. The container exits during start.

Check. Read the last lines of its log.

Log message	Fix
`... (PBAC_ADMIN_API_KEY) is required`, or the same for `PBAC_ADMIN_PEPPER`, `PBAC_CREDENTIAL_KEY` or `PBAC_GATEWAY_LINK_STATE_SECRET`	Set the value. See Harden an instance.
`pbac.bundle.api-key is set to a known-public dev key`	Set `PBAC_BUNDLE_API_KEY`. A blank value falls back to the public one.
`Error creating bean with name 'signingKeySeeder'`	`PBAC_CREDENTIAL_KEY` differs from the value the instance was started with. Restart with the old value. See Keys and secrets.

The Admin API returns HTTP 401

Symptom. A call that worked returns {"error":"unauthorized","error_description":"Valid admin API key required"}.

Check. Ask another admin key holder to list the keys and look at the revokedAt of yours.

curl -fsS "$POLICYARC_URL/admin/api/admin-keys" -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" | jq '.[] | {label, revokedAt}'

Fix. A revoked key stays revoked. Mint a new one. If you used the value of PBAC_ADMIN_API_KEY, the server may have restarted with a new value, which retires the old one. See Rotate the key in PBAC_ADMIN_API_KEY.

Send feedback about this page

A request was denied​

A sign-in does not finish​

A policy change does not take effect​

Token, register and admin calls return HTTP 500​

The server does not start​

The Admin API returns HTTP 401​

A request was denied

A sign-in does not finish

A policy change does not take effect

Token, register and admin calls return HTTP 500

The server does not start

The Admin API returns HTTP 401