Run more than one replica and upgrade

Use this page when a self-hosted instance runs more than one authorization server replica, and when you move an instance to a new image. On a hosted environment, see Updating an environment.

All replicas use one PostgreSQL database. Set the replica count for load and cost, and check the connection budget below when you raise it. The Helm value is authServer.replicaCount, 1 by default.

State	Where it lives	Across replicas
Browser sign-in sessions	The `spring_session` table	Any replica continues a sign-in. The balancer needs no session affinity.
Tokens, codes and single-use tokens	Database tables	A code or a single-use token is spent once, whichever replica sees it
The policy bundle	A cache in each replica	A write on one replica tells the others to drop their copy. A missed signal leaves a copy at most 30 seconds old.
Rate limit counts	Memory in each replica	Each replica counts on its own, so the limit across N replicas is N times `PBAC_RATE_LIMIT_REQUESTS`
Open MCP streams	Memory in the replica that holds the stream	A completion that arrives at another replica is passed to the one that holds the stream

We wrote a policy data row through one replica of a two-replica instance. Within a second, both replicas served the same new bundle ETag.

The replicas signal each other over PostgreSQL LISTEN and NOTIFY. Each replica opens one connection for this, outside its connection pool. That connection needs a direct route to PostgreSQL. A transaction-mode pooler such as PgBouncer cannot carry it.

Size the database connections

Each replica opens up to authServer.hikariMaxPoolSize pooled connections, 19 by default, plus the signal connection, so 20 in all. Set authServer.postgresMaxConnections to your server's max_connections, and the chart refuses to render when replicas × (pool + 1) + authServer.reservedConnections exceeds it. reservedConnections is 20 by default. We rendered the chart with 3 replicas. At postgresMaxConnections=79 it refused with 3 replicas x (19 pooled + 1 listen) + 20 reserved = 80 > postgresMaxConnections=79, and at 80 it rendered. The check only counts this instance. On a database server that other instances share, keep the pool small.

Prove the balancer spreads requests

A test suite that passes against one replica proves nothing about two. This block sends 40 requests through the balancer and counts how many each replica served, from each replica's own metrics.

Set POLICYARC_URL to the balancer and POD_URLS to one URL for each replica, separated by spaces. On Kubernetes, kubectl port-forward to each pod's port 8080 gives you one local URL per pod. We have not run the port-forward route.

export POLICYARC_URL="https://your-environment.example"
export POD_URLS="http://localhost:18080 http://localhost:18081"

count() {
  curl -fsS "$1/actuator/prometheus" |
    awk '/^http_server_requests_seconds_count\{/ && /uri="\/.well-known\/openid-configuration"/ {n += $NF} END {print n + 0}'
}
before=(); for pod in $POD_URLS; do before+=("$(count "$pod")"); done
for i in $(seq 1 40); do curl -fsS -o /dev/null "$POLICYARC_URL/.well-known/openid-configuration"; done
i=0; total=0; spread=1
for pod in $POD_URLS; do
  got=$(( $(count "$pod") - before[i] ))
  echo "$pod served $got of 40"
  total=$((total + got)); [ "$got" -gt 0 ] || spread=0; i=$((i + 1))
done
[ "$total" = 40 ] || { echo "Expected the replicas to serve all 40 requests, they served $total" >&2; false; }
[ "$spread" = 1 ] || { echo 'A replica served no requests' >&2; false; }

We ran it on a two-replica instance behind a round-robin balancer, and each replica served 20 of 40. Sent straight to one replica, the same block failed, because the other replica served 0.

Restart replicas without dropping requests

A rolling restart drops requests unless two settings work together. The chart's defaults set both.

Setting	Default	What it does
`authServer.preStopDrainSeconds`	`8`	Delays the stop signal so the ingress stops sending new requests first
`authServer.shutdownPhaseTimeoutSeconds`	`25`	How long the server waits for requests it already accepted
`authServer.terminationGracePeriodSeconds`	`45`	Must exceed the two above added together. The chart refuses to render otherwise. We rendered it at 33, and it refused. At 34 it rendered.

Upgrade to a new image

Each replica runs the database migrations when it starts. The migration tool takes a database lock, so one replica migrates and the others wait. You do not run migrations by hand. We have not run a migration across replicas.

Update the bundled policy rules

A new image does not update the bundled policy rules on an instance that already has them. The authorization server writes its bundled rules only into an empty rules table. An upgraded instance keeps the rules it was first given, and a release that changed a rule does not take effect until you deploy that rule. A release that adds a rule is the same.

These blocks copy the rules out of the new image, list the ones that differ from the instance, and deploy them in one batch. They need Docker, unzip and jq. Set POLICYARC_IMAGE to the image you upgraded to. The chart's image is authServer.image.repository with authServer.image.tag, and an empty tag means the chart's appVersion.

export POLICYARC_URL="https://your-environment.example"
export POLICYARC_ADMIN_KEY="paste the Admin API key"
export NEW_TAG="the image tag you deployed"
export POLICYARC_IMAGE="registry.identos.com/registry/policyarc/authorization-server:$NEW_TAG"

Copy the bundled rules out of the image.

mkdir -p shipped
cid=$(docker create --platform linux/amd64 "$POLICYARC_IMAGE")
docker cp "$cid:/app/app.jar" shipped/app.jar >/dev/null
docker rm "$cid" >/dev/null
unzip -o -q shipped/app.jar 'BOOT-INF/classes/baseline/policies/*' -d shipped

List the bundled rules that the instance lacks or holds in another version.

live=$(curl -fsS "$POLICYARC_URL/admin/api/policy-rules" -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" |
  jq -r '.[].id' | while read -r id; do
    curl -fsS "$POLICYARC_URL/admin/api/policy-rules/$id" -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY"
  done | jq -s 'map({(.path): .content}) | add')
root=shipped/BOOT-INF/classes/baseline/policies
: > changed.txt
(cd "$root" && find . -name '*.rego' | sed 's#^\./##' | sort) | while read -r path; do
  jq -e --arg p "$path" --rawfile f "$root/$path" '.[$p] == $f' <<<"$live" >/dev/null || echo "$path" >> changed.txt
done
echo "$(wc -l < changed.txt | tr -d ' ') bundled rules differ from the instance"
cat changed.txt

Read the list before you deploy. If you edited a bundled rule on purpose, the deploy replaces your edit. Deploy the listed rules in one batch, because rules depend on each other and one at a time a save can fail.

root=shipped/BOOT-INF/classes/baseline/policies
while read -r path; do jq -n --arg path "$path" --rawfile content "$root/$path" '{path: $path, content: $content}'; done < changed.txt |
  jq -s '{mode: "merge", rules: .}' |
  curl -fsS -X POST "$POLICYARC_URL/admin/api/policy-bundle/deploy" \
    -H "X-Admin-API-Key: $POLICYARC_ADMIN_KEY" -H 'Content-Type: application/json' -d @- |
  jq -e '.activation == "confirmed"'

The deploy prints true once the policy engine runs the new rules. A refused deploy returns HTTP 422 and changes nothing. See Deploy and roll back policy. Run the list block again, and it reports 0.

We changed one bundled rule on a test instance to stand for an older release, and the list block named it. After the deploy, the list block reported 0. These blocks compare rules only. They do not compare the bundled data rows.

Refresh connector tools

A new image does not refresh the routes and tools of connectors already installed. See Keep connector tools current.

Send feedback about this page

What the replicas share​

Size the database connections​

Prove the balancer spreads requests​

Restart replicas without dropping requests​

Upgrade to a new image​

Update the bundled policy rules​

Refresh connector tools​

What the replicas share