Availability and recovery

This page says what a self-hosted instance needs to keep answering, what happens when each piece stops, and what you must back up to rebuild it. Kubernetes behaviour on this page is read from the policyarc-server chart, not run on a cluster. The pod and OPA results come from a local instance with two server replicas behind an nginx balancer, run on version 0.65.0.

What the instance depends on

Piece	Replicas by default	When it stops
Authorization server pod	1, set by authServer.replicaCount	With one replica, the instance does not answer until Kubernetes starts a new pod. With more, the other replicas keep serving.
OPA pod	1, set by opa.replicaCount	The server issues no tokens. See When OPA cannot answer.
PostgreSQL	You run it	Every record lives here, so the server cannot issue or check a token without it. We have not stopped the database for this page.

Sign-in sessions, tokens, codes and single-use tokens are rows in PostgreSQL, so they survive the loss of any server pod. Replicas and upgrades lists what each replica keeps in memory.

When a server pod stops

We sent client_credentials token requests through the local balancer one after another, each with a 15 second client timeout, and stopped one of the two replicas 5 seconds into the run. The balancer is nginx with its default upstream settings, not the chart's ingress.

How the replica stopped	Window	Requests	HTTP 200	The others
docker stop	40 seconds	92	90	Two requests hung until the client timeout, about 12 and 27 seconds after the stop
docker kill, which stops the process with no shutdown	40 seconds	34	31	Three requests hung until the client timeout, the first at the moment of the kill
docker kill	60 seconds	113	110	One HTTP 502 at the moment of the kill, from a request the stopped replica was answering. Two requests hung until the client timeout, about 11 and 27 seconds after the kill.

A token issued before each stop introspected as active afterwards, through the replica that was left. The hangs come from the local balancer. Its configuration sets no max_fails, fail_timeout or proxy_connect_timeout, so it keeps the stopped replica in its list, tries it again after a pause, and waits on a connection that does not answer. So the number of failed requests grows with the length of the run. On Kubernetes, a pod leaves its Service's endpoints once it is terminating or not ready, and the chart's preStop pause gives the ingress time to notice before the server shuts down. A request that a pod is answering when its process is killed still fails, as in the second docker kill run. We have not run the pod loss on a cluster.

The chart limits voluntary disruption with a PodDisruptionBudget that lets one server pod and one OPA pod be unavailable at a time. It sets no spread constraints by default, so two replicas can run on the same node. Set authServer.topologySpreadConstraints and opa.topologySpreadConstraints to spread them across nodes or zones.

A rolling restart drops requests unless the drain and shutdown settings fit inside the pod's grace period. The chart refuses to render when they do not. Restart replicas without dropping requests gives the three settings.

When OPA cannot answer

The server asks OPA when it decides a token, authorization, registration or introspection request. We stopped the OPA container of the test instance.

Request	Result while OPA was stopped
POST /token with a valid client	HTTP 500 with a generic error body. No token, and no audit row for the request.
POST /introspect for a token issued a moment before, by a caller with a Bearer token	HTTP 200 with "active": false
The same request by a client with Basic credentials, for another client's token	HTTP 401 invalid_client, Client is not entitled to introspection or uma_protection scope. The server asks OPA whether the caller may introspect.
The same request by a client with Basic credentials, for its own token with the introspection scope	HTTP 200 with "active": false
GET /.well-known/openid-configuration	HTTP 200
The readiness probe, /actuator/health/readiness, on each replica	HTTP 200 with "status": "UP"

The server's readiness probe does not check OPA or the database, so Kubernetes keeps sending requests to server pods that cannot reach OPA. Alert on the metric opa_evaluation_seconds_count{outcome="error"}, see Monitor an instance.

Token issuance worked again one second after we started the OPA container. On the chart, OPA keeps the last bundle it activated in /tmp/opa, so a restarted OPA container starts with that policy. A pod that Kubernetes moves to another node starts with an empty volume and fetches the bundle from the server. OPA's readiness probe checks /health?bundles=true, so an OPA pod gets no requests until it has loaded a bundle. The chart configures OPA to poll the server's bundle endpoint with long polling, 1 to 2 seconds apart.

Back up and restore

The chart and the server take no backups and have no restore command. Back up these yourself.

What	Why
The PostgreSQL database	It holds every record, including the audit log, the encrypted secrets and the signing keys
PBAC_CREDENTIAL_KEY	The encrypted columns and the signing key can only be read with the key that wrote them. With another key the server does not start. See Keys and secrets.
The other four server secrets and the database password	The server refuses to start without them
Your Helm values	They set the issuer, the database URL and the replica counts

OPA needs no backup, because it fetches its policy from the server. We have not restored a database backup into a new instance for this page.