Failure behaviour
Leader failure, gateway failure, loss of the majority, a sink outage, a lost disk and a lost quorum: what pauses, what retries, what recovers, and what each assumes.
On this page
What a NAS and an operator see when part of radiusward fails. Each scenario states what to expect, what it assumes, how it recovers, where it ends, and which test checks it. The rule behind all of them is the accounting contract: nothing is acknowledged before it is safe on a majority, and nothing acknowledged is lost while a majority survives.
The scenarios: the leader is killed · a gateway fails · the majority is lost · a sink is down · a server loses its disk · two servers are lost.
The leader is killed
Gateway A
Gateway B
confirmations pause briefly
Server 1down
Server 2leader
Server 3
Export (sink)
- What to expect
- The remaining two servers elect a new leader — about 2.2 seconds after the crash with the default timeouts. The gateways resubmit commands that were pending. The device waits briefly (at most about 2.5 seconds in the benchmarks); nothing that was confirmed is lost, and a retry of a record that was already stored is answered once more, not stored twice.
- Assumptions
- Two of the three servers remain and reach each other. Leader failure is detected after cluster.leader.heartbeat.timeout (1.5 s by default; the Helm chart uses 2.5 s).
- Recovery
- Automatic. The killed server rejoins as a follower when it restarts and catches up from the log.
- Limits
- Requests in flight wait for the election; a device with a timeout shorter than the failover retransmits, which is counted once.
- Evidence
- End-to-end leader failover test on every change; the leader killed under load nightly on Kubernetes and in every benchmark run. Failure matrix: leader killed under load ↗
A gateway fails
Gateway Adown
Gateway B
records keep being confirmed
Server 1leader
Server 2
Server 3
Export (sink)
- What to expect
- Requests in flight on the failed gateway are not answered. The device retries through the other gateway; a record that was already stored is recognised and counted once.
- Assumptions
- Each device has both gateways configured, as primary and secondary server (on Kubernetes each gateway has its own address).
- Recovery
- Restart the gateway; it reconnects to the cluster. For planned restarts, gateway.shutdown.drain keeps it answering while a load balancer stops sending to it.
- Limits
- Whatever was in flight on that gateway depends on the device retrying it. An EAP conversation in progress there starts over, because each conversation stays on the gateway that began it.
- Evidence
- End-to-end test: a gateway killed between the commit and the response, the retry through the other gateway applied once; gateways rolled under load nightly on Kubernetes. Failure matrix: gateway killed after commit ↗
The majority is lost
Gateway A
Gateway B
records are not confirmed
Server 1down
Server 2down
Server 3
Export (sink)
- What to expect
- Session records are no longer confirmed — the device keeps the records and retries. Authentication goes on, by design, except for simultaneous-use limits and EAP-SIM/AKA sequence numbers, which need the cluster.
- Assumptions
- The device buffers or retries its session records while there is no answer, as RADIUS devices do.
- Recovery
- Bring a server back. Once two servers reach each other, the retries are confirmed and applied once. If their data is gone, see "Lost quorum".
- Limits
- This is the trade-off radiusward makes on purpose: consistency before availability for accounting. How long a device can buffer depends on its maker.
- Evidence
- End-to-end test: no confirmation without a majority, the retry applied once after recovery. A dedicated test that authentication continues without a majority is planned. Failure matrix: no majority ↗
A database (sink) is down
Gateway A
Gateway B
records keep being confirmed
Server 1leader
Server 2
Server 3
Export (sink)down
- What to expect
- Records keep being confirmed; the log on disk is the buffer, and the exporter catches up from its checkpoint when the sink is back.
- Assumptions
- Enough disk for the log to grow while the sink is down; a server below cluster.disk.min.free reports not ready.
- Recovery
- Automatic once the sink is reachable. An optional sink that was detached is repaired with radiusward admin resync while the log still holds the position.
- Limits
- A required sink further behind than sink.<name>.max.lag (4 hours by default) stops confirming new records until it catches up — by design, so the log cannot grow without bound.
- Evidence
- Fault injection nightly on Linux: a PostgreSQL outage under load, a blocking and a detached sink. Fault-injection report ↗
A server loses its disk
Gateway A
Gateway B
records keep being confirmed
Server 1leader
Server 2
Server 3seeding
Export (sink)
- What to expect
- The server starts empty, copies the latest snapshot and the log after it from the other two, and rejoins. The cluster keeps confirming records meanwhile.
- Assumptions
- The other two servers are up, and seeding is on (cluster.seed.timeout; the Helm chart sets it).
- Recovery
- Automatic.
- Limits
- Until the server has rejoined, the cluster cannot tolerate another loss.
- Evidence
- Integration test for seeding; a server’s volume replaced under load nightly on Kubernetes. Kubernetes report ↗
Two servers are lost
Gateway A
Gateway B
records are not confirmed
Server 1restored
Server 2restored
Server 3restored
Export (sink)
- What to expect
- No confirmations until a majority is back. If the servers’ data is gone, all three are restored from the continuous backup outside the cluster and go on from the backup’s position.
- Assumptions
- The continuous backup (radiusward cluster-backup) was running and is reachable.
- Recovery
- radiusward cluster-restore on each server, or automatically with cluster.restore.from; on Kubernetes scripts/k8s-restore.sh does the whole restore.
- Limits
- What was confirmed after the backup’s position is lost; the backup normally trails the cluster by a moment, and its lag is a metric with an alert.
- Evidence
- Integration test: a cluster that lost every server restored with every confirmed record; a lost quorum restored nightly on Kubernetes. Backup and restore guide ↗
Not covered
- Two members lost at once with their disks, and no backup: the cluster cannot be restored. Run the continuous backup.
- Storage that ignores fsync (some virtual disks, or caches without power-loss protection) voids the durability guarantee; it is tested on Linux, not on macOS.
- Members on high-latency links do not lose data, but every acknowledgement waits for the round trip.