radiusward
Reliability

Know what survives.
Know what stops.

If a server dies, what happens to the logins and to the session records? This page answers that for each failure radiusward is built for — what pauses, what continues, what recovers — and shows the tests behind each answer.

The one rule: a session record is confirmed to the device only after it is stored on two of the three servers, and nothing confirmed is lost while two of them are up. Everything below follows from it.

Read the accounting contract →Open the test reports ↗

What happens when…

Gateway A
Gateway B
confirmations pause briefly
Server 1down
Server 2leader
Server 3
Export (sink)
Server 1 was the leader; server 2 is elected, the gateways resubmit what was pending.

The leader is killed

What to expect
The remaining two servers elect a new leader — about 2.2 seconds after the crash with the default timeouts. The gateways resubmit commands that were pending. The device waits briefly (at most about 2.5 seconds in the benchmarks); nothing that was confirmed is lost, and a retry of a record that was already stored is answered once more, not stored twice.
Assumptions
Two of the three servers remain and reach each other. Leader failure is detected after cluster.leader.heartbeat.timeout (1.5 s by default; the Helm chart uses 2.5 s).
Recovery
Automatic. The killed server rejoins as a follower when it restarts and catches up from the log.
Limits
Requests in flight wait for the election; a device with a timeout shorter than the failover retransmits, which is counted once.
Evidence
End-to-end leader failover test on every change; the leader killed under load nightly on Kubernetes and in every benchmark run.

Read the explanation → Failure matrix: leader killed under load ↗

Gateway Adown
Gateway B
records keep being confirmed
Server 1leader
Server 2
Server 3
Export (sink)
Gateway A is down; the device sends its retries to gateway B.

A gateway fails

What to expect
Requests in flight on the failed gateway are not answered. The device retries through the other gateway; a record that was already stored is recognised and counted once.
Assumptions
Each device has both gateways configured, as primary and secondary server (on Kubernetes each gateway has its own address).
Recovery
Restart the gateway; it reconnects to the cluster. For planned restarts, gateway.shutdown.drain keeps it answering while a load balancer stops sending to it.
Limits
Whatever was in flight on that gateway depends on the device retrying it. An EAP conversation in progress there starts over, because each conversation stays on the gateway that began it.
Evidence
End-to-end test: a gateway killed between the commit and the response, the retry through the other gateway applied once; gateways rolled under load nightly on Kubernetes.

Read the explanation → Failure matrix: gateway killed after commit ↗

Gateway A
Gateway B
records are not confirmed
Server 1down
Server 2down
Server 3
Export (sink)
Only one server is left: session records are no longer confirmed; authentication goes on.

The majority is lost

What to expect
Session records are no longer confirmed — the device keeps the records and retries. Authentication goes on, by design, except for simultaneous-use limits and EAP-SIM/AKA sequence numbers, which need the cluster.
Assumptions
The device buffers or retries its session records while there is no answer, as RADIUS devices do.
Recovery
Bring a server back. Once two servers reach each other, the retries are confirmed and applied once. If their data is gone, see "Lost quorum".
Limits
This is the trade-off radiusward makes on purpose: consistency before availability for accounting. How long a device can buffer depends on its maker.
Evidence
End-to-end test: no confirmation without a majority, the retry applied once after recovery. A dedicated test that authentication continues without a majority is planned.

Read the explanation → Failure matrix: no majority ↗

Gateway A
Gateway B
records keep being confirmed
Server 1leader
Server 2
Server 3
Export (sink)down
The export to PostgreSQL is down; the log on disk holds what it has not received yet.

A database (sink) is down

What to expect
Records keep being confirmed; the log on disk is the buffer, and the exporter catches up from its checkpoint when the sink is back.
Assumptions
Enough disk for the log to grow while the sink is down; a server below cluster.disk.min.free reports not ready.
Recovery
Automatic once the sink is reachable. An optional sink that was detached is repaired with radiusward admin resync while the log still holds the position.
Limits
A required sink further behind than sink.<name>.max.lag (4 hours by default) stops confirming new records until it catches up — by design, so the log cannot grow without bound.
Evidence
Fault injection nightly on Linux: a PostgreSQL outage under load, a blocking and a detached sink.

Read the explanation → Fault-injection report ↗

Gateway A
Gateway B
records keep being confirmed
Server 1leader
Server 2
Server 3seeding
Export (sink)
Server 3 starts with an empty disk and copies the snapshot and log from the others.

A server loses its disk

What to expect
The server starts empty, copies the latest snapshot and the log after it from the other two, and rejoins. The cluster keeps confirming records meanwhile.
Assumptions
The other two servers are up, and seeding is on (cluster.seed.timeout; the Helm chart sets it).
Recovery
Automatic.
Limits
Until the server has rejoined, the cluster cannot tolerate another loss.
Evidence
Integration test for seeding; a server’s volume replaced under load nightly on Kubernetes.

Read the explanation → Kubernetes report ↗

Gateway A
Gateway B
records are not confirmed
Server 1restored
Server 2restored
Server 3restored
Export (sink)
All three servers are restored from the same backup and go on from its position.

Two servers are lost

What to expect
No confirmations until a majority is back. If the servers’ data is gone, all three are restored from the continuous backup outside the cluster and go on from the backup’s position.
Assumptions
The continuous backup (radiusward cluster-backup) was running and is reachable.
Recovery
radiusward cluster-restore on each server, or automatically with cluster.restore.from; on Kubernetes scripts/k8s-restore.sh does the whole restore.
Limits
What was confirmed after the backup’s position is lost; the backup normally trails the cluster by a moment, and its lag is a metric with an alert.
Evidence
Integration test: a cluster that lost every server restored with every confirmed record; a lost quorum restored nightly on Kubernetes.

Read the explanation → Backup and restore guide ↗

The evidence

Fault injection ↗Nightly on Linux: power loss of all members with LazyFS, partitions, full disks, overload, a PostgreSQL outage.Kubernetes ↗Nightly on kind under load: leader killed, members and gateways rolled, a volume replaced, a lost quorum restored.Benchmarks ↗Throughput and latency ladders and a leader killed under load, with the machine and the method.Failure matrix ↗Every failure case, the expected behaviour and the test that checks it.

Evidence published by the project is not an independent audit, and results from one environment are not a guarantee for another.

Search

Loading…