Appearance
Redundancy (Primary and Backup)
Last updated
Redundancy (admin section of the sidebar, /admin/redundancy) runs two appliances as a pair. One is active: it polls and writes the devices, runs schedules, logic and notifications, and holds the shared address. The other is the standby: its device drivers are stopped, and it keeps a copy of the active appliance's configuration and history. When the active appliance stops answering, the standby takes over. You can also hand over by hand (Switch over) to service one of them.
Redundancy is included on every appliance; no license is needed. It starts switched off: an administrator turns it on in Settings → Feature Toggles (Redundancy) on both appliances. It talks over the Mesh link, so Mesh Networking must be on too and the two appliances must be paired on the Mesh page first. Admin only.
Setting up a pair
- Install both appliances with the same software version.
- Turn on Mesh Networking and Redundancy on both, and pair them on the Mesh page.
- On the appliance that should normally run the plant, open Redundancy, tick This appliance is part of a redundant pair, set This appliance is the to Primary, choose The other appliance, fill in the shared settings and Save.
- On the other, do the same with Backup and the primary as The other appliance. The settings under Settings both appliances use come from the primary and are read-only here.
Everything on the backup is replaced by the primary's configuration the first time it receives it: devices, tags, dashboards, users and passwords, scripts and the rest. What it had before is saved once as backups/pre-redundancy-<date>.db. The backup keeps its own licence, Mesh pairings, cloud registration, fleet role and redundancy settings.
The primary keeps running its devices when redundancy is switched on. The backup stops its devices as soon as it is set up as the backup.
What happens when
| Situation | What the appliances do |
|---|---|
| Both start together | Neither runs devices until it has heard the other; then the primary becomes active |
| One starts while the other is active | It becomes the standby and never runs devices |
| The active appliance dies, hangs or loses power | The standby waits the Failover timeout, checks that the active one's own address does not answer and that the witness agrees, then takes over |
| The active appliance's API crashes but the machine keeps running | Its driver service stops renewing a lease and drops every device within half the failover timeout, before the standby takes over |
| The link between the two breaks but both still run | Without a witness, both can become active. With a witness, only the one that can still reach it is active; the other stops its devices |
| Both were active and see each other again | The one that took over most recently keeps running; the other stops at once and raises a Redundancy: both appliances were active alarm |
| The old primary comes back after a failover | It becomes the standby. It hands back only when you Switch over, or by itself after After it has been in sync for (s) if Hand back to the primary automatically when it returns is on |
The epoch shown on the page goes up by one each time an appliance becomes active. It is how the two tell which took over most recently.
Never two masters
Only the active appliance talks to devices. Every way to a device checks this first: connecting, polling, reading, writing (from any screen, script, workflow, ladder logic or OPC-UA client), browsing and device actions. A write refused on the standby says This appliance is the standby of a redundant pair; …. Simulated and demo devices follow the same rule.
The Python driver service is a separate process that keeps running when the API restarts. While redundancy is on, the API renews a small lease file (in /dev/shm) twice a second; the driver service drops all its devices and refuses new ones when the lease says standby, or when it is not renewed for half the failover timeout. So even a hung or crashed API lets go of the devices before the standby can take over.
On the standby, these stay quiet too, so nothing happens twice: schedules and report schedules, script and workflow triggers (running one by hand still works), ladder logic, transaction groups, outbound REST calls, remote database forwarding, SMS and alarm notifications, and the Sparkplug B connection. Startup triggers run when an appliance starts, not when it takes over.
Witness: who is cut off?
A standby that hears nothing from the active appliance cannot tell on its own whether the active one is down or whether it is the one cut off. Witness settles that:
| Witness | The standby takes over when… | The active appliance stops itself when… |
|---|---|---|
| None | the heartbeat stops and the active one's address does not answer either | never |
| Gateway or host | …and the address in Witness address or appliance answers (a ping, or a TCP connection to <host>:<port>) | it loses both the standby and that address |
| Third appliance | …and a third paired appliance has lost the active one too | it loses both the standby and the third appliance |
| Manual (prefer the primary) | never by itself: an operator uses Take over now | never |
Before any takeover the standby also asks the active appliance directly at its address (a signed answer, outside the Mesh link). If the active one answers, the standby does not take over, and the page says the link between them needs fixing.
The third appliance must run this software version and be paired with both. It needs nothing else set up.
The shared address (virtual IP)
With Virtual IP address set, the active appliance adds that address to the interface named in Network interface for the virtual IP (this appliance) and announces it with gratuitous ARP, so HMIs, OPC-UA clients and browsers reach whichever appliance is active at one address. The standby removes it. The active appliance checks every 10 seconds that it still holds it and adds it again if something removed it.
The plant network must allow this. Check before relying on it:
- The address must be free, in the same subnet as both appliances' interfaces, and outside any DHCP range.
- Port security, MAC pinning or static ARP entries on switches and firewalls must allow one IP address to move between two MAC addresses.
- Some PLCs and HMIs ignore gratuitous ARP and keep the old MAC until their ARP entry expires (often minutes). Point those at the address only after testing a switchover with them.
The installer adds a small root-owned helper (/usr/local/sbin/scadaink-vip) and one sudo rule for it, and installs arping. On a development machine, or before the installer has been re-run, the page shows dry run: nothing is really moved.
What is copied
The active appliance copies its whole configuration database, its Data Tables and its uploaded files (images, stylesheets, documents, vision test images, the OPC-UA trust list) to the standby every Copy configuration every (s) when something changed. Only the parts that changed travel. The standby keeps the copy aside and makes it live by restarting its own API: when the configuration changed, and always on the way to taking over, so it takes over with the latest values and log entries too. Takeover time is therefore the failover timeout plus one API restart.
Not copied, because they belong to each appliance: the licence, Mesh pairings and their catch-up queues, cloud identity and feed, fleet role and fleet jobs, network interface names, certificates, the InfluxDB write buffer, the installed Python packages (install the same ones on both appliances), and these redundancy settings.
Queues are copied: Event Streams batches and dead letters, and data connector queues, so after a takeover the new active appliance sends what the other had not delivered yet (a record may arrive twice). On the standby, Event Streams, data connectors, scheduled scripts and workflows, alarm notifications (including those for paired appliances' alarms) and a fleet hub's scheduled backups all stay quiet; people can still run a script or a fleet job there by hand.
When the standby makes a copy live it writes one entry in its audit log, Redundancy: this standby replaced its configuration and audit trail with the active appliance's copy …, saying whether the copied audit trail verifies. The copied trail is the active appliance's own, so it verifies on the standby too. This entry is not a backup restore.
The JWT secrets are not copied either. Stored credentials (secrets, database and broker passwords, MFA keys) are encrypted with a key derived from them, so both appliances must have the same JWT_SECRET and JWT_REFRESH_SECRET. Copy both values from the primary's .env to the backup's and restart it; the page warns while they differ. Users also stay signed in across a takeover only when they match.
History
Each appliance keeps its own InfluxDB history. Each records when it ran the devices, and the other copies the tag history of those periods: the standby continuously, a few seconds behind, and a returning appliance for the time it was away. A gap remains only for the time between the active appliance dying and the standby taking over, when nobody polled. With the SQL historian (an external server both write to) nothing needs copying. Appliance status tags (_STATUS/ on Local Tags) and paired-appliance tags are not copied.
Switching over
Switch over on either appliance hands over to the other. The active appliance takes one last snapshot, the standby makes it live (restarting if it has to) while the active one still runs the devices, then the active appliance stops its devices and releases the address and the standby takes over. If the standby does not take over, the first one takes over again. Progress shows under Events. Do not change the configuration while a switchover runs: a change made in those few seconds can be lost.
Take over now appears on the standby when the active appliance is not answering. Use it only when you are sure the other appliance is off or disconnected from the devices.
Alarms and status tags
Alarms (no rule needed): Redundancy: failover (critical; stays until acknowledged), Redundancy: both appliances were active, Redundancy: the active appliance is not responding (manual witness), Redundancy: this appliance was cut off and stopped its devices, Redundancy: the virtual IP could not be taken over, Redundancy: the other appliance is not responding and Redundancy: the standby is behind (warnings that clear by themselves).
Status tags on Local Tags, for alarm rules, dashboards and history: _STATUS/Redundancy/Role, _STATUS/Redundancy/IsActive, _STATUS/Redundancy/PeerHealthy, _STATUS/Redundancy/PeerRole, _STATUS/Redundancy/ReplicationLagS, _STATUS/Redundancy/HistoryLagS and _STATUS/Redundancy/Failovers.
Honest limits
- Without a witness, a broken link between two running appliances makes both active until they see each other again. Use a witness where that matters.
- A device that only one appliance can reach (a serial device, or a second network only one is wired to) is not redundant.
- Takeover is not instant: the failover timeout (default 6 s) plus one API restart.
- Values polled in the few seconds before the active appliance died may reach the standby's history only when the dead appliance comes back.
- A standby applying new configuration restarts its API; its own pages are unavailable for a few seconds.
- Both appliances must run the same software version; configuration is not copied while they differ.
- Switching redundancy off on the standby while the other appliance is active (for example in Feature Toggles) keeps its devices stopped until its API restarts, so the two never both poll. Switch it off on the active appliance first.
- Tested in software with two appliances on one machine. The virtual IP, gratuitous ARP and the driver lease under systemd need checking on the actual hardware before relying on them.
Troubleshooting
| Message | Cause | Fix |
|---|---|---|
| The other appliance must be paired on the Mesh page first. | No Mesh pairing | Pair them, then save again |
| The other appliance is not set up to pair with this one. | Its Redundancy settings name another appliance, or are off | Set it up on its own Redundancy page |
| Both appliances are set as … | Two primaries or two backups | Make one the primary and the other the backup |
| The two appliances run different software versions … | Database schemas differ | Update both to the same version |
| The two appliances have different JWT secrets … | JWT_SECRET differs | Copy it from the primary's .env |
| The other appliance answers on its address … but not over the mesh link. | The Mesh link is down but both run | Check the Mesh page and the network between them |
| Not taking over: the witness … | The witness cannot be reached, or still hears the active appliance | Check the witness address, or the link between the pair |
| The other appliance is active and running the devices. Switch redundancy off there first … | Switching off redundancy on the standby would make both poll | Switch it off on the active appliance first, or unpair them |
| dry run next to the virtual IP | The address helper is not installed, or not Linux | Re-run the installer |