13. High Availability¶
High Availability is a new feature that will be present in dr-provision starting with version 4.3.0. To begin with, the working model will be Active/Passive with unidirectional data replication and manual failover. Automated failover and failback may be added in future versions of dr-provision. Currently, high-availability should be considered a technical preview – the details of how it is implemented and how to manage it are likely to change based on customer feedback.
There are a few conditions that need to be met in order to set up an HA cluster of dr-provision machines:
- A fast network between the nodes you want to run dr-provision on. Data replication between HA nodes uses synchronous log replay and file transfer, so a fast network (at least gigabit Ethernet) is required to not induce too much lag into the system as a whole.
- A virtual IP address that can be moved from node to node. dr-provision will handle adding and removing the virtual IP from a specified interface and sending out gratuitous ARP packets on failover events. If the nodes forming the HA cluster are not in the same layer 2 broadcast domain or subnet, then you will need to arrange for the appropriate route changes required.
- Enough storage space on each node to store a complete copy of all replicated data. dr-provision will wind up replicating all file, ISO, job log, content bundle, plugin, and writable data to all nodes participating in the HA cluster. The underlying storage should be fast enough that write speeds do not become a bottleneck – we recommend backing the data with a fast SSD or NVME device.
- A high-availability entitlement in your license. High-availability is a licensed enterprise feature. If you are not sure if your license includes high-availability support, contact email@example.com.
The high-availability support code in dr-provision assumes a model where the machines are in the same layer 2 broadcast domain to allow for moving the HA IP address via gratuitous AR), and that the writable storage that dr-provision uses is not shared (via NFS, iSCSI, drbd, or whatever) between servers running dr-provision.
- If you are running on separate broadcast domains, you will need to either ensure that there is an alternate mechanism for ensuring that packets destined for the HA IP address get routed correctly, or accept that provisioning operations will fail from the old network until the clients are informed of the new IP address. In the latter case, you probably do not want to use the integrated HA support directly, as it assumes that it has a virtual IP that will migrate from one dr-provision instance to another.
- If you are using a shared storage mechanism (NFS, DRBD, iSCSI, or some form of distributed filesystem), then you should not use our integrated HA support, as it will lead to data corruption. You should also make sure you never run more than one instance of dr-provision on the same backing data at the same time, or the data will be corrupted.
Aside from settings listed later in this section, configuration flags and startup options for the dr-provision services participating in an HA cluster should be identical. It is not required that the servers participating in the HA cluster have identical versions, but they must be running on the same OS and system architecture types. If you try to add a server version to a cluster that is incompatible, it will fail with an error telling you what to do to resolve the issue.
13.3.1. High Availability Startup Options¶
- –static-ip (or the environment variable RS_STATIC_IP)
- Not specifically a high-availability startup option, if it is configured it must be different on each server.
- –drp-id (or the environment variable RS_DRP_ID)
- Also not specifically a high-availability startup option, this must be different on each server.
- –ha-id (or the environment variable RS_HA_ID)
- Must be the same on all nodes participating in an HA cluster.
- –ha-enabled (or the environment variable RS_HA_ENABLED)
- Must be true on all nodes participating in an HA cluster.
- –ha-address (or the environment variable RS_HA_ADDRESS)
- This is the IP address and netmask of the virtual IP that the active cluster member will use for communication. It must be in CIDR format (aa.bb.cc.dd/xx)
- –ha-interface (or the environment variable RS_HA_INTERFACE)
- This is the Ethernet interface that the ha address should be added to and removed from when dr-provision transitions between active and passive.
- –ha-passive (or the environment variable RS_HA_PASSIVE)
- This must be true on the nodes that should start as passive nodes by default. In practice, this means every node after the initial node.
- –ha-token (or the environment variable RS_HA_TOKEN)
This is the authentication token that HA nodes use to authenticate and communicate with each other. It should be identical across the nodes, and it should be a superuser auth token with a long lifetime. With the default usernames, you can generate such a token with:
drpcli users token rocketskates ttl 3y
and then extracting the Token field from the resulting JSON.
- –ha-interface-script (or the environment variable RS_HA_INTERFACE_SCRIPT)
This is the full path to the script that should be run whenever dr-provision needs to add or remove the ha address to the ha interface. If not set, dr-provision defaults to using
ip addr addand
ip addr delinternally on Linux, and
ifconfigon Darwin. You can use the following example as a starting point:
#/usr/bin/env bash # $1 is the action to perform. "add" and "remove" are the only ones supported for now. # $2 is the network interface to operate on. It will be set to the value of --ha-interface. # $3 is the address to add or remove. It will be set to the value of --ha-address. case $1 in add) sudo ip addr add "$3" dev "$2";; remove) sudo ip addr del "$3" dev "$2";; *) echo "Unknown action $1"; exit 1;; esac
Customize to taste to suit your preferred method of getting authority to add and remove addresses to interfaces.
This bootstrapping documentation will assume that you are working with dr-provision running as a native service managed by systemd on a Linux server.
13.4.1. The Initially Active Node¶
To start bootstrapping an HA cluster, start by installing what you want to be the default active dr-provision node. Once it is up and running, create a file named /etc/systemd/system/dr-provision.service.d/20-ha.conf with the following contents:
[Service] # RS_HA_ENABLED tells dr-provision to operate in high-availability mode. Environment=RS_HA_ENABLED=true # RS_HA_INTERFACE is the network interface that dr-provision will add/remove the # virtual IP address to. This interface should be one that machines being managed by # dr-provision can access. Environment=RS_HA_INTERFACE=kvm-test # RS_HA_ADDRESS is the IP address and netmask in CIDR format that all communication to # and from dr-provision will use. Environment=RS_HA_ADDRESS=192.168.124.200/24 # RS_HA_ID is the cluster ID. This must be the same for all members participating in the cluster. Environment=RS_HA_ID=8c:ec:4b:ea:d9:fe # RS_HA_TOKEN is a long-lived access token that the cluster nodes will use to authenticate with each other. # You can generate a usable token with: # # $ drpcli users token rocketskates ttl 3y |jq -r '.Token' Environment=RS_HA_TOKEN=your-token # RS_HA_PASSIVE is an intial flag (not used after synchronization) to identify the active endpoint. Environment=RS_HA_PASSIVE=false
Once that file is created, reload the config and restart dr-provision:
$ systemctl daemon-reload $ systemctl restart dr-provision
When dr-provision comes back up, it will be running on the IP address you set aside as the HA IP address.
13.4.2. The Initially Passive Nodes¶
WARNING: Do not start a passive endpoint(s) in “normal mode.” When installing a passive endpoint, the active endpoint _must_ be available when the endpoint is started.
Perform the same installation steps you used for the initially active node, but change the RS_HA_PASSIVE line to false in the /etc/systemd/system/dr-provision.service.d/20-ha.conf file
which will cause the node to come up as a passive node when you start it up. The first time you start up the node,
it will replicate all of the runtime data from the active mode, which (depending on your network bandwidth and
how busy the active node is) may take awhile. You can monitor the progress of the replication by
watching the output of
`journalctl -fu dr-provision` — when it says “Stream switch to realtime streaming” the
passive node is fully caught up to the active node.
13.5. Switching from Active to Passive¶
To switch a dr-provision instance between states, an API call will need to be done. drpcli can be used to send that API call. Issuing a POST request with empty JSON object to /api/v3/system/active and /api/v3/system/passive will cause the system to transition to active or passive, respectively.
As of right now, there are no other mechanisms (automated or manual) for changing HA state on a node.
When doing a practice failover, the active endpoint should be stopped first.
To stop the active endpoint (becomes passive):
// deactivate endpoint (goes into passive mode) drpcli system passive
To promote a passive endpoint to active
// activate endpoint (goes into active mode) drpcli system active
Prior to v4.5.0, Signals were used to shift state. SIGUSR2 was used to go from active to passive and SIGUSR1 was used to go from passive to active.
13.6. Install Video¶
This video was created at the time of v4.3 beta: https://youtu.be/xM0Zr3iL5jQ. Please check for more recent updates.
13.7.1. Log Verification¶
It is normal to see
Error during replication: read tcp [passive IP]:45786->[cluster IP]:8092: i/o timeout on the
passive endpoints logs when the active endpoint is killed or switches to passive mode. This is an indication that the
active endpoint has stopped sending updates.
13.7.2. Transfer Start-up Time¶
It may take up to a minute for a passive endpoint to come online after it has received
13.7.3. Network Interface Locked¶
It is possible for the HA interface to become locked if you have to stop and restart the service during configuration
testing. To clear the interface, use
`ip addr del [ha ip] dev [ha interface]`
This happens because Digital Rebar is attaching to (and detaching from) the cluster IP. If this process is interrupted, then the association may not be correctly removed.
13.7.4. WAL File Checksums¶
When operating correctly, all the WAL files should match on all endpoints. You can check the signature of the wal files using hexdump -C
cd /var/lib/dr-provision/wal hexdump -C base.0 |less
13.7.5. Active Endpoint File ha-state is Passive:true¶
Digital Rebar uses the
ha-state.json file in it’s root directory (typically
/var/lib/dr-provision) to track
transitions from active to passive state.
removing this file incorrectly can cause very serious problems! This is a last resort solution.
ha-state.json file has a single item JSON schema that changes from true to false depending on the endpoint HA state. This file can be updated or change to force a reset. The dr-provision server must be restarted afterwards.
When making this changes, stop ALL dr-provision servers in the HA cluster. Fix the state files for all servers. Start the selected Active endpoint first. After it is running, start the passive endpoints.