Skip to content

Deployments

Digital Rebar Provision is intended to be deployed as both a DHCP server and a Provisioner. There are cases where one or the other are desired. Each feature can be disabled by command line flags.

  • --disable-dhcp - Turns off the DHCP server
  • --disable-provisioner - Turns off the Provisioner servers (TFTP and HTTP)

The rs_api doesn't change based upon these flags, only the services being provided.

DHCP Disabled

If a DHCP environment already exists or a more declarative mode is more desirable, there are a couple of things in each case to be aware of. For either case, the underlying assumption is that something will direct the node to use the provisioner as its NextBoot server.

Declarative Mode

Each machine must be declared through the rs_cli or the rs_api. The IP address in the machine object will be used to populate the bootenv information. The provisioner will provide these files through TFTP and HTTP. It is left to the admin to figure out how to get the node to reference them.

External DHCP

With DHCP disabled, the admin can provide a DHCP server for distributing addresses. The DHCP will need to do the following:

  • Set NextServer to an IP that routes to Digital Rebar Provision
  • Set Option 3 (gateway) - This is optional, but may be required depending on the network routing.
  • Set Option 6 (dns server) - This is optional, but often useful in conjunction with Option 15.
  • Set Option 15 (dns domain) - This is needed for discovery boots to construct a meaningful FQDN for the node.
  • Set Option 67 (bootfile) - This is required and can be complex, see below.

Setting a bootfile is required. If you have only one architecture and boot mode, this is simply the name of the bootloader. For example, if you are only booting legacy bios x86 systems, then you can set lpxelinux.0 and be done. If you have to support both UEFI and Legacy or multiple architecture types or iPXE as well, you will need a more complex configuration.

For example, this snippet works for most systems when using the ISC DHCP Server. It will set the bootfile for legacy, UEFI, or iPXE booting clients and set the next server parameter to 192.168.100.3. Place this snippet inside a subnet or host definition.

if exists user-class and option user-class = "iPXE" {
filename "default.ipxe";
} else if option arch = 00:07 {
filename "ipxe.efi";
} else if option arch = 00:09 {
filename "ipxe.efi";
} else {
filename "ipxe.pxe";
}
next-server 192.168.100.3;

Secure Boot Support and External DHCP

The primary feature lost when using an external DHCP server is secure boot support. Due to the requirement to control the chain of signed boot components, using an external DHCP server that doesn't know the final destination OS, can't provide a consistent chain of signed bootloaders. Since the DRP DHCP server understands the final destination, it can ensure the components needed to get through discovery, os installation or image deployment, and local disk booting maintain properly signed paths. The biggest issue is that each OS provide uses a different set of signatures that are NOT compatible with each other.

With regard to PXE booting, the process follows something like this:

  1. We are assuming an x86 system with BIOS and a network connection (basically ignoring, arm, apple products, or custom embedded (NUCs - are strange). Your Cisco gear qualifies.
  2. There is a DHCP server and a tftp server in the network.
  3. For fun, we will also include an intermediate router.

In the network we have this:

(server) ---- (router) ----- (router) ---- (drp/dhcp/tftp servers)

All the components are configured except the server which is powered off.

Let us also for the moment assume the SERVER has never had an OS and is from the factory. IPMI and other OOB solutions can get us to this state or start the process, but that can be discussed later.

The customer powers on the SERVER (by pushing the button, ipmi, or whatever).

The first thing the SERVER does is start the POST process. This is loading and running the BIOS.

Most of these types of servers will operate in "Legacy" Or "Uefi"mode. Lenovos even try hybrid mode which is a fallback of the two.

The POST process discovers internal server devices, initializes them, and starts running an "OS", either the legacy BIOS OS (embedded custom by vendor) or UEFI "OS" which is a more standardized composable modular.

In either mode, the boot order of devices is used to figure out where the real OS is loaded from. Both types of booting use different boot order lists. If the system is "fresh" from the factory, the default is usually disk then network devices. Since it doesn't have an OS, it will start a network boot.

The first part of this process is to BROADCAST DHCP messages into the void. The ROUTER picks up the broadcast and forwards it to the configured DHCP server in the ROUTER. This is called DHCP Helper or DHCP/BOOTP helpers on the ROUTER. The DHCP server picks an address and forwards it back to the ROUTER. The ROUTER sends it to the SERVER. This handshake happens twice with the exchange of options that define what is needed by the SERVER to boot. The DHCP server tries to meet those options. This is the traditional DORA DHCP process.

In this case, the DHCP server will tell the SERVER what single file to boot and where to get that file. The protocol is generally so old that the file is transferred through tftp.

The SERVER will configure its networking stack and then try to get that file from the tftp server and boot that file.

In this case, the next OS (bootloader) that it is going to run MUST match the environment it is going to run in. This must be either a legacy-enabled or UEFI-enabled bootloader.

Here is a doc link in RackN docs about this process: https://docs.rackn.io/en/latest/doc/workflows.html#basic-discovery This text is more detailed level, but those pictures describe the process somewhat.

Because the DHCP sends the file to boot, it must match the BOOT type. The DHCP messages contain information (normally, but not always) to indicate which type of system it is. The DHCP can use that to decide what file to send back.

DRP DHCP handles this and chooses to send back an ipxe image. There is an ipxe bootloader for both LEGACY (ipxe.pxe) and UEFI (ipxe.efi). At this point, we are trying to normalize the two paths into a common control path. The ipxe program gets loaded and starts booting.

At this point, the SERVER starts over because it is a "new" OS. It does the DHCP process again, BUT it includes an additional option to indicate that it is an iPXE OS that is booting. The DRP DHCP server sees this and sends the file, " default.ipxe", which contains a script that knows how to get the real next target OS.

The default.ipxe is rendered by DRP to allow chaining scripts into machine-specific bootenvs or the default discovery bootenv. This will then load the appropriate destination OS of sledgehammer in the discovery/unknown device case. Once the last OS is loaded, the booting continues and maybe does DHCP again to get an address, but doesn't boot any files.

This is GREAT until secure boot enters the picture. Once secure boot shows up, the requirement to have signed OS components at all points get challenging. The requirement is that all components be signed by the same signing chain. The original chain uses an RH signed component, a windows signed component, and a final OS signed component. Those signing paths don't always align. The DRP DHCP server is smart enough to know early enough to skip the intermediate steps and stick to a single signed chain of boot loaders. This is why we say you can't do secure boot device management with an external DHCP server. You MUST have a separate orchestrator to toggle the external DHCP server around to make it work and if you are doing that then just use DRP DHCP server.

To keep things simple, DRP DHCP always tries to follow this process. BIOS/UEFI OS -> IPXE OS -> IPXE SCRIPT -> Destination OS. When secure boot is needed, the DRP DHCP server knows to just go directly to the Destination OS.

When using an external DHCP server, the DHCP server MUST be able to handle that difference between the initial boot OS and the iPXE OS to handle Secure Boot.

Provisioner Disabled

In this mode, Digital Rebar Provision acts as a DHCP server only. The dhcp models describe how to use the server. Set the DHCP options that will direct to the next boot servers and other needs.