Skip to content

Multi-API OOB/BMC Control

The ipmi plugin adds the ability to do multi-vendor out-of band management (e.g. IPMI, Redfish, iDRAC, ILO, etc) actions on machines. Additionally, the plugin provides a stage, tasks, and parameters to facilitate the configuration of the BMC as part of a workflow. The plugin can also create machines from their BMC presence if the feature is enabled.

Training Video: https://www.youtube-nocookie.com/embed/Zs6KvwBvZYQ

!!! note To use this plugin, ipmitool needs to be installed on the DRP endpoint. The plugin will fail to load/configure if missing.

The following actions have become a loose API across all IPMI plugins.

  • Machine Actions:

    • poweron
    • poweroff
    • powercycle
    • nextbootpxe
    • nextbootdisk
    • identify - not implemented by all
    • powerstatus - not implemented by all

  • Plugin Actions:

    • range-scan

Each of these will be described more below. The actions trigger through the normal API conventions for machine actions. API access is through the /api/v3/machines/UUID/actions/ACTION POST endpoint. The drpcli actions call drpcli machines runaction UUID ACTION. Or UX as them on a dropdown in the Bulk Action section. See the other sections of documentation for action information.

The configuration / discovery stage, ipmi-configure, can be added to a workflow to at a minimum discover the machine's BMC address, but additionally configure the BMC's networking and a single access user. There are parameters to describe this behavior.

Each of the methods will be described below.

Configuration / Discovery

The ipmi-configure stage when applied through a workflow can configure the BMC's network and configure a user for out-of-band management operations. These configuration operations can be toggled on or off depending upon your environment's needs. The user will be configured first and then the network configured.

The ipmi-configure stage will take no actions by default. The stage has three task elements. The first is a task that will make sure that the ipmitool is available and install it if missing. This is included in sledgehammer by default and no action is required. The second task element is a quirks JSON file. This defines unique variations required to address the variances in BMCs by hardware type. In general, this file does not need to be altered by the user. If a BMC misbehaves, notify RackN with the problems. The third task element is the action configuration script itself. This does the actual work and processes the quirks file as needed.

The ipmi-configure stage can be told to fail if no BMC is detected. This is help if a BMC is broken or non-operative. The ipmi/configure/no-bmc-fatal boolean set to true will cause failure in this case.

Regardless of what configuration parts the stage is allowed to do, the stage will always record the current IPMI network address associated with the BMC and store that in the ipmi/address parameter for the operational actions.

User Configuration

The ipmi-configure stage will by default take no configuration actions. To enable user configuration, the ipmi/configure/user boolean will need to be set to true. This can be done globally, in a profile attached to a machine, or directly on the machine's parameters.

First, the ipmi-configure task will check to see if the requested username, ipmi/configure/username, is already defined. If this is present, the userid is recorded. If the userid is not found, the ipmi-configure task will check the ipmi/configure/userid parameter to see if a specific BMC numeric userid should be used for this control user. If the userid is still not specified, a free userid will be found to user. If no userids are found, the first user id will be overwritten.

Additional checks are performed to see if the first id (2) is editable. This is not the case on some BMCs.

Once a userid is selected or discovered, the username, password, privileges, and remote access are enabled for that user. Once all those are success at the end of the task, the username and password are recorded in the ipmi/username and ipmi/password fields respectively.

Network Configuration

The ipmi-configure stage will by default take no configuration actions. To enable network configuration, the ipmi/configure/network boolean will need to be set to true. This can be done globally, in a profile attached to a machine, or directly on the machine's parameters.

Once set to enabled, the first thing will be for Dell-based BMCs to have whether they are shared or dedicated set.

The task will then check to see if specific address is requested by seeing if the ipmi/configure/address field is set. If so and the ipmi/configure/reserve-ip is set to true, the task will create a Reservation for that IP address with the BMC's MAC address in the DRP DHCP server. The task will fail if requested to reserve an address with no address specified and set to static mode. This means that DRP is an IPAM and that is not currently supported.

With the reservation in place if requested, the stage will run use the ipmi/configure/ip-mode to set which mode the BMC should operate in. The options are dhcp or static. dhcp mode will enable the BMC to get an address through DHCP. This address will get recorded into the ipmi/address parameter.

If the mode is static, the ipmi/configure/address, ipmi/configure/netmask, and ipmi/configure/gateway parameters are used to set the BMC's network information.

Upon completion of everything, the current IP address will be recorded in the ipmi/address parameters and a DRP DHCP reservation will be created if the address was not specified in the ipmi/configure/address parameter and DHCP mode was set.

The overall flow of network configuration is to allow for static configuration (static mode), DHCP with an existing reservation in the DHCP server (reserve-ip set to false and dhcp mode), DHCP with a known address that becomes a static DHCP address (reserve-ip set to true, address specified in ipmi/configure/address, and dhcp mode), and DHCP without a known address that becomes statically bound in the DHCP server (reserve-ip set to true, address NOT specified in ipmi/configure/address, and dhcp mode).

Machine Creation

The goal of the DRP feature is to create machines via their BMC presence. DRP determines the presence of machines through events generated by the server or a manual range scan. It then logs in using default or user-provided credentials to determine access and creates a machine object.

Enabling Machine Discovery and Creation

In order to enable machine discovery and creation, the ipmi/discover/enabled parameter must be set to true.

Machine Discovery from Lease Objects

Autodiscovery today is done from lease events that are generated by the DRP server when a new DHCP lease is generated.

Setup

In order to enable auto-discovery of OOBM controllers, the ipmi/discover/enabled and the ipmi/discover/from-lease params should be set to true.

In addition to this you will need to provide IPMI credentials which can be used to login to these controllers. This can be done in two ways:

  1. Using the ipmi/discover/oobm-default-credentials param which is a list of default credentials which will be used for all controllers
  2. Using the ipmi/discover/oobm-user-provided-credentials which are credentials specific to a machine. The list uses an identifier per machine to determine what machine to try the credentials with. This identifier can be an IP address or the MAC address of the machine.

Optionally, you can also define the following params for further configuration:

  • ipmi/discover/parallel: The number of events that can be processed in parallel. Defaults to 200.

  • ipmi/discover/buffer-size: The number of events that are held in the queue at a given time. Defaults to 20000. If the number of events exceeds the queue size, they will be lost.

  • ipmi/discover/workflow: The workflow that needs to be set on the machine once it has been created. It defaults to universal-discover.

(Process)[auto-discover-process]

The machine discovery and creation process consists of the following steps:

  • Ensure that a machine with the same IP address or MAC address does not already exist. If a machine is found, the process stops.
  • Ensure that the Redfish API is up and running and that we are able to login using the machine specific or the default credentials. We are unable to proceed without a successful login/connection.
  • Retrieve necessary information from the controller using redfish APIs.
  • Create a machine object with the retrieved information, IPMI information and the specified/default workflow.

Note

If any errors occur during this process, an error message is logged, and the process stops.

Range Scan

The setup is identical to the auto-discovery set up except the ipmi/discover/from-lease param does not need to be set.

Process
  • Ensure the IP addresses are in the right format
  • Check that the start IP is lesser than the end IP
  • Iterate over the list of IPs and for each IP follow the same process as auto discovery process

To perform a manual range scan, use drpcli to run an action against the ipmi plugin.

drpcli plugins runaction ipmi scan-range ipmi/discover/scan-start-address "1.1.1.1" ipmi/discover/scan-end-address "2.2.2.2"

Note

Finalization

Once configuration is finished, some BMCs need to be reset to take effect. This can take up to a minute in some cases.

Actions

Actions are enabled on the machines by have three parameters set. Actions will not be visible upon the machines if not present. The following parameters need to be set somehow. If the ipmi-configure stage is NOT used, the IPMI actions can still function if the following three parameters are applied to the machine in some manner.

Parameters:

* `ipmi/address` - The IP address to access the BMC
* `ipmi/username` - The username to initiate BMC actions
* `ipmi/password` - The password to initiate BMC actions

Optionally, the network Port to connect to can be defined if the specific ipmi/mode driver type has been configured to use non-standard Port numbers for the given protocol. For example, if the IPMI protocol has been relocated from default UDP port 623 to 1623, use the appropriate optional Param setting below. The options are as follows:

  • (optional) ipmi/port-ipmitool - sets the network Port for the IPMI protocol
  • (optional) ipmi/port-racadm - sets the network Port for the iDRAC RACADM commands
  • (optional) ipmi/port-redfish - sets the network Port for the Redfish API service

The configuration parts of the ipmi-configure stage will set all three if enabled. Without configuration enabled, the ipmi/address field will be populated regardless if the ipmi-configure stage is applied. This enables discovery of BMC addresses for existing or already configured systems.

The ipmi/address parameter should be unique to the machine and specified directly in the machine's parameters. The ipmi/username and ipmi/password fields may be specified directly in the machine's parameters, in a profile attached to the machine, or global profile depending upon the scope of the username and password in the environment.

Using those three parameters, the actions will use ipmitool do the following actions. The administrator will need to make sure that the DRP Endpoint can access the BMC addresses.

All actions will return a 200 HTTP return code and a string of the ipmitool output. Upon failure, a 409 HTTP return code is returned and the error output of the ipmitool command.

powerstatus

This will issue chassis power status and return the results as a string.

poweron

This will issue chassis power on and return the results as a string.

poweroff

This will issue chassis power off and return the results as a string.

powercycle

This will issue chassis power cycle and return the results as a string.

nextbootpxe

This will issue chassis bootdev pxe and return the results as a string. This will be a single next boot action to boot from the network. On most systems, this is not persistent.

nextbootdisk

This will issue chassis bootdev disk and return the results as a string. This will be a single next boot action to boot from the disk. On most systems, this is not persistent.

identify

This will issue chassis identify number and return the results as a string. The number is specified by the ipmi/identify-duration parameter and defaults to 60 (seconds).

range-scan

The action allows you to scan a range of IPs to detect BMC presence by setting the ipmi/discover/scan-start-address and ipmi/discover/scan-end-address params. If a BMC is detected, DRP attempts to log in using default or user-provided credentials. If successful, a machine object will be created. Also see machine discovery

Potential Errors

In configuring various ipmi/mode types, you may receive errors when Actions are performed. Some of those errors may include the following, along with potential reasons for the error condition.

* `(404) plugin: ipmi out: Error: Unable to establish IPMI v2 / RMCP+ session`

    Typically this error may be encountered, the IPMI protocol mode has not been enabled on the Baseboard Management Controller (BMC; Dell iDRAC, HPE iLO, etc.).