dr-provision fails with error 312
Symptom¶
After an upgrade, when rebooting or restarting the dr-provision service, it fails to restart. Looking at the journal logs, you will see something similar to the following. If you are watching the journal, it may be repeating this sequence.
$ journalctl -fu dr-provision
May 04 09:53:51 example.rackn.io (rovision)[61437]: dr-provision.service: Failed to locate executable /usr/local/bin/dr-provision: Permission denied
May 04 09:53:51 example.rackn.io (rovision)[61437]: dr-provision.service: Failed at step EXEC spawning /usr/local/bin/dr-provision: Permission denied
May 04 09:53:51 example.rackn.io systemd[1]: Started dr-provision.service - DigitalRebar Provision Integrated DHCP and File Provisioner.
May 04 09:53:51 example.rackn.io systemd[1]: dr-provision.service: Main process exited, code=exited, status=203/EXEC
May 04 09:53:51 example.rackn.io systemd[1]: dr-provision.service: Failed with result 'exit-code'.
May 04 09:53:51 example.rackn.io systemd[1]: dr-provision.service: Scheduled restart job, restart counter is at 56.
May 04 09:53:51 example.rackn.io systemd[1]: Stopped dr-provision.service - DigitalRebar Provision Integrated DHCP and File Provisioner.
Problem¶
The key stanzas to consider are the following.
and
The problem is that the dr-provison, and possibly the drpcli binaries in your $BIN-DIR (/usr/local/bin by default) are mislabeld with selinux. This is caused by the upgrade process as it copies over drpcli and dr-provision binaries from your $DATA-DIR (/var/lib/dr-provision by default).
You can verify this by checking the binaries.
$ ls -lZ /usr/local/bin/{dr-provision,drpcli}
-rwxr-xr-x. 1 root root system_u:object_r:var_lib_t:s0 18444288 May 2 16:00 /usr/local/bin/drpcli
-rwxr-xr-x. 1 root root system_u:object_r:var_lib_t:s0 102402793 May 2 15:59 /usr/local/bin/dr-provision
Solution¶
This is a problem with SELinux as noted with the exit status of 203. The label is set to system_u:object_r:var_lib_t:s0 on the dr-provision binary in the /usr/local/bin directory. When installed, the label for those binares should be system_u:object_r:bin_t:s0 or unconfined_u:object_r:bin_t:s0. Updating doesn't currently relabel the binaries when moving the files in place.
Relabeling the binaries will correct this problem. The simplest way is to run the following.
$ sudo restorecon -v /usr/local/bin/{dr-provision,drpcli}
Relabeled /usr/local/bin/dr-provision from system_u:object_r:var_lib_t:s0 to system_u:object_r:bin_t:s0
Relabeled /usr/local/bin/drpcli from system_u:object_r:var_lib_t:s0 to system_u:object_r:bin_t:s0
This relabels the binaries to the default label for the directory it's in. You'll note that it tells you what the label was and what it was changed to. It's good to check with the following afterwards to verify the change occured.
$ ls -lZ /usr/local/bin/{dr-provision,drpcli}
-rwxr-xr-x. 1 root root system_u:object_r:bin_t:s0 18444288 May 2 16:00 /usr/local/bin/drpcli
-rwxr-xr-x. 1 root root system_u:object_r:bin_t:s0 102402793 May 2 15:59 /usr/local/bin/dr-provision
It is likely that the dr-provision service has already started, but you can check via journalctl, and by connecting via the UX, CLI or API. If it is still down, consider restarting dr-provision or checking for further errors in the journal.
Additional Information¶
Additional resources and information related to this Knowledge Base article.
See Also¶
- Red Hat's Using SELinux document, specifically Fixing analyzed SELinux denials
Versions¶
All versions of DRP
Keywords¶
dr-provision, selinux, 203