I Upgraded The Linux Kernel and Chef Server Stopped Working

Sean Horn -

If this is an issue for you, the likely explanation is that a kernel upgrade was made without a reinstall(rebuild) of the drbd kernel module. This results in /proc/drbd going missing and no device files found in /dev, like /dev/drbdX

Now that the new kernel is in place, you should uninstall the packages listed and reinstall them, reinstalling the repo key and ELRepo for RHEL if necessary as shown at https://github.com/chef/chef-server/blob/11.1.4/files/private-chef-cookbooks/private-chef/recipes/drbd.rb#L53-L56

Once you do this, run private-chef-ctl or chef-server-ctl reconfigure to link the standard drbd configuration into our custom Chef Server setup.

Once the “drbd service” startup finishes, you should have a working DRBD device once again.

Takeaways:

  • Patches and other upgrades should be tested in a working copy of the production system being changed.
  • Kernel upgrades disable DRBD and other module installs
  • We encourage everyone to disallow automatic kernel updates and updates of the drbd kernel module and the drbd tools as part of standard upgrades on Chef Server systems with DRBD.
Have more questions? Submit a request

Comments

Powered by Zendesk