Azure VM Extension Chef Infra Client fails to run (version 1210.13.3.1 onwards) – Chef Operations and Troubleshooting Guide

Issue

When installing Chef via the Azure VM extension, Chef installs begin to fail. The logs for the install (either in your CI/CD pipeline or via the Azure console) appear to indicate there was a failure during installation of the extension:

ERROR ExtHandler ExtHandler Event: name=Chef.Bootstrap.WindowsAzure.LinuxChefClient, op=Install, message=eJy9kLFOwzAURfd8hT8A23FDSpMNqm4IJBaGKqqc+CW1FN6L7Gca9esJXWBgQSpsV7pXOjp3v5sZMHrC5wmC5SXsQqDQiCdCeYZAAmbPoiMHtTA3Qr/boEff6pO1AyDr7RF69UDEkYOd1KtHR6d4f04B1KPHNH8OtqNfttKsTK5MoQpltMfIdhxVPGb7yI4SN1l2iRBCk70kRI+DiB1NIGwUCT3X4he0wzfaoXRr11e2kLDpCnm7zkFWpTESbNtuNlV7V1ZWXVjZ1Qxbj7pbKvmlulxYX+/CHwEDvNUCiUVPCd0f26zyf9b5AJ9w6+o=, duration=0

The 'duration=0' and 'op=Install' suggests that the installer has encountered an issue.

Environment

Chef Infra Client 15 onwards when installed using the Azure VM Extension for Chef.

NOTE: 1210.13.3.1 only supports Chef-15, you cannot specify Chef-16 - CHANGELOG.md

Cause

There are a few possible ways that an Azure VM extension could encounter an error. The primary cause in the scenario whereby you are upgrading to version 1210.13.3.1 is that whilst Chef Client was able to install, it was unable to configure due to a silent license acceptance failure due to having switched latest client version to Chef Client 15:

Azure VM Chef Extension version : 1210.13.2.3 
Chef Client version : 14.15.6

Azure VM Chef Extension version : 1210.13.3.1 
Chef Client version : 15.11.8

You can confirm that your deployment is using a the above latest available version via the Azure CLI:

az vm extension image list-versions -l northeurope -p Chef.Bootstrap.WindowsAzure -n ChefClient  {
    "computeRole": null,
    "handlerSchema": null,
    "id": "/Subscriptions/e6b872d2-4d5a-42aa-9ac9-8f5e03f556dc/Providers/Microsoft.Compute/Locations/northeurope/Publishers/Chef.Bootstrap.WindowsAzure/ArtifactTypes/VMExtension/Types/ChefClient/Versions/1210.13.2.3",
    "location": "northeurope",
    "name": "1210.13.2.3",
    "operatingSystem": null,
    "supportsMultipleExtensions": null,
    "tags": null,
    "type": null,
    "vmScaleSetEnabled": null
  },
  {
    "computeRole": null,
    "handlerSchema": null,
    "id": "/Subscriptions/e6b872d2-4d5a-42aa-9ac9-8f5e03f556dc/Providers/Microsoft.Compute/Locations/northeurope/Publishers/Chef.Bootstrap.WindowsAzure/ArtifactTypes/VMExtension/Types/ChefClient/Versions/1210.13.3.1",
    "location": "northeurope",
    "name": "1210.13.3.1",
    "operatingSystem": null,
    "supportsMultipleExtensions": null,
    "tags": null,
    "type": null,
    "vmScaleSetEnabled": null
  }

It is also likely that any misconfiguration of the client.rb file or incorrect acceptance of the license (if pinning to a Chef Client version of 15 or later) could also be culprits here.

Resolution

As the underlying issue pertains to a combination of defaults in the Azure VM Chef Extension and Chef Client version being set in the configuration, there are a few possible approaches to mitigating this:

Pin the Azure Chef Extension version (this will be specific to how you automate your deployment, see MS Azure chef extension schema):

{
  "type": "Microsoft.Compute/virtualMachines/extensions",
  "name": "[concat(variables('vmName'),'/', parameters('chef_vm_extension_type'))]",
  "apiVersion": "2017-12-01",
  "location": "[parameters('location')]",
  "dependsOn": [
    "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'))]"
  ],
  "properties": {
    "autoUpgradeMinorVersion": false,
    "publisher": "Chef.Bootstrap.WindowsAzure",
    "type": "[parameters('chef_vm_extension_type')]",
    "typeHandlerVersion": "1210.13",
    "settings": {
      "bootstrap_version": "[parameters('bootstrap_version')]",
      "bootstrap_options": {
        "chef_server_url": "[parameters('chef_server_url')]",
        "validation_client_name": "[parameters('chef_validation_client_name')]"
      },
      "CHEF_LICENSE" : "accept-no-persist",
      "runlist": "[parameters('chef_runlist')]"
    },
    "protectedSettings": {
      "validation_key": "[parameters('chef_validation_key')]"
    }
  }
}

It's possible (depending on region/version) that if you require minor/patch pinning in your ARM template that this will fail to pass the syntax-checking in ARM and you will only be able to pass MAJOR version:

"typeHandlerVersion": "1210.13.3.1" - possible fail

"typeHandlerVersion": "1210.13" - always accepted

If you deploy via the Azure CLI you'll want to pass the --version flag on the command line like so:

az vm extension set \
 --resource-group myResourceGroup \
 --vm-name myExistingVM \
 --name LinuxChefClient \
 --publisher Chef.Bootstrap.WindowsAzure \
 --version 1210.13 --protected-settings '{"validation_key": "<validation_key>"}' \
 --settings '{ "bootstrap_options": { "chef_server_url": "<chef_server_url>", "validation_client_name": "<validation_client_name>" }, "runlist": "<run_list>" }'

Pin the bootstrap_version specifically to Chef Client 14 as per Azure extension publicconfig.json
- ```
"bootstrap_version": "<version of chef-client>",
```
Set the license acceptance as per the Azure extension publicconfig.json (see Azure Chef Extension usage):
- ```
"CHEF_LICENSE" : "accept-no-persist"
```

Set license acceptance through the ARM Command (see ARM commands for Azure Chef Extension):

CHEF_LICENSE: Affected product versions which require accepting the CHEF EULA license (requires chef 15 + ). Set `CHEF_LICENSE` with either of these values `accept`, `accept-silent` or `accept-no-persist`. Refer to [CHEF EULA license] (https://docs.chef.io/chef_license_accept/#accept-the-chef-eula)

No matter which of the above you choose to do, we'd also like to make you aware that it is likely that you will need to test/refactor cookbooks should you choose to roll forward to Chef 15.

Upgrading Chef Client across major releases (13, 14 ,15, 16)