SSL/TLS Certificates are required for all Chef's Server products which present services via HTTPS. Most of our systems will default to using internally-generated self-signed certificates when installed, but most customers use certificates from public vendors or internal PKI infrastructure. We've seen some news being made by Certificate Authorities (CAs) expiring root or intermediate certificates, invalidating all certificates issued by those roots or intermediates.
In Chef Infra Server, you might see messages like this if the chain isn't properly constructed:
[2020-07-09T21:33:03+00:00] ERROR: SSL Error connecting to
SSL_connect returned=1 errno=0 state=error: certificate verify failed
(unable to get local issuer certificate)
|Chef Infra Server||12.x+||Standalone|
Chef products expect certificates to be in PEM format. A certificate chain should look like the following (intermediate certificate isn't required unless your certificate was issued from an intermediate server. Here's an example of what a chain looks like when the issuing server is an intermediate, there's a root CA which signed the issuer's certificate, and the server certificate is first in the chain:
-----BEGIN CERTIFICATE----- <server certificate content> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <intermediate certificate content> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <root certificate content> -----END CERTIFICATE-----
If you use a third party commercial issuer, ensure that whatever they give you can be converted into the proper format. Many commercial issuers build the ability to output properly a formatted PEM certificate chain with the server certificate at the beginning of the file and the root certificate at the end, take advantage of that if you are able.
If you use an internal PKI to issue certificates, make sure the administrators who issue certificates read this and relevant documentation to ensure that they're providing you with the proper information.
Once you have the certificate, ensure that you are properly storing this critical private information. If you use an artifact store or secrets store for provisioning, make sure that the retrieved secrets match what was given to you by the issuer. Testing these stores and processes can be helpful in the event of a need to rebuild your Chef infrastructure for testing or disaster recovery purposes.
Double check your certificates to make sure they are properly formatted and not expired prior to applying them to your systems.
You can use the commands given in the Troubleshooting section of this document to check expiration.
Set a calendar reminder to have your certificates reissued with enough lead time before expiration that you can replace them with little to no interruption in service. Monitoring is good, proactively replacing the certificate so you never get alerted is better.
When applying your certificate to Chef Infra Server, the PEM encoded certificate chain would go into the file you're providing to the
See the server security documentation for Chef Infra Server for more details.
When attempting to diagnose this issue, some things can be helpful.
To check whether the certificate you're using matches the key you're providing in configuration, use openssl like so:
# openssl rsa -noout -modulus -in server.key | openssl md5 # openssl x509 -noout -modulus -in server.crt | openssl md5
If the checksums match, the certificate and key are usable. If they don't check to see whether you're providing the key you used to get the new certificate.
To check whether the certificate is expired, use openssl like so:
openssl x509 -enddate -noout -in file.pem
The output from this command should look something like this:
openssl x509 -enddate -noout -in example.com/fullchain.pem
notAfter=Sep 22 07:48:30 2020 GMT
Note: This will not identify an issue where the root or an intermediate issuer has expired. To get that information, you will need to split the bundle into individual PEM certificate blocks and run the same command against that content.
In most cases, remediation is achieved by following the documentation to install properly constructed certificate chain and corresponding key.
If your certificates are in another format than PEM, you will need to convert them for use by Chef products. To convert a certificate from another format to PEM, use openssl. If you've purchased a certificate from a vendor, they likely have documentation on how to do this addressing their interface. Here are some examples:
PKCS7 (cert.p7, cert.p7a, cert.p7b, cert.p7c) to PEM
# openssl pkcs7 -inform DER -in cert.p7b -print_certs -out cert.pem
PKCS12 to PEM (the resulting bundle will need to be separated out into certificate and key content for Chef products to use them)
openssl pkcs12 -in pkcs-bundle -out pem-bundle
DER (file likely named cert.cer or cert.der) to PEM
if the included keys are DSA:
openssl dsa -inform DER -outform PEM -in cert.cer -out cert.pem
if the included keys are RSA:
openssl rsa -inform DER -outform PEM -in cert.cer -out cert.pem
Further Reading: N/A