Summary

SSL/TLS Certificates are required for all Chef's Server products which present services via HTTPS. Most of our systems will default to using internally-generated self-signed certificates when installed, but most customers use certificates from public vendors or internal PKI infrastructure. We've seen some news being made by Certificate Authorities (CAs) expiring root or intermediate certificates, invalidating all certificates issued by those roots or intermediates.

In Automate, you might see services showing CRITICAL status, and messages such as this in journalctl when a certificate has expired:

upstream SSL certificate verify error: (10:certificate has expired)
 while SSL handshaking to upstream

If the issue is a mismatch between the certificate and the key, Automate will log something similar to this:

Jul 09 21:33:03 automate hab[2932]:
 automate-load-balancer.default(O): 2020/07/09 21:33:03 [emerg]
 6944#0: SSL_CTX_use_PrivateKey("/hab/svc/automate-load-balancer
/data/automate.example.com.key") failed (SSL: error:0B080074:x509 
certificate routines:X509_check_private_key:key values mismatch)

Distribution

Product	Version	Topology
Chef Automate	2+	Standalone
Chef Infra Client	15.x+	N/A
Chef Habitat	0.81+	N/A

Process

Plan

Preparation:

Chef products expect certificates to be in PEM format. A certificate chain should look like the following (intermediate certificate isn't required unless your certificate was issued from an intermediate server. Here's an example of what a chain looks like when the issuing server is an intermediate, there's a root CA which signed the issuer's certificate, and the server certificate is first in the chain:

If you use a third party commercial issuer, ensure that whatever they give you can be converted into the proper format. Many commercial issuers build the ability to output properly a formatted PEM certificate chain with the server certificate at the beginning of the file and the root certificate at the end, take advantage of that if you are able.

If you use an internal PKI to issue certificates, make sure the administrators who issue certificates read this and relevant documentation to ensure that they're providing you with the proper information.

Design:

Once you have the certificate, ensure that you are properly storing this critical private information. If you use an artifact store or secrets store for provisioning, make sure that the retrieved secrets match what was given to you by the issuer. Testing these stores and processes can be helpful in the event of a need to rebuild your Chef infrastructure for testing or disaster recovery purposes.

Configure

Evaluation:

Double check your certificates to make sure they are properly formatted and not expired prior to applying them to your systems.

You can use the commands given in the Troubleshooting section of this document to check expiration.

Application:

Set a calendar reminder to have your certificates reissued with enough lead time before expiration that you can replace them with little to no interruption in service. Monitoring is good, proactively replacing the certificate so you never get alerted is better.

When applying your certificate to Chef Automate, the entire PEM encoded block would go in between the sets of three double quotes in the TOML file you're using to patch the config, as shown here:

[[global.v1.frontend_tls]]
# The TLS certificate for the load balancer frontend.
cert = """-----BEGIN CERTIFICATE-----
<your certificate>
-----END CERTIFICATE-----
"""

# The TLS RSA key for the load balancer frontend.
key = """-----BEGIN RSA PRIVATE KEY-----
<your private key>
-----END RSA PRIVATE KEY-----
"""

Troubleshoot

Analysis:

When attempting to diagnose this issue, some things can be helpful.

To check whether the certificate you're using matches the key you're providing in configuration, use openssl like so:

# openssl rsa -noout -modulus -in server.key | openssl md5
 # openssl x509 -noout -modulus -in server.crt | openssl md5

If the checksums match, the certificate and key are usable. If they don't check to see whether you're providing the key you used to get the new certificate.

To check whether the certificate is expired, use openssl like so:

openssl x509 -enddate -noout -in file.pem

The output from this command should look something like this:

openssl x509 -enddate -noout -in example.com/fullchain.pem
notAfter=Sep 22 07:48:30 2020 GMT

Note: This will not identify an issue where the root or an intermediate issuer has expired. To get that information, you will need to split the bundle into individual PEM certificate blocks and run the same command against that content.

Remediation:

In most cases, remediation is achieved by following the documentation to install properly constructed certificate chain and corresponding key.

If your certificates are in another format than PEM, you will need to convert them for use by Chef products. To convert a certificate from another format to PEM, use openssl. If you've purchased a certificate from a vendor, they likely have documentation on how to do this addressing their interface. Here are some examples:

PKCS7 (cert.p7, cert.p7a, cert.p7b, cert.p7c) to PEM

# openssl pkcs7 -inform DER -in cert.p7b -print_certs -out cert.pem

PKCS12 to PEM (the resulting bundle will need to be separated out into certificate and key content for Chef products to use them)

openssl pkcs12 -in pkcs-bundle -out pem-bundle

DER (file likely named cert.cer or cert.der) to PEM

if the included keys are DSA:

openssl dsa -inform DER -outform PEM -in cert.cer -out cert.pem

if the included keys are RSA:

openssl rsa -inform DER -outform PEM -in cert.cer -out cert.pem

Appendix

How to configure and troubleshoot issues with Chef Automate SSL/TLS certificates