Summary
In some configurations a Chef Automate-deployed Chef Server may
not be able to forward data collector traffic. Specifically if
you:
- Deployed Chef Server using Automate,
- Configured an "External Automate" server via the
`erchef.v1.sys.external_automate` or
`global.v1.external.automate` configuration options, and
- The external Automate server is using a TLS certificate that,
(1) includes an intermediate certificate,
(2) does not correctly match the hostname of the machine, or
(3) is a wildcard certificate
Then you may not be able to forward traffic from your chef-server
to data-collector after upgrade.
The symptoms manifest themselves in the Chef Infra Server's journalctl logs:
Aug 17 12:10:40 chef-server hab: automate-load-balancer.default(O): - [17/Aug/2020:17:10:40 +0000] "POST /organizations/chef-test/data-collector HTTP/1.1" 502 "0.013" 170 "-" "Chef Client/15.12.22 (ruby-2.6.6-p146; ohai-15.12.0; x86_64-linux; +https://chef.io)" "10.10.10.10:10200" "502" "0.012" 1526
Aug 17 12:10:42 chef-server hab: automate-cs-nginx.default(O): 2020/08/17 17:10:42 [error] 27403#0: *326 upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream, client: 10.200.70.33, server: , request: "POST /organizations/chef-test/data-collector HTTP/1.1", upstream: "https://10.10.10.10:443/data-collector/v0/", host: "chef-server.chef.io:443"
Distribution
| Product | Version | Topology |
| Chef Automate | 2020081214709 | Standalone |
| Chef Infra Server | 14.0.22 | Standalone (Automate Deployed) |
Process
Plan
Preparation: N/A
Design: N/A
Configure
Evaluation: N/A
Application: N/A
Troubleshoot
Analysis:
If you are unable to forward traffic and have deployed Infra Server using Automate 2020081214709 then this should suffice as notice to take the remediation steps.
Remediation:
The next release of Automate should patch this bug and will be made available as soon as possible. If deployed as part of an air-gapped environment then you should look to perform an air gap upgrade with https://automate.chef.io/docs/cli-chef-automate/#chef-automate-upgrade once a release later than that currently deployed is available. If you haven't disabled auto-updates then expect your system to update to the newest release in due course (at the time of writing the engineering team are prioritising a fix to be merged - https://github.com/chef/automate/pull/4267) .
If you are not deployed in production and need a fix to continue your testing then we recommend patching your configuration using the below contents in a patch.toml file to disable ssl_verification:
cs_nginx]
[cs_nginx.v1]
[cs_nginx.v1.sys]
ssl_verify = "off"
see https://automate.chef.io/docs/cli-chef-automate/#chef-automate-config-patch to apply.