When using Chefs public artifact/package services it is possible that you may either experience difficulty reaching these due to the configuration of the networking/firewalling of your organizations environment or that some anomalous behaviour sourcing from the public IP space connecting to chef services has triggered throttling/halting package downloads. The latter can be observed as a HTTP 405 and abrupt halting of data transmission:
Exception: 405 "Send an email to email@example.com to request removal from
our block list.
|Chef Infra Client||14.x+||N/A|
|Chef Infra Server||12.x||Standalone|
When using Chef in an air-gapped/restricted/private environment it will be necessary to configure components to provide access to chef software packages/artifacts. Depending on the requirements of your organisation you may only require internet access to reach Chef's package/artifacts repositories.
If you are using internet access and chef services we would advise reaching out to your network/security team to make sure a request is made for direct/proxied access from the internal networks on which target nodes reside to the following addresses (as you need them):
- Omnitruck (https://omnitruck.chef.io/): provides the default install script for bootstrapping chef-clients
- Downloads (https://downloads.chef.io/): provides packages to the various installation methods used to bootstrap chef.
- Packages (https://packages.chef.io/ - redirects to https://downloads.chef.io/) see above
The content associated with these URLs is cached with our CDN vendor Fastly. You can see which IP's/ranges will need to be whitelisted on your firewalls/network edge here https://api.fastly.com/public-ip-list.
Its also plausible that your organisation's security team may monitor inbound/outbound downloading using IDS/IPS (Intrusion detection/prevention system) and halt the further download from outside the network. Ensure thats not the case by making your security team aware that node(s) will need to reach out to the aforementioned destinations and they should be exceptions if there is no intention to maintain/host chef packages internally.
Should you need to mitigate against the failure of your ISP you may also need to ask you Unix and Infrastructure teams for a distribution point which mirrors Chef's package/artifacts repositories in functionality and, in the event of an internet outage, will ensure business continuity through the availability of Chef software in your environment. We would suggest that the design encompass HA/DR plans and that all necessary artifacts are routinely synchronized to this distribution point.
Once you have the necessary infrastructure components you can/should load test these if you are in any doubt about their ability to handle large, parallel downloads from your fleet. If you have an existing internal distribution point we would suggest that you ensure the additional load you place on this is insufficient to impact other services.
Bootstrapping target nodes is beyond the scope of this article. Please see further reading for documentation pertaining to bootstrapping from an air gapped/internal environment.
Ensure that the DNS entry is resolvable for whichever URL you are attempting to reach and the IP is routable with ping/lookup tests. If you cannot reach the public URL's of Chef services we recommend revising the planning phase of this article and ensuring the connectivity is established and in place from the target node and that no IPS/IDS is potentially augmenting the download behaviour at the network edge.
It is possible that due to perceived abuse of our public facing services from node(s) within your internal network that we may have blocked the public IP behind which the node(s) source and are NAT'd to.
To ascertain the Public IP for your organization please run the following from the failing target node:
curl -s -k https://ifconfig.co
please send an email to firstname.lastname@example.org stating the public IP/range as allocated to the NAT pool (your networking team can retrieve this for you if you do not have access) and we can check and confirm if it has been blocked on our end. The operations team can subsequently unblock this once we have verified that the behaviour has since ceased.