Summary

This article describes necessary system tunings and checks to ensure your Chef Automate instance is performant, and able to receive configuration changes. This is essential for general operability as well as recovering from certain failure states.

Distribution

Product	Version	Topology
Chef Automate	2	All

Process

Plan

Preparation: Evaluate your host instance(s) sizing needs appropriately according to published minimum system requirements as well as our scaling documentation ( Chef Automate: Deployment Planning and Performance tuning (transcribed from 'Scaling Chef Automate Beyond 100,000 nodes') ).

Configure the appropriate monitoring and alerting on thresholds which will provide ample time to request more disk space and/or rotate logs. Also be considerate as to whether adequate local disk space for data migrations during upgrades is in place. We recommend that you should have an excess of anything up to 50% free disk if you have a remote share/location to which back-ups are directed or 70% free disk if a local backup location is part of the implementation.

Design: If needs seem greater than a standalone Chef Automate instance can sustain, multi-node topologies need also be considered. Contact your account team for further information deploying a clustered instance of Chef Automate.

Configure

Evaluation: N/A

Application: Very importantly, ensure that the following system tunings are in place:

# sysctl vm.max_map_count
vm.max_map_count = 262144
# sysctl vm.dirty_expire_centisecs
vm.dirty_expire_centisecs = 20000

If they are not, set them:

# sysctl -w vm.max_map_count=262144
# sysctl -w vm.dirty_expire_centisecs=20000

And add them to /etc/sysctl.conf so that they persist through system reboots:

# tail -n 2 /etc/sysctl.conf
vm.max_map_count=262144
vm.dirty_expire_centisecs=20000

Troubleshoot

Analysis: If you are still seeing errors or cannot apply new Automate configuration, you may need to also increase the limit of file descriptors available to the service.

Remediation: Open up the file handle limit for Automate services:

# systemctl stop chef-automate
# mkdir -p /etc/systemd/system/chef-automate.service.d
# cat <<EOF >> /etc/systemd/system/chef-automate.service.d/custom.conf
[Service]
LimitNOFILE = 128000
EOF
# systemctl daemon-reload
# systemctl start chef-automate

Appendix

Chef Automate services down, Elasticsearch and Compliance service in CRITICAL state, web UI and client runs failing with 5xx errors due to insufficient Elasticsearch heap size

Essential System Tunings and Prerequisite Checks for Chef Automate 2