Chef Server 413 opscode-erchef max_request_size and Automate

Problem

In all versions of Chef Server, and especially those that support proxying data-collector data reports to an Automate system, the following condition will happen in the Chef Client log file when Chef Clients try to send their (too large) node and compliance data back through to the Chef Server / Automate system:

413 "Request Entity Too Large"

You may see the same issue in Automate when a message like this is occurring in the Automate logs. This means that a message was sent to Automate that was larger than 4MB.

rpc error: code = ResourceExhausted desc = grpc: received message larger than max

Solution

To increase this limit on the Chef Server side, set the following in /etc/opscode/chef-server.rb

opscode_erchef['max_request_size'] = 3000000

This will set a max request size of 3MB up from either 1MB or 2MB depending on the version of Chef Server. And then reconfigure with:

chef-server-ctl reconfigure

Next, try your Chef Client run again. If it does not work, do not continually raise this limit if you do not know why you are doing it. The ultimate limit for a GRPC message, beyond which Automate will no longer accept reports, is 4000000 bytes. The most common causes of this error follow:

In Chef Client < 15.x, the :Sessions and :Passwd plugins should be disabled if you find that your node sizes are above 1MB on average. Also, consider :Filesystem and :Filesystem2
Chef Client is running the audit cookbook and a profile has gathered and sent enough data during the profile run reporting phase that it has overwhelmed the opscode_erchef['max_request_size']

Chef Server is not a bulk storage system. Once you are moving gigabytes of node data around, things are going to become very slow. It's better to understand why the overload is happening than to continually raise the limits without knowing where the size is coming from.

You can find out how large your nodes are with this:

knife node show NODE_NAME -F json -l | wc -c
113999

This example node is about 114KB.

To disable the plugins, use the following in your /etc/chef/client.rb and see Ohai settings in client.rb for how to tell whether the change took effect. If you are running Docker on this host, take a look at disabling the :Filesystem and :Filesystem2 plugins as well by adding them to the list.

ohai.disabled_plugins = [ :Passwd, :Session ]

You can see how large the nodes being written to or requested from your chef server are in /var/log/opscode/nginx/access.log -- runner-2-432.lxc here for example was about 82956 bytes:

10.0.3.82 - - [2019-07-02T15:47:41+00:00] "PUT /organizations/delivery/nodes/runner-2-432.lxc HTTP/1.1" 200 "0.042" 17673 "-" "Chef Client/14.2.0 (ruby-2.5.1-
p57; ohai-14.2.0; x86_64-linux; +https://chef.io)" "127.0.0.1:8000" "200" "0.041" "14.2.0" "algorithm=sha1;version=1.1;" "runner-2-432.lxc" "2019-07-02T15:47:4
1Z" "UX/quhGheY4AeHrqg0eJLZ406VM=" 82956 "1213ecfd-9926-4881-ba35-0192dce07df7"

And here is an example of a data-collector call (a report from a completed, successful Automate compliance profile being sent through the audit cookbook's reporter handler). The request size turned out to be around 84899 bytes:

10.0.3.82 - - [2019-07-02T15:03:52+00:00] "POST /organizations/delivery/data-collector HTTP/1.1" 204 "0.008" 0 "-" "Chef Client/14.2.0 (ruby-2.5.1-p57; ohai-1
4.2.0; x86_64-linux; +https://chef.io)" "10.0.3.231:443" "204" "0.003" "14.2.0" "algorithm=sha1;version=1.1;" "runner-2-432.lxc" "2019-07-02T15:03:52Z" "u//w8J
yz8uT8avsl8g29Jszw+TI=" 84899 "efe553ad-17ec-4147-ac8b-6d4dd18279f0"

Problem

Solution

See also

Comments