Automate2/Chef Backend HA/Chef Infra Server 10k returned hits limit

Failure of Chef tools and Automate internal searches

The knife status command and all other searches as implemented today in Automate2/Chef Backend HA/Chef Infra Server will fail after 10k returned hits. Nodes are the object that is most likely to hit this limit. The limit can be changed, at the cost of a less performant overall system. When the limit is being hit, you will see output like this from Elasticsearch in the logs

[2021-09-04T03:26:42,726][DEBUG][o.e.a.s.TransportSearchAction] [somenode.com] All shards failed for phase: [query]
org.elasticsearch.search.query.QueryPhaseExecutionException: Result window is too large, from + size must be less than or equal to: [10000] but was [10479]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.

Change the max window size

To increase the returned hits limit to 50k, do the following on any of the backend nodes. You will only need to run the command on one node and it will affect the whole cluster

curl -XPUT 'localhost:9200/_settings?pretty' -H 'Content-Type: application/json' -d'
{
   "index" : {
   "max_result_window" : 50000
   }
}
'

For Chefs needing more detail, review https://getchef.zendesk.com/agent/tickets/15355

Elastics's discussion: https://discuss.elastic.co/t/how-to-increase-the-default-size-limit-from-10000-to-1000000-in-elasticsearch/208807

Feature description: https://chef-software.ideas.aha.io/ideas/AUTO-I-81