I try to search on my Chef Infra Server 14.x system, but some searches work, and some don't that should, and others return only partial results.
I look at the output of curl -sS localhost:9200/_cat/indices?v and
notice that the "docs.count" seems low at 2310. I have ~4000 nodes and
~4000 clients. Each bootstrapped node gets its own client, so I should
have at least 8000 items indexed, and probably a few more for other
object types like roles and environments, data bags, and so on.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open chef ga39zu9pSsGsLRDAXCgygg 5 1 2310 0 228.4mb 228.4mb
I already tried to reindex with chef-server-ctl reindex ORGNAME -d
-t, but still some searches that should give results do not, or give
less results than they should. And if I look at the "docs.count" after
the reindex, I still see 2310.
After running the reindex, I saw logging like this in the
/var/log/opscode/opscode-erchef/erchef.log that seem to indicate that
a reindex happened, but Elasticsearch has not changed.
2021-07-22 22:12:19.026 [info] reindexing[delivery]: reindex requested for delivery
2021-07-22 22:12:19.027 [info] reindexing[delivery]: 4000 items of type node to reindex
2021-07-22 22:12:19.111 [info] reindexing[delivery]: 1 items of type role to reindex
2021-07-22 22:12:19.120 [info] reindexing[delivery]: 1 items of type environment to reindex
2021-07-22 22:12:19.130 [info] reindexing[delivery]: 4000 items of type client to reindex
2021-07-22 22:12:19.140 [info] reindexing[delivery]: reindex complete!
A read-only flag having been set manually or automatically is often
the culprit for this behavior.
If the read-only flag has been set by running out of disk space, it is
persistent until it is removed by a human. More on that at Chef Automate Disk Full. The "read_only_allow_delete" flag persists and will not be
automatically deleted, even if the disk space is once again
adequate. The flag must be deleted manually. Afterwards, a reindex
will proceed and complete.
Once we have verified that there is room in /var/opt/opscode, we can
go ahead and remove the flag for all indices, like this. There is only one on a Chef Infra Server, so if you need to, substitute the word "chef" for "_all" below
curl -X PUT "localhost:10141/_all/_settings" -H 'Content-Type: application/json' -d'
Afterwards, a reindex should be able to succeed if there are no other
issues with Elasticsearch.