Compare Database and Search Index IDs

Sean Horn -

A known issue with Chef Server for all install types is that knife can become stuck in a search loop when the entries in the postgres database do not agree with the equivalent entries in the Solr or Elasticsearch search indexes.

An example of this issue would be the following appearing in the output of a run of `knife status -VV` against a server displaying this issue

shorn:14916 sean$ knife status -VV | grep POST | head -10
DEBUG: Initiating POST to https://chef-stg.example.com/organizations/example/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=0
DEBUG: Initiating POST to https://chef-stg.example.com/organizations/example/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=17
DEBUG: Initiating POST to https://chef-stg.example.com/organizations/example/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=17
DEBUG: Initiating POST to https://chef-stg.example.com/organizations/example/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=17
DEBUG: Initiating POST to https://chef-stg.example.com/organizations/example/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=17
DEBUG: Initiating POST to https://chef-stg.example.com/organizations/example/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=17
DEBUG: Initiating POST to https://chef-stg.example.com/organizations/example/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=17
DEBUG: Initiating POST to https://chef-stg.example.com/organizations/example/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=17
DEBUG: Initiating POST to https://chef-stg.example.com/organizations/example/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=17
DEBUG: Initiating POST to https://chef-stg.example.com/organizations/example/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=17

 
The problem with this behavior can be found in the "start=" at the end of the lines. It's repeating, which means that the `knife status` command is unable to make progress as it should. Normally that entry would be increasing by a regular amount for each trip through the loop.

The simplest thing to do in this case would be reindex your search index for the given org, "example" in this case, with

 

chef-server-ctl reindex example



Investigating differences
----------------------------

A good thing to do beforehand would be to investigate any differences with the attached script.
The script has instructions in its header that tell what items need to be changed to match your specific system type and orgname, and the indexed type you are trying to investigate.

Once you have the script downloaded, place it on a Standalone Chef Server or any of the frontends for a cluster, chmod, and run it like this

chmod 0700 diff-index-postgres.sh
bash ./diff-index-postgres.sh

Depending on the 3 settings you configure at the top of the script (documented there) it will provide output files for testing the difference between the database and search index for various items a chef server can contain, environments, nodes, databag items, and so on. Once you have this data, please contact Chef support and we can work with you figure out where to go if a difference is found.

 

Credit to Jeremiah Snapp for the original version of this cool little script!

Have more questions? Submit a request

Comments

Powered by Zendesk