Chef Backend HA Default ES Mapping Search Failure

Sean Horn -

When the mappings/settings on a search index are the Elasticsearch defaults rather than what would be set by a successful `chef-server-ctl reconfigure` while no traffic is coming in to the chef backend HA cluster, you will see the following symptoms, regardless of the Chef Client/Knife version.

 

Here is an example of what the Elasticsearch defaults look like. These mappings/settings will result in a failure to search

root@be1:~# curl -XGET http://localhost:9200/chef
{"chef":{"aliases":{},"mappings":{"object":{"properties":{"X_CHEF_database_CHEF_X":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"X_CHEF_id_CHEF_X":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"X_CHEF_type_CHEF_X":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"content":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}},"settings":{"index":{"creation_date":"1538494993057","number_of_shards":"5","number_of_replicas":"1","uuid":"XT4ZqoP2SmSA9KPKIBF8iA","version":{"created":"5040199"},"provided_name":"chef"}}}}

 

Another way to tell the index is arranged improperly. See how this readout has "_source" in it? Thats incorrect, and results from the elasticsearch chef index being configured with the default mappings assigned by ES when an ingestion occurs before the Chef Server specific mappings/settings are applied to the chef index.

 

root@be1:~# curl -s 'localhost:9200/chef/_search?q=*' | head -c 500
{"took":7,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"chef","_type":"object","_id":"c77909d508ffcf758daec51033a09d97","_score":1.0,"_source":{"content":"/__=__devices /__=__fs_type /__=__inodes_available /__=__inodes_percent_used /__=__inodes_used /__=__kb_available /__=__kb_size /__=__kb_used /__=__label /__=__mount_options /__=__percent_used /__=__total_inodes /__=__uuid /dev__=__devices /dev__=__fs_type /dev__=__inodes

 

knife client side


root@client1:~# knife search node '*:*'
ERROR: knife search failed: invalid index name or query

 

frontend erchef request log


2018-10-02T15:43:35Z erchef@127.0.0.1 method=GET; path=/organizations/4thcoffee/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=0; status=400; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEDAACIqgAAAAEAAAAA; org_name=4thcoffee; msg={solr_400,"/chef/_search"}; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=40; rdbms_time=1; rdbms_count=3; solr_time=32; solr_count=1; user=me; req_api_version=1;

 

Elasticsearch


org.elasticsearch.transport.RemoteTransportException: [e95fd83d2b58d1e33ae6ae42d03bb246][192.168.33.216:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [X_CHEF_id_CHEF_X] in order to load fielddata in memory b
y uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:336) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:111) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:166) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:277) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:156) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:621) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.search.SearchService.createContext(SearchService.java:472) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:448) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:339) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:336) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1528) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.1.jar:5.4.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
[2018-10-02T15:43:35,238][DEBUG][o.e.a.s.TransportSearchAction] [643daf1bb807c4b554dff4880a5d5158] [chef][3], node[0qUOXOabSl-CN3s1qMDjcQ], [R], s[STARTED], a[id=HXsB8c2PStq5R
bIacrcY1A]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[chef], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true,
expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[], routing='null', preference='null',
requestCache=null, scroll=null, source={
Have more questions? Submit a request

Comments

Powered by Zendesk