Submit Feedback
Legacy Knowledge Base
Published Jun. 30, 2025

ElasticsearchStatusException[Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]]

Written By

Sergio Alonso

How To articles are not official guidelines or officially supported documentation. They are community-contributed content and may not always reflect the latest updates to Liferay DXP. We welcome your feedback to improve How To articles!

While we make every effort to ensure this Knowledge Base is accurate, it may not always reflect the most recent updates or official guidelines.We appreciate your understanding and encourage you to reach out with any feedback or concerns.

Legacy Article

You are viewing an article from our legacy "FastTrack" publication program, made available for informational purposes. Articles in this program were published without a requirement for independent editing or verification and are provided"as is" without guarantee.

Before using any information from this article, independently verify its suitability for your situation and project.

Issue

  • We are getting lots of following messages in log:

    ERROR [http-nio-8080-exec-3][ElasticsearchIndexSearcher:152] [] null
    java.lang.RuntimeException: org.elasticsearch.ElasticsearchStatusException: ElasticsearchStatusException[Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]]
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[?:?]
    at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[?:?]
    at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[?:?]
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[?:?]
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137) ~[?:?]
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105) ~[?:?]
    at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1367) ~[?:?]
    at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.search.SearchSearchRequestExecutorImpl._getSearchResponse(SearchSearchRequestExecutorImpl.java:135) ~[?:?]
    at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.search.SearchSearchRequestExecutorImpl.execute(SearchSearchRequestExecutorImpl.java:68) ~[?:?]
    at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.search.ElasticsearchSearchRequestExecutor.executeSearchRequest(ElasticsearchSearchRequestExecutor.java:78) ~[?:?]
    at com.liferay.portal.search.engine.adapter.search.SearchSearchRequest.accept(SearchSearchRequest.java:37) ~[?:?]
    at com.liferay.portal.search.engine.adapter.search.SearchSearchRequest.accept(SearchSearchRequest.java:26) ~[?:?]
    at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.ElasticsearchSearchEngineAdapterImpl.execute(ElasticsearchSearchEngineAdapterImpl.java:94) ~[?:?]
    at com.liferay.portal.search.elasticsearch7.internal.ElasticsearchIndexSearcher._search(ElasticsearchIndexSearcher.java:558) ~[?:?]
    at com.liferay.portal.search.elasticsearch7.internal.ElasticsearchIndexSearcher.search(ElasticsearchIndexSearcher.java:140) ~[?:?]
    at com.liferay.portal.search.internal.IndexSearcherHelperImpl.search(IndexSearcherHelperImpl.java:59) ~[?:?]
    at com.liferay.portal.kernel.search.IndexSearcherHelperUtil.search(IndexSearcherHelperUtil.java:27) ~[portal-kernel.jar:?]
    ...
    ...
    ...

Environment

  • Liferay Quarterly Release 2023.Q4 and Elasticsearch 7

Resolution

  • As first step, let's get Elasticsearch current status:
    • Cluster status:
    • $ curl -X GET "[HOST ES]:[PORT ES]/_cluster/health?pretty"
    • Indices status:
    • $ curl -X GET [HOST ES]:[PORT ES]/_cat/indices
    • Shards status:
    • $ curl -X GET [HOST ES]:[PORT ES]/_cat/shards

  • We get following data:

    • $ curl -X GET "http://[HOST ES]:[PORT ES]/_cluster/health?pretty"

      {
      "cluster_name" : "LiferayElasticsearchCluster",
      "status" : "yellow",
      "timed_out" : false,
      "number_of_nodes" : 1,
      "number_of_data_nodes" : 1,
      "active_primary_shards" : 23,
      "active_shards" : 23,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 12,
      "delayed_unassigned_shards" : 0,
      "number_of_pending_tasks" : 0,
      "number_of_in_flight_fetch" : 0,
      "task_max_waiting_in_queue_millis" : 0,
      "active_shards_percent_as_number" : 55.00000000000001
      }

      There are unassigned shards. And customer is using one node only. Therefore, cluster health is yellow (instead of green).

       

      $ curl -X GET "http://[HOST ES]:[PORT ES]/_cat/indices" 

      green open liferay-20096-product-content-commerce-ml-recommendation Wsaniu7FTeG3pr7lKxEnig 1 0 0 0 226b 226b
      green open liferay-20096-search-tuning-rankings iLvhwAjQSsOMKTskgI5rLw 1 0 0 0 226b 226b
      yellow open liferay-20096 r-cx6VZaTe65iib2QsyFFg 3 2 920416 369043 276.8mb 276.8mb
      yellow open liferay-20096-workflow-metrics-processes Q6QLeEdGQiqYwIQXatv_ag 1 1 3 0 12.1kb 12.1kb
      green open liferay-20096-frequent-pattern-commerce-ml-recommendation CdustgiFSyS-AC78Jx99IQ 1 0 0 0 226b 226b
      yellow open liferay-20096-workflow-metrics-instances XJZnuqk3Tu6GtT4hBSBqPg 1 1 1292 39 1.1mb 1.1mb
      green open liferay-20096-user-commerce-ml-recommendation 7JYYWUIST_KOcDv5zjLBTQ 1 0 0 0 226b 226b
      green open .geoip_databases wcQlrpijRviQkscoyThu8g 1 0 38 0 36.6mb 36.6mb
      yellow open liferay-0 nIQrW3ihRn2nP5-asu7IGQ 3 2 429 429 1mb 1mb
      green open liferay-20096-commerce-ml-forecast L6QAqF4NQumr5Th2jI4Yng 1 0 0 0 226b 226b
      green open liferay-20096-search-tuning-synonyms 1_qYEDzYRGCrupWDeFWVwQ 1 0 0 0 226b 226b
      yellow open liferay-20096-workflow-metrics-sla-task-results u117C6LQSv24wc1Q4GFGvQ 1 1 45 0 56.8kb 56.8kb
      yellow open liferay-20096-workflow-metrics-nodes bTn2gt-pQWeyiJT9TMeH6A 1 1 94 41 87.3kb 87.3kb
      green open liferay-20096-product-interaction-commerce-ml-recommendation -bfiaWW6TYqUd899L0yPig 1 0 0 0 226b 226b
      yellow open liferay-20096-workflow-metrics-transitions fxOYyBnxSLOgka03jXmK6Q 1 1 104 0 101.1kb 101.1kb
      yellow open liferay-20096-workflow-metrics-tasks q08sQnlbRRuLv0ldLGeg3g 1 1 1284 175 1mb 1mb

      There are indices with yellow status (instead of green).

       

      $ curl -X GET "http://[HOST ES]:[PORT ES]/_cat/shards" 

      liferay-0 1 p STARTED 129 308.4kb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-0 1 r UNASSIGNED 
      liferay-0 1 r UNASSIGNED 
      liferay-0 2 p STARTED 149 426.9kb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-0 2 r UNASSIGNED 
      liferay-0 2 r UNASSIGNED 
      liferay-0 0 p STARTED 152 401kb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-0 0 r UNASSIGNED 
      liferay-0 0 r UNASSIGNED 
      liferay-20096-product-content-commerce-ml-recommendation 0 p STARTED 0 226b 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-search-tuning-synonyms 0 p STARTED 0 226b 10.244.4.37 search-56c49f7b94-7ns27
      .geoip_databases 0 p STARTED 38 36.7mb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-workflow-metrics-sla-task-results 0 p STARTED 1077 382.5kb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-workflow-metrics-instances 0 p STARTED 37 161.2kb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096 1 p STARTED 570561 123.3mb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096 1 r UNASSIGNED 
      liferay-20096 1 r UNASSIGNED 
      liferay-20096 2 p STARTED 571791 160.2mb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096 2 r UNASSIGNED 
      liferay-20096 2 r UNASSIGNED 
      liferay-20096 0 p STARTED 564546 122.4mb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096 0 r UNASSIGNED 
      liferay-20096 0 r UNASSIGNED 
      liferay-20096-product-interaction-commerce-ml-recommendation 0 p STARTED 0 226b 10.244.4.37 search-56c49f7b94-7ns27
      .ds-.logs-deprecation.elasticsearch-default-2024.10.08-000001 0 p STARTED 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-workflow-metrics-sla-instance-results 0 p STARTED 0 226b 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-workflow-metrics-nodes 0 p STARTED 4427 2mb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-workflow-metrics-tasks 0 p STARTED 20 115.9kb 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-workflow-metrics-transitions 0 p STARTED 0 226b 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-user-commerce-ml-recommendation 0 p STARTED 0 226b 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-workflow-metrics-processes 0 p STARTED 0 226b 10.244.4.37 search-56c49f7b94-7ns27
      .ds-ilm-history-5-2024.10.08-000001 0 p STARTED 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-frequent-pattern-commerce-ml-recommendation 0 p STARTED 0 226b 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-commerce-ml-forecast 0 p STARTED 0 226b 10.244.4.37 search-56c49f7b94-7ns27
      liferay-20096-search-tuning-rankings 0 p STARTED 0 226b 10.244.4.37 search-56c49f7b94-7ns27
      The replica shards are unassigned. Only the primary shards are started.
  • Therefore, it seems there is a problem related to assign shards to the Elasticsearch node. Let's try to diagnose, using Elasticsearch API, why those shards are not being assigned.
    • $ curl -X GET 'http://[HOST ES]:[PORT ES]/_cluster/allocation/explain' \
      -H 'Content-Type: application/json' \
      -d '{
      "index" : "liferay-0",
      "shard" : 0,
      "primary" : false
      }'

      {"index":"liferay-0","shard":0,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"INDEX_CREATED","at":"2024-10-22T13:59:31.528Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"N4PXaS33R4qtSK-76sCd_A","node_name":"search-56c49f7b94-7ns27","transport_address":"10.244.4.37:9300","node_attributes":{"ml.machine_memory":"12884901888","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"512","ml.max_jvm_size":"8589934592"},"node_decision":"no","weight_ranking":1,"deciders":[{"decider":"same_shard","decision":"NO","explanation":"a copy of this shard is already allocated to this node [[liferay-0][0], node[N4PXaS33R4qtSK-76sCd_A], [P], s[STARTED], a[id=RAoyo5cjSQ6WOvvV9q6HvQ]]"}]}]}




      $ curl -X GET 'http://[HOST ES]:[PORT ES]/_cluster/allocation/explain' \
      -H 'Content-Type: application/json' \
      -d '{
      "index" : "liferay-0",
      "shard" : 1,
      "primary" : false
      }'

      {"index":"liferay-0","shard":1,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"INDEX_CREATED","at":"2024-10-22T13:59:31.528Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"N4PXaS33R4qtSK-76sCd_A","node_name":"search-56c49f7b94-7ns27","transport_address":"10.244.4.37:9300","node_attributes":{"ml.machine_memory":"12884901888","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"512","ml.max_jvm_size":"8589934592"},"node_decision":"no","weight_ranking":1,"deciders":[{"decider":"same_shard","decision":"NO","explanation":"a copy of this shard is already allocated to this node [[liferay-0][1], node[N4PXaS33R4qtSK-76sCd_A], [P], s[STARTED], a[id=ASVwb5SvQ8yQW31ChYIaFg]]"}]}]}




      $ curl -X GET 'http://[HOST ES]:[PORT ES]/_cluster/allocation/explain' \
      -H 'Content-Type: application/json' \
      -d '{
      "index" : "liferay-0",
      "shard" : 2,
      "primary" : false
      }'

      {"index":"liferay-0","shard":2,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"INDEX_CREATED","at":"2024-10-22T13:59:31.528Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"N4PXaS33R4qtSK-76sCd_A","node_name":"search-56c49f7b94-7ns27","transport_address":"10.244.4.37:9300","node_attributes":{"ml.machine_memory":"12884901888","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"512","ml.max_jvm_size":"8589934592"},"node_decision":"no","weight_ranking":1,"deciders":[{"decider":"same_shard","decision":"NO","explanation":"a copy of this shard is already allocated to this node [[liferay-0][2], node[N4PXaS33R4qtSK-76sCd_A], [P], s[STARTED], a[id=u2SsH6l9RlGbHyebXkXDXg]]"}]}]}


      $ curl -X GET 'http://[HOST ES]:[PORT ES]/_cluster/allocation/explain' \
      -H 'Content-Type: application/json' \
      -d '{
      "index" : "liferay-20096",
      "shard" : 0,
      "primary" : false
      }'

      {"index":"liferay-20096","shard":0,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"INDEX_CREATED","at":"2024-10-22T14:08:13.486Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"N4PXaS33R4qtSK-76sCd_A","node_name":"search-56c49f7b94-7ns27","transport_address":"10.244.4.37:9300","node_attributes":{"ml.machine_memory":"12884901888","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"512","ml.max_jvm_size":"8589934592"},"node_decision":"no","weight_ranking":1,"deciders":[{"decider":"same_shard","decision":"NO","explanation":"a copy of this shard is already allocated to this node [[liferay-20096][0], node[N4PXaS33R4qtSK-76sCd_A], [P], s[STARTED], a[id=bncWS3q2RsCbwzWwfenczA]]"}]}]}




      $ curl -X GET 'http://[HOST ES]:[PORT ES]/_cluster/allocation/explain' \
      -H 'Content-Type: application/json' \
      -d '{
      "index" : "liferay-20096",
      "shard" : 1,
      "primary" : false
      }'

      {"index":"liferay-20096","shard":1,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"INDEX_CREATED","at":"2024-10-22T14:08:13.486Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"N4PXaS33R4qtSK-76sCd_A","node_name":"search-56c49f7b94-7ns27","transport_address":"10.244.4.37:9300","node_attributes":{"ml.machine_memory":"12884901888","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"512","ml.max_jvm_size":"8589934592"},"node_decision":"no","weight_ranking":1,"deciders":[{"decider":"same_shard","decision":"NO","explanation":"a copy of this shard is already allocated to this node [[liferay-20096][1], node[N4PXaS33R4qtSK-76sCd_A], [P], s[STARTED], a[id=XUBWCYO1QIWHLXJaGzANBQ]]"}]}]}




      $ curl -X GET 'http://[HOST ES]:[PORT ES]/_cluster/allocation/explain' \
      -H 'Content-Type: application/json' \
      -d '{
      "index" : "liferay-20096",
      "shard" : 2,
      "primary" : false
      }'

      {"index":"liferay-20096","shard":2,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"INDEX_CREATED","at":"2024-10-22T14:08:13.486Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"N4PXaS33R4qtSK-76sCd_A","node_name":"search-56c49f7b94-7ns27","transport_address":"10.244.4.37:9300","node_attributes":{"ml.machine_memory":"12884901888","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"512","ml.max_jvm_size":"8589934592"},"node_decision":"no","weight_ranking":1,"deciders":[{"decider":"same_shard","decision":"NO","explanation":"a copy of this shard is already allocated to this node [[liferay-20096][2], node[N4PXaS33R4qtSK-76sCd_A], [P], s[STARTED], a[id=GxlO7aBYTcaL_Bgm8p3Q0w]]"}]}]}

  • Root cause is Elasticsearch cluster has only one node, and Elasticsearch does not allow to assign any replica shard to the same node where a primary shard is already assigned (as preventive measure against hardware failures).

  • There are two options as possible solution:

    • Either adding additional nodes to the Elasticsearch cluster. Thus, replica shards will be assigned to them. This would be the recommend option.

    • Or do not assign any replica shard to the Elasticsearch cluster. This would be a less recommend option.

      $ curl -X PUT "[HOST ES]:[PORT ES]/_settings?pretty" \
      -H 'Content-Type: application/json' \
      -d '{
      "index.number_of_replicas": 0
      }'

      Once previous curl command was executed, you can check if replica shards do not exist:

      $ curl -X GET "http://[HOST ES]:[PORT ES]/_cat/shards"

      It replicas shards are still present, a full reindex would be necessary to recreate instances indices:

      Control Panel -> CONFIGURATION -> Search -> Index Actions tab -> Configuration - Reindex Mode: Full -> Configuration - Reindex Scope: All Instances -> Actions - GLOBAL - All Search Indexes -> click on Reindex button.

 

 

 

Did this article resolve your issue ?
Legacy Knowledge Base
Did this article resolve your issue ?