[CI] CohereServiceMixedIT testRerank failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #22789 / 8.19.0_bwc-snapshots
• elasticsearch-intake #22787 / 8.19.0_bwc-snapshots
• elasticsearch-intake #22786 / 8.19.0_bwc-snapshots
• elasticsearch-intake #22785 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #71016 / 8.19.0_bwc-snapshots

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:mixed-cluster:v8.19.0#javaRestTest" -Dtests.class="org.elasticsearch.xpack.inference.qa.mixed.CohereServiceMixedIT" -Dtests.method="testCohereEmbeddings" -Dtests.seed=3944E0D378F145C9 -Dtests.bwc=true -Dtests.locale=ar-JO -Dtests.timezone=Asia/Novosibirsk -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:36815], URI [_inference/text_embedding/mixed-cluster-cohere-embeddings-float], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"Inference endpoint not found [mixed-cluster-cohere-embeddings-float]"}],"type":"resource_not_found_exception","reason":"Inference endpoint not found [mixed-cluster-cohere-embeddings-float]"},"status":404}

Issue Reasons:

• [main] 5 failures in test testRerank (1.9% fail rate in 260 executions)
• [main] 5 failures in step 8.19.0_bwc-snapshots (4.0% fail rate in 124 executions)
• [main] 4 failures in pipeline elasticsearch-intake (12.5% fail rate in 32 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testRerank (0.8% fail rate in 256 executions)
• [main] 2 failures in step 8.19.0_bwc-snapshots (1.7% fail rate in 121 executions)

Build Scans:

• elasticsearch-intake #22785 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #71016 / 8.19.0_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[jonathan-buttner]

#127872 (comment)

[jonathan-buttner]

PR to fix: #128000