Fix elasticsearch #62

Bueller87 · 2025-11-13T17:00:41Z

Summary

Adds support for deploying Cadence with PostgreSQL + Elasticsearch + Kafka for advanced visibility. This configuration enables powerful workflow search and filtering capabilities through Elasticsearch while using PostgreSQL as the primary persistence layer.

New Example Configuration: `values.postgres-es7.yaml`

This branch introduces a complete, production-ready example for deploying Cadence with:

PostgreSQL - Primary persistence for workflow state and history
Elasticsearch 7 - Advanced visibility with rich search capabilities
Kafka - Asynchronous visibility event processing for better performance
Kafka topic auto-provisioning - Eliminates manual setup and race conditions

Testing

Clean deployment with all pods healthy (no scheduling failures)
Kafka topics auto-provisioned with correct replication factors
Workflows visible in Cadence Web UI immediately after execution
No visibility-related errors in service logs
Repeatable deployments (tested multiple tear-down/redeploy cycles)

Caveats

Single-broker Kafka: Configured for development/testing with replicationFactor: 1. For production, increase Kafka broker count and replication factors to 3+.
Resource requirements: Deploys 9+ pods (Cadence services + PostgreSQL + Elasticsearch + Kafka). Ensure your cluster has adequate resources.
Fresh deployments recommended: If upgrading from a previous configuration, a clean redeploy is recommended to ensure Kafka topics are properly provisioned.

Checklist

[ x] Chart.yaml: Chart version bumped
[ x] values.yaml: global.image.tag (should match with Chart.yaml appVersion)
[ x] Dependencies in Chart.yaml (check for updates with helm dependency update, aim for latest major-1 stable)
[ x] DCO signed
[ x] Test deployment locally
[ x] Run helm-docs to update documentation

) Description of this PR schema upgrade job should fail open on db creation to handle already exist errors add example of cadence with postgres Local Test namespace and release name is still fixed unfortunately. cd charts/cadence helm install cadence-release . -n cadence-postgres --values examples/values.postgres.yaml Signed-off-by: “Kevin” <[email protected]>

…dence-workflow#57) * docs: Convert slack links to CNCF, add link to contributing guide Signed-off-by: [email protected] <[email protected]> Signed-off-by: “Kevin” <[email protected]>

Signed-off-by: CosminL-DEV <[email protected]> Signed-off-by: “Kevin” <[email protected]>

Signed-off-by: “Kevin” <[email protected]>

When using Elasticsearch for visibility, the PostgreSQL visibility database is not needed. Removed visibilityDbname and initdb script that were creating an unused database. Also updated dynamic config to use es instead of es-visibility to match Cadence's internal visibility store naming convention. Signed-off-by: “Kevin” <[email protected]>

The schema job should only create the SQL/Cassandra visibility database when Elasticsearch is disabled. When using Elasticsearch for advanced visibility, the SQL visibility database is not needed. Changed condition from checking if visibilityDbname is set to checking if ES_ENABLED is false, allowing ES-only deployments to skip unnecessary database creation. Signed-off-by: “Kevin” <[email protected]>

Changed visibilityIndex from 'cadence-visibility' to 'cadence-visibility-v1' to match the Elasticsearch template pattern 'cadence-visibility-*'. This ensures the template with proper keyword field mappings is applied to the index, fixing visibility queries that require exact term matching on fields like DomainID. Signed-off-by: “Kevin” <[email protected]>

Set visibilityStore to empty string to prevent dual-write mode when using Elasticsearch-only advanced visibility. This fixes 'Operation is not supported' errors when Cadence tries to write to a non-existent SQL visibility database. Signed-off-by: “Kevin” <[email protected]>

The visibility store name in dynamic config should be 'es' (a special keyword for advanced visibility), not the datastore name 'es-visibility'. This matches the Cadence upstream configuration pattern where advancedVisibilityStore is 'es-visibility' but system.writeVisibilityStoreName is 'es'. Signed-off-by: “Kevin” <[email protected]>

Removed advancedVisibilityStore from persistence config to enable Kafka-based async visibility mode. When advancedVisibilityStore is set, Cadence tries to use dual-write mode (SQL + ES) which fails without a SQL visibility database. With Kafka mode, the history service writes to Kafka, and the worker service reads from Kafka and writes to ES. Signed-off-by: “Kevin” <[email protected]>

Must explicitly override the default value of 'es-visibility' with an empty string to disable direct ES writes and enable Kafka-based async visibility. Signed-off-by: “Kevin” <[email protected]>

Cadence requires at least one of visibilityStore or advancedVisibilityStore to be set. We keep advancedVisibilityStore='es-visibility' to define the datastore, and use dynamic config system.writeVisibilityStoreName='es' to enable Kafka-based async visibility instead of direct ES writes. Signed-off-by: “Kevin” <[email protected]>

Added Kafka provisioning configuration to pre-create required topics: - __consumer_offsets: Internal Kafka topic for consumer offset tracking - cadence-visibility: Main visibility events topic - cadence-visibility-dlq: Dead letter queue for failed visibility events This prevents race conditions where topics get auto-created with incorrect replication factors before the broker config is fully loaded. All topics are created with replication-factor: 1 to match our single-broker setup. Signed-off-by: “Kevin” <[email protected]>

Signed-off-by: “Kevin” <[email protected]>

charts/cadence/templates/server-configmap.yaml

charts/cadence/examples/values.postgres-es7.yaml

Signed-off-by: “Kevin” <[email protected]>

Reverts the visibility store conditional logic to include the elasticsearch enabled check. This prevents creating an unnecessary SQL visibility datastore when users enable Elasticsearch without explicitly disabling visibilityStore. Without this check, the default visibilityStore value from values.yaml would be used even when Elasticsearch is enabled, creating both SQL and ES visibility stores unintentionally. Signed-off-by: “Kevin” <[email protected]>

Added kafka.enabled: false to values.postgres.yaml to prevent unnecessary Kafka deployment when using SQL-only visibility. Kafka is only needed for async visibility processing with Elasticsearch. Signed-off-by: “Kevin” <[email protected]>

shijiesheng · 2025-11-25T18:53:13Z

charts/cadence/templates/server-configmap.yaml

+        {{- if and .Values.config.persistence.visibilityStore (not .Values.config.persistence.elasticsearch.enabled) }}
        # Visibility datastore
-        {{ if .Values.config.persistence.visibilityStore -}}
-        {{ .Values.config.persistence.visibilityStore | default "visibility" }}:


we need this default visibility

When using Elasticsearch with Kafka, visibility events flow through Kafka to ES. The SQL visibility datastore is not used. (current config)

In fact I found when BOTH visibilityStore and advancedVisibilityStore are configured, Cadence enters dual-write mode where it tries to write visibility events to BOTH datastores.

I encountered multiple errors when I did this, e.g.
{"level":"warn","msg":"requested visibility mode is not available, falling back to db"}
{"level":"error","msg":"Error writing to visibility: Operation is not supported"}

Through debugging I saw this happening:
First, Cadence tried to write to BOTH SQL and ES.
But, the SQL cadence_visibility database didn't exist (schema job skipped it because ES was enabled).
Writes to the SQL visibility failed with "Operation is not supported",
and visibility was broken

pretty sure there is no change needed here. I've reverted your change and the plan looks exactly the same. Please revert it.

shijiesheng and others added 11 commits November 14, 2025 10:52

docs: Convert slack links to CNCF, add link to contributing guide (ca…

502a562

…dence-workflow#57) * docs: Convert slack links to CNCF, add link to contributing guide Signed-off-by: [email protected] <[email protected]> Signed-off-by: “Kevin” <[email protected]>

Bump Web v4.0.11 (cadence-workflow#60)

e63fb49

Signed-off-by: CosminL-DEV <[email protected]> Signed-off-by: “Kevin” <[email protected]>

add es examples and add to chart dependencies

ecc3e4e

Signed-off-by: “Kevin” <[email protected]>

add additional comment

4d1be74

Signed-off-by: “Kevin” <[email protected]>

add kafka values to elasticsearch example

45a7f51

Signed-off-by: “Kevin” <[email protected]>

fix configmap so visibility datastore is created

9122e55

Signed-off-by: “Kevin” <[email protected]>

change cadence verion to 1.3.7

481cf79

Signed-off-by: “Kevin” <[email protected]>

Set cadence version to v1.3.6 in chart and values

be5335d

Signed-off-by: “Kevin” <[email protected]>

fix: always create visibility database when configured

d8bb312

Signed-off-by: “Kevin” <[email protected]>

run helm-docs to update Readme

c85e226

Signed-off-by: “Kevin” <[email protected]>

Bueller87 force-pushed the fix-elasticsearch branch from 9ca4016 to c85e226 Compare November 14, 2025 18:53

Bueller87 added 10 commits November 17, 2025 14:22

fix: explicitly set advancedVisibilityStore to empty string

297dcad

Must explicitly override the default value of 'es-visibility' with an empty string to disable direct ES writes and enable Kafka-based async visibility. Signed-off-by: “Kevin” <[email protected]>

Ran helm-docs

4ad9d9d

Signed-off-by: “Kevin” <[email protected]>

shijiesheng reviewed Nov 20, 2025

View reviewed changes

charts/cadence/templates/server-configmap.yaml Outdated Show resolved Hide resolved

charts/cadence/templates/server-configmap.yaml Show resolved Hide resolved

charts/cadence/examples/values.postgres-es7.yaml Outdated Show resolved Hide resolved

Bueller87 and others added 4 commits November 20, 2025 13:16

Merge branch 'main' into fix-elasticsearch

47b90b7

fix: change visibility index name

15d3d14

Signed-off-by: “Kevin” <[email protected]>

shijiesheng reviewed Nov 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix elasticsearch #62

Fix elasticsearch #62

Bueller87 commented Nov 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shijiesheng Nov 25, 2025

Uh oh!

Bueller87 Nov 25, 2025

Uh oh!

shijiesheng Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Fix elasticsearch #62

Are you sure you want to change the base?

Fix elasticsearch #62

Conversation

Bueller87 commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

New Example Configuration: values.postgres-es7.yaml

Testing

Caveats

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shijiesheng Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Bueller87 Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

shijiesheng Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Bueller87 commented Nov 13, 2025 •

edited

Loading

New Example Configuration: `values.postgres-es7.yaml`