Dapr state store clickhouse #3675

middt · 2025-02-23T04:10:23Z

Description

This PR adds a new state store component for ClickHouse, a column-oriented database management system. The ClickHouse state store component provides the following features:

Basic CRUD operations for state management
Support for TTL (Time-To-Live)
ETag support for optimistic concurrency
Bulk operations support
Username/password authentication
Uses ReplacingMergeTree engine for efficient updates

Key implementation details:

Uses the official ClickHouse Go driver
Implements the full state.Store interface
Handles connection management and cleanup
Includes comprehensive integration tests
Follows Dapr's component development guidelines

Implementation Details

The component includes:

State store implementation (clickhouse.go)
- Full CRUD operations
- Bulk operations
- TTL support
- ETag support
- Authentication support
Tests (clickhouse_test.go)
- Integration tests for CRUD operations
- Metadata validation tests
- Authentication tests
- Cleanup handling

Configuration options:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore
  namespace: default
spec:
  type: state.clickhouse
  version: v1
  metadata:
  - name: clickhouseURL
    value: "tcp://localhost:9000"
  - name: databaseName
    value: "dapr_state"
  - name: tableName
    value: "state_store"
  - name: username
    value: "default"
  - name: password
    value: ""

Checklist

Code compiles correctly
Created/updated tests
Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Testing Done

Unit tests for metadata validation
Integration tests for:
- Basic CRUD operations
- TTL functionality
- ETag support
- Bulk operations
- Authentication
- Error handling

Tests were run against ClickHouse v23.8 using the official Go driver.

Additional Notes

The component uses ReplacingMergeTree engine for better update performance
Includes proper cleanup in tests to avoid test pollution
Follows ClickHouse best practices for table design
Implements proper connection management and cleanup

Signed-off-by: Mehmet TOSUN <[email protected]>

JoshVanL · 2025-03-13T14:58:17Z

Thanks @middt, implementation generally looks good to me!

Would it be possible to add conformance tests for this too? It seems like we should be able to spin up a click house docker compose.

…tation Signed-off-by: Mehmet TOSUN <[email protected]>

…tion handling Signed-off-by: Mehmet TOSUN <[email protected]>

… connection handling Signed-off-by: Mehmet TOSUN <[email protected]>

middt · 2025-03-14T07:04:22Z

Thank you for the feedback! @JoshVanL I've implemented the conformance tests for the ClickHouse state store in this PR:

Created a Docker Compose setup in /tests/certification/state/clickhouse/docker-compose.yml that spins up a ClickHouse server with proper configuration
Added a conformance test file in /tests/certification/state/clickhouse/clickhouse_test.go that verifies all state store operations
Created configuration files in /tests/config/state/clickhouse/clickhouse.yml for the test environment
Added ClickHouse to the state store tests configuration in /tests/config/state/tests.yml

The tests verify all the key functionality including:

Basic CRUD operations
ETag support for optimistic concurrency
Error handling for empty and non-existent keys
Feature reporting

I've also addressed authentication issues by properly configuring username and password in both the Docker Compose setup and the state store implementation.

All unit tests are now passing, confirming that the implementation works correctly with the ClickHouse server.

Signed-off-by: Mehmet TOSUN <[email protected]>

JoshVanL · 2025-04-09T19:18:24Z

Thanks @middt, I think the only thing left is to do a go mod tidy and it should be good to go 🙂

Appreciate the work on this!

JoshVanL · 2025-08-29T12:39:03Z

Hey @middt, tests are failing

- Add environment variable check for RUN_CLICKHOUSE_INTEGRATION_TEST - Skip integration tests when ClickHouse is not available - This fixes CI test failures when ClickHouse server is not running Signed-off-by: Mehmet TOSUN <[email protected]>

middt · 2025-08-31T03:40:20Z

Hey @middt, tests are failing

Hi @JoshVanL,
Fixed! Tests now pass:, Ready for review!

Signed-off-by: Mehmet TOSUN <[email protected]>

…tests Signed-off-by: Mehmet TOSUN <[email protected]>

Signed-off-by: Mehmet TOSUN <[email protected]>

github-actions · 2025-11-08T16:47:15Z

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

github-actions · 2025-11-15T16:55:19Z

This pull request has been automatically closed because it has not had activity in the last 37 days. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

sicoyle

few comments. Nice job 👏

sicoyle · 2025-12-09T20:58:33Z

state/clickhouse/clickhouse.go

@@ -0,0 +1,435 @@
+/*
+Copyright 2021 The Dapr Authors


Suggested change

Copyright 2021 The Dapr Authors

Copyright 2025 The Dapr Authors

sicoyle · 2025-12-09T21:03:02Z

state/clickhouse/clickhouse.go

+	// Create database if not exists
+	// Note: Database and table names are validated during metadata parsing
+	// and come from trusted configuration, so direct string concatenation is acceptable here
+	createDBQuery := "CREATE DATABASE IF NOT EXISTS " + c.config.DatabaseName


we need to make sure this behavior is documented. Have you created a docs PR for these changes by chance? We try to have a docs PR open before merging new components please 🙏

sicoyle · 2025-12-09T21:09:18Z

state/clickhouse/clickhouse.go

+	if config.DatabaseName == "" {
+		return config, errors.New("ClickHouse database name is missing")
+	}
+
+	if config.TableName == "" {


are there fixed char limits on these we should check for by chance?

sicoyle · 2025-12-09T21:10:09Z

state/clickhouse/clickhouse_integration_test.go

@@ -0,0 +1,569 @@
+/*
+Copyright 2021 The Dapr Authors


Suggested change

Copyright 2021 The Dapr Authors

Copyright 2025 The Dapr Authors

sicoyle · 2025-12-09T21:11:54Z

state/clickhouse/clickhouse_test.go

@@ -0,0 +1,240 @@
+/*
+Copyright 2021 The Dapr Authors


Suggested change

Copyright 2021 The Dapr Authors

Copyright 2025 The Dapr Authors

sicoyle · 2025-12-09T21:36:03Z

state/clickhouse/metadata.yaml

+  - title: "Username and password"
+    description: "Authenticate using username and password."
+    metadata:
+      - name: Username
+        type: string
+        required: false
+        description: |
+          Username for ClickHouse host. Defaults to empty.
+        example: "default"
+        default: ""
+      - name: Password
+        type: string
+        required: false
+        sensitive: true
+        description: |
+          Password for ClickHouse host. No default. Use secretKeyRef for
+          secret reference
+        example: "mypassword"
+        default: ""


this says that none of the fields for auth are required. Can you please confirm if this is this true that we can connect without a username and password?

sicoyle · 2025-12-09T21:46:34Z

state/clickhouse/metadata.yaml

+        example: "mypassword"
+        default: ""
+metadata:
+  - name: ClickhouseURL


can you pls update these all to be camelcase?

sicoyle · 2025-12-09T21:46:50Z

tests/certification/state/clickhouse/clickhouse_test.go

+// +build conftests
+
+/*
+Copyright 2021 The Dapr Authors


Suggested change

Copyright 2021 The Dapr Authors

Copyright 2025 The Dapr Authors

sicoyle · 2025-12-09T21:49:52Z

@JoshVanL did you have any other feedback here by chance?

middt added 18 commits November 13, 2024 14:35

Added clickhouse state store

798489e

Signed-off-by: Mehmet TOSUN <[email protected]>

Merge branch 'main' into dapr-state-store-clickhouse

d4ba4a7

Merge branch 'main' into dapr-state-store-clickhouse

a14a0b4

Merge branch 'main' into dapr-state-store-clickhouse

16af996

Merge branch 'main' into dapr-state-store-clickhouse

47b4b8b

Merge branch 'main' into dapr-state-store-clickhouse

2292ac1

Merge branch 'main' into dapr-state-store-clickhouse

f141667

Merge branch 'main' into dapr-state-store-clickhouse

cc80a66

Merge branch 'main' into dapr-state-store-clickhouse

40590ba

Merge branch 'main' into dapr-state-store-clickhouse

8d795ba

Merge branch 'main' into dapr-state-store-clickhouse

1137d31

Merge branch 'main' into dapr-state-store-clickhouse

8cfc8c1

Merge branch 'main' into dapr-state-store-clickhouse

97ab7c7

Merge branch 'main' into dapr-state-store-clickhouse

3f8679f

Merge branch 'main' into dapr-state-store-clickhouse

d541fda

Merge branch 'main' into dapr-state-store-clickhouse

3e2b6b0

Merge branch 'main' into dapr-state-store-clickhouse

8c81e28

Merge branch 'main' into dapr-state-store-clickhouse

43e4645

middt requested review from a team as code owners February 23, 2025 04:10

Merge branch 'main' into dapr-state-store-clickhouse

7c3242e

middt added 3 commits March 14, 2025 10:02

Add conformance tests for ClickHouse state store and improve implemen…

7a33ea7

…tation Signed-off-by: Mehmet TOSUN <[email protected]>

Improve ClickHouse state store with authentication support and connec…

0fcdce3

…tion handling Signed-off-by: Mehmet TOSUN <[email protected]>

Improve ClickHouse state store with authentication support and proper…

2c8934c

… connection handling Signed-off-by: Mehmet TOSUN <[email protected]>

middt force-pushed the dapr-state-store-clickhouse branch from 1b2a8de to 2c8934c Compare March 14, 2025 07:02

middt added 2 commits March 14, 2025 10:05

Merge branch 'main' into dapr-state-store-clickhouse

441acfd

Signed-off-by: Mehmet TOSUN <[email protected]>

Merge branch 'main' into dapr-state-store-clickhouse

c7c2507

Signed-off-by: Mehmet TOSUN <[email protected]>

JoshVanL previously approved these changes Aug 29, 2025

View reviewed changes

dapr-bot and others added 2 commits August 29, 2025 06:45

Merge branch 'main' into dapr-state-store-clickhouse

c23e006

middt dismissed JoshVanL’s stale review via d0b6ea8 August 31, 2025 03:36

Fix deps and align with v1.16

9526231

Signed-off-by: Mehmet TOSUN <[email protected]>

middt force-pushed the dapr-state-store-clickhouse branch from 755a9f2 to 9526231 Compare August 31, 2025 13:50

Fix ClickHouse certification tests and add comprehensive integration …

e35c604

…tests Signed-off-by: Mehmet TOSUN <[email protected]>

middt force-pushed the dapr-state-store-clickhouse branch from 4f3e99d to e35c604 Compare September 1, 2025 07:33

middt and others added 8 commits September 2, 2025 15:56

Fix ClickHouse conformance test URL format

004d30a

Signed-off-by: Mehmet TOSUN <[email protected]>

Merge branch 'main' into dapr-state-store-clickhouse

e221875

Merge branch 'main' into dapr-state-store-clickhouse

1e07076

Merge branch 'main' into dapr-state-store-clickhouse

ffc1a92

Merge branch 'main' into dapr-state-store-clickhouse

0628cd6

Merge branch 'main' into dapr-state-store-clickhouse

1de3b44

Merge branch 'main' into dapr-state-store-clickhouse

10423ae

Merge branch 'main' into dapr-state-store-clickhouse

ab6b744

github-actions bot added the stale label Nov 8, 2025

github-actions bot closed this Nov 15, 2025

yaron2 reopened this Nov 15, 2025

github-actions bot removed the stale label Nov 15, 2025

dapr-bot added 4 commits November 15, 2025 10:08

Merge branch 'main' into dapr-state-store-clickhouse

233a6ec

Merge branch 'main' into dapr-state-store-clickhouse

19ec6de

Merge branch 'main' into dapr-state-store-clickhouse

d7ccb49

Merge branch 'main' into dapr-state-store-clickhouse

322e946

sicoyle reviewed Dec 9, 2025

View reviewed changes

	Copyright 2021 The Dapr Authors
	Copyright 2025 The Dapr Authors

Dapr state store clickhouse #3675

Are you sure you want to change the base?

Dapr state store clickhouse #3675

Conversation

middt commented Feb 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Implementation Details

Checklist

Testing Done

Additional Notes

Uh oh!

JoshVanL commented Mar 13, 2025

Uh oh!

middt commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoshVanL commented Apr 9, 2025

Uh oh!

JoshVanL commented Aug 29, 2025

Uh oh!

middt commented Aug 31, 2025

Uh oh!

github-actions bot commented Nov 8, 2025

Uh oh!

github-actions bot commented Nov 15, 2025

Uh oh!

sicoyle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sicoyle commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

middt commented Feb 23, 2025 •

edited

Loading

middt commented Mar 14, 2025 •

edited

Loading