You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While standard Kafka glossterm:topics[] are ideal for high-throughput, cost-sensitive workloads, Cloud Topics are optimized for high-throughput, cost-sensitive workloads. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, GCS, MinIO) as the primary mechanism for backing up data. The difference in response time is typically 500ms, which is only impactful for latency-sensitive applications. So when using Cloud Topics, you get the same durability that you have with standard topics, but you also eliminate the significant cross-AZ costs typically incurred when you have high-throughput workloads (for example, for audit logs).
14
+
Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible. While standard Kafka xref:config-topics.adoc[topics] are ideal for latency-sensitive, high-throughput workloads (for example, for audit logs), Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, GCS, MinIO) as the primary mechanism for backing up data. The difference in response time is typically 500ms, which is only impactful for latency-sensitive applications.
15
15
16
16
== Prerequisites
17
17
18
-
- rpk v25.3.1: See xref:get-started:rpk-install.adoc[].
18
+
- xref:get-started:rpk-install.adoc[] v25.3 or later.
19
19
ifndef::env-cloud[]
20
20
- xref:manage:tiered-storage.adoc#set-up-tiered-storage[Set up] and xref:manage:tiered-storage.adoc#enable-tiered-storage[enable Tiered Storage] on your Redpanda cluster.
To check if you already have a license key applied to your cluster:
25
25
@@ -38,10 +38,16 @@ Not all Redpanda features are available when using Cloud Topics (beta). The foll
38
38
- Topic recovery
39
39
- Remote Read Replicas
40
40
41
+
Also note that Cloud Topics do not work in simple Docker containers unless Cloud storage is set up in them.
42
+
41
43
IMPORTANT: You should expect to throw away any cluster running the Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be a Cloud Topic cannot subsequently be converted back to a standard, or non-Cloud Topic.
42
44
43
45
== Create Cloud Topics
44
46
47
+
You can make a topic a Cloud Topic only at creation time. This is a two-step process where you first create the Cloud Topic, then enable that Cloud Topic in your cluster.
48
+
49
+
IMPORTANT: Once you create a Cloud Topic, you cannot subsequently convert it back into a standard Kafka topic.
50
+
45
51
To create a Cloud Topic:
46
52
47
53
[,bash]
@@ -59,7 +65,9 @@ To enable a Cloud Topic in a cluster:
59
65
60
66
[,bash]
61
67
----
62
-
rpk cluster config set unstable_beta_feature_cloud_topics_enabled = true
68
+
rpk cluster config set unstable_beta_feature_cloud_topics_enabled=true <topic-name>
63
69
----
64
70
71
+
NOTE: For non-replicated cross-AZ traffic (for example, for producers), if your throughput costs remain elevated, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, and helps eliminate this traffic
0 commit comments