Doc 1602: Cloud Topics (Beta)

docs-data/property-overrides.json

@@ -469,31 +469,40 @@
       "config_scope": "cluster"
     },
     "cloud_topics_disable_reconciliation_loop": {
-      "exclude_from_docs": true,
       "config_scope": "cluster"
     },
     "cloud_topics_enabled": {
-      "exclude_from_docs": true,
+      "description": "Enable Cloud Topics for the cluster. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.",
+      "related_topics": [
+        "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]",
+        "xref:reference:properties/topic-properties.adoc#redpandacloudtopicenabled[`redpanda.cloud_topic.enabled`]"
+      ],
       "config_scope": "cluster"
     },
     "cloud_topics_long_term_garbage_collection_interval": {
-      "exclude_from_docs": true,
       "config_scope": "cluster"
     },
     "cloud_topics_produce_batching_size_threshold": {
-      "exclude_from_docs": true,
       "config_scope": "cluster"
     },
     "cloud_topics_produce_cardinality_threshold": {
-      "exclude_from_docs": true,
       "config_scope": "cluster"
     },
     "cloud_topics_produce_upload_interval": {
-      "exclude_from_docs": true,
+      "description": "Time interval after which data is uploaded to object storage for Cloud Topics. When this time threshold is reached, Redpanda triggers an upload of buffered data to the object storage backend (S3, GCS, or MinIO), regardless of whether the size or cardinality thresholds have been met.\n\nThis property works together with <<cloud_topics_produce_batching_size_threshold,`cloud_topics_produce_batching_size_threshold`>> and <<cloud_topics_produce_cardinality_threshold,`cloud_topics_produce_cardinality_threshold`>> to control when uploads occur. An upload is triggered when any of these three thresholds is reached.",
+      "related_topics": [
+        "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]",
+        "xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_batching_size_threshold[`cloud_topics_produce_batching_size_threshold`]",
+        "xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_cardinality_threshold[`cloud_topics_produce_cardinality_threshold`]"
+      ],
       "config_scope": "cluster"
     },
     "cloud_topics_reconciliation_interval": {
-      "exclude_from_docs": true,
+      "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud.\n\nThis reconciliation process is essential for Cloud Topics to achieve their cost-efficiency goals by minimizing the amount of expensive local storage required, while ensuring data remains accessible from durable cloud storage.",
+      "related_topics": [
+        "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]",
+        "xref:reference:properties/cluster-properties.adoc#cloud_topics_long_term_garbage_collection_interval[`cloud_topics_long_term_garbage_collection_interval`]"
+      ],
       "config_scope": "cluster"
     },
     "cluster_id": {
@@ -1542,9 +1551,13 @@
       "category": "redpanda"
     },
     "redpanda.cloud_topic.enabled": {
+      "description": "Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.\n\nIMPORTANT: You can only set this property when creating a topic. After a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <<cloudtopicsenabled,`cloud_topics_enabled`>> property.",
+      "related_topics": [
+        "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]",
+        "xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`]"
+      ],
       "config_scope": "topic",
-      "category": "tiered-storage",
-      "exclude_from_docs": true
+      "category": "tiered-storage"
     },
     "redpanda.iceberg.delete": {
       "description": "Whether the corresponding Iceberg table is deleted upon deleting the topic.",
@@ -1678,8 +1691,8 @@
         "xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe`]",
         "xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`]",
         "xref:reference:properties/cluster-properties.adoc#default_topic_replication[`default_topic_replication`]",
-        "xref:develop:config-topics.adoc#choose-the-replication-factor[Choose the replication factor]",
-        "xref:develop:config-topics.adoc#change-the-replication-factor[Change the replication factor]",
+        "xref:develop:manage-topics.adoc#choose-the-replication-factor[Choose the replication factor]",
+        "xref:develop:manage-topics.adoc#change-the-replication-factor[Change the replication factor]",
         "xref:reference:properties/cluster-properties.adoc#default_topic_replication[default_topic_replication]"
       ],
       "config_scope": "topic"
@@ -2060,7 +2073,7 @@
     "write.caching": {
       "description": "The write caching mode to apply to a topic.\n\nWhen `write.caching` is set, it overrides the cluster property xref:cluster-properties.adoc#write_caching_default[`write_caching_default`]. Write caching acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. With `acks=all`, this provides lower latency while still ensuring that a majority of brokers acknowledge the write. Fsyncs follow <<flushms, `flush.ms`>> and <<flushbytes, `flush.bytes`>>, whichever is reached first.",
       "related_topics": [
-        "xref:develop:config-topics.adoc#configure-write-caching[Write caching]",
+        "xref:develop:manage-topics.adoc#configure-write-caching[Write caching]",
         "xref:manage:tiered-storage.adoc[Tiered Storage]",
         "xref:reference:properties/cluster-properties.adoc#write_caching_default[`write_caching_default`]",
         "xref:cluster-properties.adoc#write_caching_default[`write_caching_default`]"
@@ -2070,7 +2083,7 @@
     "write_caching_default": {
       "related_topics": [
         "xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`]",
-        "xref:develop:config-topics.adoc#configure-write-caching[Write caching]"
+        "xref:develop:manage-topics.adoc#configure-write-caching[Write caching]"
       ],
       "config_scope": "cluster",
       "description": "The default write caching mode to apply to user topics. Write caching acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. With `acks=all`, this provides lower latency while still ensuring that a majority of brokers acknowledge the write. \n\nFsyncs follow <<raft_replica_max_pending_flush_bytes,`raft_replica_max_pending_flush_bytes`>> and <<raft_replica_max_flush_delay_ms,`raft_replica_max_flush_delay_ms`>>, whichever is reached first.\n\nThe `write_caching_default` cluster property can be overridden with the xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`] topic property."

modules/ROOT/nav.adoc

@@ -36,7 +36,9 @@
 ** xref:develop:kafka-clients.adoc[Kafka Compatibility]
 ** xref:develop:benchmark.adoc[]
 ** xref:develop:http-proxy.adoc[]
-** xref:develop:config-topics.adoc[]
+** xref:develop:manage-topics/index.adoc[]
+*** xref:develop:manage-topics/config-topics.adoc[]
+*** xref:develop:manage-topics/cloud-topics.adoc[]
 ** xref:console:ui/edit-topic-configuration.adoc[Edit Topic Configuration]
 ** xref:develop:produce-data/index.adoc[Produce Data]
 *** xref:develop:produce-data/configure-producers.adoc[]

modules/console/pages/ui/edit-topic-configuration.adoc

@@ -19,4 +19,4 @@ include::console:partial$operator-console-version-note.adoc[]
 == Suggested reading
 
 - xref:reference:properties/topic-properties.adoc[]
-- xref:develop:config-topics.adoc[]
+- xref:develop:manage-topics/config-topics.adoc[]

modules/deploy/pages/redpanda/manual/production/dev-deployment.adoc

@@ -8,7 +8,7 @@ You can deploy Redpanda using well-known configuration properties optimized for
 
 [NOTE]
 ====
-* Development mode enables write caching by default. This is a relaxed mode of xref:develop:produce-data/configure-producers.adoc#acksall[`acks=all`] that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. Write caching provides lower latency while still ensuring that a majority of brokers acknowledge the write. For more information, or to disable this, see xref:develop:config-topics.adoc#configure-write-caching[write caching].
+* Development mode enables write caching by default. This is a relaxed mode of xref:develop:produce-data/configure-producers.adoc#acksall[`acks=all`] that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. Write caching provides lower latency while still ensuring that a majority of brokers acknowledge the write. For more information, or to disable this, see xref:develop:manage-topics/config-topics.adoc#configure-write-caching[write caching].
 * Development mode also bypasses `fsync`, acknowledging messages before they're stored to disk. This reduces the durability of messages, could cause potential data loss, and could give unrealistic performance characteristics for a production environment.
 ====
 

modules/develop/pages/kafka-clients.adoc

@@ -60,7 +60,7 @@ endif::[]
 * HTTP Proxy (pandaproxy): Unlike other REST proxy implementations in the Kafka ecosystem, Redpanda HTTP Proxy does not support topic and ACLs CRUD through the HTTP Proxy. HTTP Proxy is designed for clients producing and consuming data that do not perform administrative functions.
 ifdef::env-cloud[]
 +
-* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:get-started:config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their xref:get-started:create-topic.adoc[retention limits].
+* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` cleanup policy. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their retention limits.
 endif::[]
 ifndef::env-cloud[]
 +

modules/develop/pages/manage-topics/cloud-topics.adoc

@@ -0,0 +1,92 @@
+= Manage Cloud Topics
+:description: Cloud Topics are Redpanda topics that enable users to trade off latency for lower costs.
+:page-beta: true
+// tag::single-source[]
+
+ifndef::env-cloud[]
+[NOTE]
+====
+include::shared:partial$enterprise-license.adoc[]
+====
+endif::[]
+
+Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-sensitive, low-throughput workloads (for example, for audit logs), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-AZ networking charges) are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments where streaming latency isn't critical.
+
+Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud storage (S3, ADLS,GCS, MinIO) as the primary mechanism to back up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500 ms - 1 s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective.
+
+== Prerequisites
+
+- xref:get-started:rpk-install.adoc[] v25.3 or later.
+ifndef::env-cloud[]
+- xref:manage:tiered-storage.adoc#set-up-tiered-storage[Enable Tiered Storage] on your Redpanda cluster.
++
+[NOTE]
+====
+When you enable Tiered Storage at the cluster level, by default, all new topics are Tiered Storage (or standard) topics. However, if you plan to use Cloud Topics for all new topics in a Redpanda cluster, be sure to set the following cluster-level properties to `false`:
+
+* config_ref:cloud_storage_enable_remote_write,false,properties/object-storage-properties[]
+* config_ref:cloud_storage_enable_remote_read,false,properties/object-storage-properties[]
+
+This ensures that newly-created Redpanda topics are Cloud Topics by default.
+
+For details, see xref:manage:tiered-storage.adoc#enable-tiered-storage-for-a-cluster[Enable Tiered Storage for a cluster].
+====
+
+- xref:manage:tiered-storage.adoc#configure-object-storage[Configure object storage].
+- Ensure that you have an Enterprise license.
++
+To check your license status, run:
++
+[,bash]
+----
+rpk cluster license info
+----
+
+If you intend to use Docker Compose while working with Cloud Topics, see the https://docs.docker.com/compose/install/[Docker Compose installation documentation^]. To verify you have it installed, run:
+
+[source,bash]
+----
+docker compose version
+----
+
+You should see the installed Docker Compose version.
+
+For more details about using Docker Compose with Redpanda, see the xref:get-started:quick-start.adoc[Redpanda Self-Managed Quickstart].
+
+endif::[]
+
+== Limitations
+
+The following features are not supported in this beta release of Cloud Topics:
+
+- Kafka compaction
+- Iceberg topics
+- Topic recovery
+- Remote Read Replicas
+
+IMPORTANT: You should expect to throw away any cluster running Cloud Topics (beta) after it becomes generally available (GA). Redpanda may be unable to continue using Cloud Topics created with the beta version, and you may be unable to upgrade to future versions. Also, any topic specified to be a Cloud Topic cannot be subsequently converted back to a standard Redpanda topic that uses local or Tiered Storage.
+
+== Enable Cloud Topics for your cluster
+
+You must first enable Cloud Topics for your cluster:
+
+[,bash]
+----
+rpk cluster config set unstable_beta_feature_cloud_topics_enabled=true
+Successfully updated configuration. New configuration version is 3.
+----
+
+Once enabled, create the Cloud Topic:
+
+[,bash]
+----
+rpk topic create -c cloud_topics_enabled=true <new-cloud-topic-name>
+TOPIC                    STATUS
+audit.analytics.may2025  OK
+----
+
+You can make a topic a Cloud Topic only at topic creation time.
+
+NOTE: In addition to replication, cross-AZ ingress (producer) and egress (consumer) traffic can also contribute substantially to cloud networking cost. When running multi-AZ clusters in general, Redpanda strongly recommends using xref:../consume-data/follower-fetching.adoc[Follower Fetching], which allows consumers to avoid crossing network zones. When possible, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, providing a similar benefit for ingress traffic. These features can add additional savings to the replication cost savings of Cloud Topics.
+
+// end::single-source[]

modules/develop/pages/config-topics.adoc → modules/develop/pages/manage-topics/config-topics.adoc

@@ -1,6 +1,7 @@
 = Manage Topics
 :page-categories: Clients, Development
 :description: Learn how to create topics, update topic configurations, and delete topics or records.
+:page-aliases: develop:config-topics.adoc
 // tag::single-source[]
 
 include::develop:partial$topic-defaults.adoc[]
@@ -88,6 +89,7 @@ NOTE: The replication factor can't exceed the number of Redpanda brokers. If you
 
 endif::[]
 
+
 === Change the cleanup policy
 
 The cleanup policy determines how to clean up the partition log files when they reach a certain size:

modules/develop/pages/manage-topics/index.adoc

@@ -0,0 +1,4 @@
+= Topics
+:page-categories: Clients, Development
+:description: Learn how to manage topics in Redpanda, including creation, configuration, and advanced features.
+:page-layout: index

modules/develop/pages/produce-data/configure-producers.adoc

@@ -74,7 +74,7 @@ Kafka, a message is considered acknowledged without the requirement that it has
 been fsynced. Messages that have not been fsynced to disk may be lost in the
 event of a broker crash. So when using `acks=all`, the Redpanda default
 configuration is more resilient than Kafka's. You can also consider
-using xref:develop:config-topics.adoc#configure-write-caching[write caching], which is a relaxed mode of `acks=all` that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. 
+using xref:develop:manage-topics/config-topics.adoc#configure-write-caching[write caching], which is a relaxed mode of `acks=all` that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. 
 
 endif::[]
 

modules/get-started/pages/quick-start.adoc

@@ -455,7 +455,7 @@ docker exec -it redpanda-0 rpk topic create chat-room \
   -X pass=secretpassword
 ```
 +
-<1> Set a replication factor of 3 to replicate the topic across all 3 brokers. This replication factor provides high availability and data durability. For more details, see xref:develop:config-topics.adoc#choose-the-replication-factor[Choose the replication factor].
+<1> Set a replication factor of 3 to replicate the topic across all 3 brokers. This replication factor provides high availability and data durability. For more details, see xref:develop:manage-topics/config-topics.adoc#choose-the-replication-factor[Choose the replication factor].
 <2> Enable remote reads for this topic to read offloaded records from object storage.
 <3> Enable remote writes for this topic to offload older records to object storage. For more details, see xref:manage:tiered-storage.adoc[].
 +

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

@@ -12,7 +12,7 @@ image::shared:compaction-example.png[Example of topic compaction]
 
 This diagram illustrates a compacted topic. Imagine a remote sensor network that uses image recognition to track appearances of red pandas in a geographic area. The sensor network employs special devices that send records to a topic when they detect one. You might enable compaction to reduce topic storage while still maintaining a record in the topic of the last time each device saw a red panda, perhaps to see if they stop frequenting a given area. The left side of the diagram shows all records sent across the topic. The right side illustrates the results of compaction; older records for certain keys are deleted from the log.
 
-NOTE: If your application requires consuming every record for a given key, consider using the `delete` xref:develop:config-topics#change-the-cleanup-policy.adoc[cleanup policy] instead.
+NOTE: If your application requires consuming every record for a given key, consider using the `delete` xref:develop:manage-topics/config-topics.adoc#change-the-cleanup-policy[cleanup policy] instead.
 
 IMPORTANT:  When using xref:manage:tiered-storage.adoc[Tiered Storage], compaction functions at the local storage level. As long as a segment remains in local storage, its records are eligible for compaction. Once a segment is uploaded to object storage and removed from local storage it is not retrieved for further compaction operations. A key may therefore appear in multiple segments between Tiered Storage and local storage.
 

modules/manage/pages/cluster-maintenance/disk-utilization.adoc

@@ -85,7 +85,7 @@ Redpanda runs a log cleanup process in the background to apply these policy sett
 See also:
 
 * xref:manage:tiered-storage.adoc#manage-local-capacity-for-tiered-storage-topics[Manage local capacity for Tiered Storage topics]
-* xref:develop:config-topics.adoc#delete-records-from-a-topic[Delete records from a topic]
+* xref:develop:manage-topics/config-topics.adoc#delete-records-from-a-topic[Delete records from a topic]
 
 [[set-time-based-retention]]
 === Set time-based retention