Doc 1602: Cloud Topics (Beta) #1469
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the 📝 WalkthroughWalkthroughThis PR reorganizes Redpanda's topic-related documentation by creating a new Estimated code review effort🎯 2 (Simple) | ⏱️ ~15 minutes
Areas to review:
Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
modules/develop/pages/manage-topics/config-topics.adoc (1)
92-116: Fix CLI flag typo in alter-config example.
The example uses a long dash beforeset. It should be--set(two hyphens) or the command fails.Apply:
- rpk topic alter-config [TOPICS…] —-set cleanup.policy=compact + rpk topic alter-config [TOPICS...] --set cleanup.policy=compactAlso applies to: 105-111
🧹 Nitpick comments (2)
modules/develop/pages/manage-topics/index.adoc (1)
1-4: Nice addition. Consider brief “What’s here” bullets.
Add 2–3 bullets linking to child pages (config-topics, cloud-topics) to help orientation.modules/develop/pages/manage-topics/cloud-topics.adoc (1)
17-24: Beta feature guardrails look good; add cluster-level enable note if required.
If Cloud Topics require a cluster flag (for dev/testing), add a note referencing the relevant cluster property (for example, if any) or document that none is needed.Also applies to: 32-40
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Jira integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (16)
modules/ROOT/nav.adoc(1 hunks)modules/console/pages/ui/edit-topic-configuration.adoc(1 hunks)modules/deploy/pages/redpanda/manual/production/dev-deployment.adoc(1 hunks)modules/develop/pages/kafka-clients.adoc(1 hunks)modules/develop/pages/manage-topics/cloud-topics.adoc(1 hunks)modules/develop/pages/manage-topics/config-topics.adoc(2 hunks)modules/develop/pages/manage-topics/index.adoc(1 hunks)modules/develop/pages/produce-data/configure-producers.adoc(1 hunks)modules/get-started/pages/quick-start.adoc(1 hunks)modules/manage/pages/cluster-maintenance/compaction-settings.adoc(1 hunks)modules/manage/pages/cluster-maintenance/disk-utilization.adoc(1 hunks)modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc(1 hunks)modules/manage/partials/audit-logging.adoc(1 hunks)modules/reference/pages/properties/cluster-properties.adoc(1 hunks)modules/reference/pages/properties/topic-properties.adoc(2 hunks)modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc(1 hunks)
🧰 Additional context used
🧠 Learnings (5)
📓 Common learnings
Learnt from: Feediver1
Repo: redpanda-data/docs PR: 1153
File: modules/reference/pages/properties/topic-properties.adoc:45-50
Timestamp: 2025-07-16T19:33:20.420Z
Learning: In the Redpanda documentation, topic property cross-references like <<max.compaction.lag.ms>> and <<min.compaction.lag.ms>> require corresponding property definition sections with anchors like [[maxcompactionlagms]] and [[mincompactionlagms]] to prevent broken links.
📚 Learning: 2025-07-16T19:33:20.420Z
Learnt from: Feediver1
Repo: redpanda-data/docs PR: 1153
File: modules/reference/pages/properties/topic-properties.adoc:45-50
Timestamp: 2025-07-16T19:33:20.420Z
Learning: In the Redpanda documentation, topic property cross-references like <<max.compaction.lag.ms>> and <<min.compaction.lag.ms>> require corresponding property definition sections with anchors like [[maxcompactionlagms]] and [[mincompactionlagms]] to prevent broken links.
Applied to files:
modules/manage/pages/cluster-maintenance/compaction-settings.adocmodules/reference/pages/properties/cluster-properties.adocmodules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adocmodules/reference/pages/properties/topic-properties.adocmodules/manage/pages/cluster-maintenance/topic-property-configuration.adocmodules/console/pages/ui/edit-topic-configuration.adocmodules/get-started/pages/quick-start.adocmodules/develop/pages/kafka-clients.adocmodules/deploy/pages/redpanda/manual/production/dev-deployment.adoc
📚 Learning: 2025-05-07T01:06:00.937Z
Learnt from: kbatuigas
Repo: redpanda-data/docs PR: 1113
File: modules/manage/partials/iceberg/use-iceberg-catalogs.adoc:100-107
Timestamp: 2025-05-07T01:06:00.937Z
Learning: In AsciiDoc documentation for Redpanda, the syntax `+` and `--` around content blocks within a `[tabs]` section are valid AsciiDoc formatting elements for tabbed content. The `+` after a tab name (like `rpk::`) indicates that the following block belongs to that tab, and the `--` markers enclose the content for that tab. These are not diff artifacts and should not be removed.
Applied to files:
modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc
📚 Learning: 2025-08-25T21:00:26.626Z
Learnt from: micheleRP
Repo: redpanda-data/docs PR: 1334
File: modules/manage/partials/rbac-dp.adoc:93-98
Timestamp: 2025-08-25T21:00:26.626Z
Learning: In cloud documentation (env-cloud), Security is at the top level navigation, so ACL references should use `security:authorization/rbac/acl.adoc`. In self-managed documentation, Security is nested under Manage, so ACL references use `manage:security/authorization/acl.adoc`. The different xref paths in conditional blocks reflect these different navigation structures.
Applied to files:
modules/develop/pages/manage-topics/cloud-topics.adocmodules/manage/pages/cluster-maintenance/topic-property-configuration.adocmodules/ROOT/nav.adoc
📚 Learning: 2025-08-19T19:26:57.364Z
Learnt from: paulohtb6
Repo: redpanda-data/docs PR: 1324
File: modules/deploy/partials/high-availability.adoc:537-537
Timestamp: 2025-08-19T19:26:57.364Z
Learning: In AsciiDoc documentation, page aliases (`:page-aliases:`) should never be removed as they serve as old redirects to maintain backward compatibility when pages are moved or renamed. They preserve SEO and prevent broken links for users.
Applied to files:
modules/develop/pages/manage-topics/config-topics.adoc
🔇 Additional comments (16)
modules/manage/pages/cluster-maintenance/compaction-settings.adoc (1)
15-15: Verify the anchor#change-the-cleanup-policyexists in the reorganized target.This line references an anchor that should exist in the moved config-topics.adoc file. Based on the retrieved learnings, anchors for topic configuration properties and sections must be explicitly defined or auto-generated from section headers.
modules/deploy/pages/redpanda/manual/production/dev-deployment.adoc (1)
11-11: LGTM.The cross-reference update correctly points to the reorganized manage-topics/config-topics.adoc path and maintains the same anchor reference for write caching configuration. The context and surrounding text remain appropriate.
modules/manage/partials/audit-logging.adoc (1)
436-436: LGTM.The cross-reference update correctly reflects the new manage-topics directory structure. Without an anchor fragment, this reference will point to the top-level page, which is appropriate for a general "Related topics" reference.
modules/manage/pages/cluster-maintenance/disk-utilization.adoc (1)
88-88: Verify anchor#delete-records-from-a-topicexists in target file.This anchor reference should correspond to a section in manage-topics/config-topics.adoc covering how to delete records from topics. Based on the PR's learnings about anchors, confirm this section (or an explicit anchor) exists in the reorganized target file.
modules/develop/pages/kafka-clients.adoc (1)
63-63: Verify module prefix change is intentional.This cross-reference is unique among the PR changes: it shifts the module prefix from
get-started:todevelop:in addition to restructuring the path tomanage-topics/config-topics.adoc. This suggests config-topics.adoc has been moved to the develop module (rather than just reorganized within it).Confirm this module relocation aligns with the PR's overall reorganization strategy and that no residual reference in the get-started module should exist.
modules/get-started/pages/quick-start.adoc (1)
457-457: Verify anchor#choose-the-replication-factorexists in target file.This anchor reference is expected to correspond to a section in manage-topics/config-topics.adoc that explains replication factor selection. Confirm this section or an explicit anchor with this ID exists in the reorganized target file to prevent broken links.
modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc (1)
223-223: LGTM.The cross-reference update correctly points to the new manage-topics/config-topics.adoc path. This is a straightforward top-level page reference without an anchor, appropriate for the "Related topics" section context.
modules/develop/pages/produce-data/configure-producers.adoc (1)
77-77: Anchor verification complete—xref is valid.The xref fragment
#configure-write-cachingwill be auto-generated from the section heading "=== Configure write caching" at line 117 of the target file. AsciiDoc/Antora automatically generates anchors from headings using lowercase text with hyphens replacing spaces. The reorganized cross-reference is valid and will not produce broken links.modules/console/pages/ui/edit-topic-configuration.adoc (1)
22-22: LGTM — updated xref path is correct.modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc (1)
13-13: LGTM — write-caching xref now targets the new location.modules/develop/pages/manage-topics/config-topics.adoc (1)
4-4: Good — alias preserves old URL (back-compat).modules/ROOT/nav.adoc (1)
39-41: LGTM — nav reflects new Manage topics hub.modules/reference/pages/properties/topic-properties.adoc (1)
488-489: LGTM — write-caching cross-ref updated to new section.modules/develop/pages/manage-topics/cloud-topics.adoc (2)
41-42: Strong GA migration statement — confirm with PM.
“Throw away any cluster…” is severe. If true for Public Beta, add context (applies only to beta clusters) and guidance on migration; otherwise soften.
14-14: Avoid exact latency claim unless documented.
“The difference in response time is typically 500ms” — either add a “typical, workload-dependent” qualifier with a citation or remove the number.modules/reference/pages/properties/cluster-properties.adoc (1)
6549-6549: LGTM — related topics link updated to the new anchor. Verification complete.Verification confirms:
- The xref at line 6549 in cluster-properties.adoc correctly points to
develop:manage-topics/config-topics.adoc#configure-write-caching- The target anchor exists in config-topics.adoc as an auto-generated anchor from the section heading "=== Configure write caching" (line 117)
- No lingering old-style references to
develop:config-topics.adoc(without the manage-topics path) remain in the codebase
| @@ -0,0 +1,89 @@ | |||
| = Cloud Topics | |||
| :description: Cloud Topics are Kafka topics that enable users to trade off latency for lower costs. | |||
There was a problem hiding this comment.
"Cloud Topics are Redpanda topics that enable users to trade off latency for lower costs."
| ==== | ||
| endif::[] | ||
|
|
||
| Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible. While standard Kafka xref:config-topics.adoc[topics] are ideal for latency-sensitive, high-throughput workloads (for example, for audit logs), Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, GCS, MinIO) as the primary mechanism for backing up data. The difference in response time is typically 500ms, which is only impactful for latency-sensitive applications. |
There was a problem hiding this comment.
"Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] using local storage or tiered storage are ideal for latency-sensitive, lower throughput workloads, Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-zone networking charges) are a major consideration and can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds or just development environments where streaming latency isn't critical. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, , ADLS, GCS, MinIO etc) as the primary mechanism for storing topic data, eliminating over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500ms-1s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that were previously not cost effective. Redpanda also allows you to mix and match cloud topics workloads with low latency workloads in the same cluster, and will (at GA time) allow you to migrate existing local or tiered storage topics to become cloud topics.
There was a problem hiding this comment.
@Feediver1 re-watching Noah's demo, I think it does make sense to include some of these rough cost savings numbers somewhere (though we should be clear this is just replication costs), e.g. something like this:
"The infrastructure cost savings with Cloud Topics can be quite dramatic. By way of example, in one cloud environment tested by Redpanda, the networking costs of data replication to run a single 100MBps workload were reduced from over $10,000 per month to ~$80 per month. While this doesn't represent 100% of total operating costs, and noting that kafka batch sizes can affect this cost reduction slightly, at high throughputs this savings still accounts for the vast majority of cloud provider expense, reducing the total infrastructure cost of running Kafka workloads by an order of magnitude or more."
| + | ||
| [NOTE] | ||
| ==== | ||
| Be sure to set the following cluster-level properties to `false` to disable Tiered Storage for new topics: |
There was a problem hiding this comment.
"While testing Cloud Topics, If you plan to use Cloud Topics for all topics in a redpanda cluster, be sure to set the following cluster-level properties to false to disable Tiered Storage for newly created topics, as Tiered Storage topics cannot currently be converted into cloud topics"
| - Topic recovery | ||
| - Remote Read Replicas | ||
|
|
||
| Also note that Cloud Topics do not work in simple Docker containers unless Cloud storage is set up in them. |
There was a problem hiding this comment.
@Feediver1 this isn't really true. Even our Quickstarts that use Docker (via Docker compose) explain how to enable Tiered Storage and embed MinIO within them so you can do that, in which case Cloud Topics would work: https://docs.redpanda.com/current/get-started/quick-start.
Since anyone following our docs COULD you cloud topics, we should probably just remove this
|
|
||
| Also note that Cloud Topics do not work in simple Docker containers unless Cloud storage is set up in them. | ||
|
|
||
| IMPORTANT: You should expect to throw away any cluster running Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be a Cloud Topic cannot subsequently be converted back to a standard Kafka topic. |
There was a problem hiding this comment.
'may be unable'
"IMPORTANT: You should expect to throw away any cluster running Cloud Topics (beta) after it becomes generally available (GA). Redpanda may be unable to continue using topics creates with the beta version of this feature and you may be unable to upgrade to future versions. Also, any topic specified to be a Cloud Topic cannot subsequently be converted back to a standard Redpanda topic using local or tiered storage"
|
|
||
| To set up Cloud Topics: | ||
|
|
||
| . Enable the Cloud Topic for your cluster. |
There was a problem hiding this comment.
'Enable Cloud Topics for your cluster"
|
|
||
| . Create the Cloud Topic. | ||
|
|
||
| You can make a topic a Cloud Topic only at creation time. |
There was a problem hiding this comment.
"You can make a topic a Cloud Topic only at topic creation time."
(Maybe this is overkill but want to make sure they dont confuse that with cluster creation time)
|
|
||
| You can make a topic a Cloud Topic only at creation time. | ||
|
|
||
| IMPORTANT: Once you create a Cloud Topic, you cannot subsequently convert it back into a standard Kafka topic. |
There was a problem hiding this comment.
"IMPORTANT: Once you create a Cloud Topic, you cannot subsequently convert it back into a standard Redpanda topic."
|
|
||
| [,properties] | ||
| ---- | ||
| unstable_beta_feature_cloud_topics_enabled: true |
There was a problem hiding this comment.
Should we show the whole command?
| audit.analytics.may2025 OK | ||
| ---- | ||
|
|
||
| NOTE: For non-replicated cross-AZ traffic (for example, for producers), if your throughput costs remain elevated, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, and helps eliminate this traffic. |
There was a problem hiding this comment.
"NOTE: In addition to replication, cross-AZ ingress (Producer) and egress (Consumer) traffic can also contribute substantially to cloud networking cots. When running multi-AZ clusters in general, Redpanda strongly recommends using follower fetching allowing consumers to avoid crossing network zones, and (where possible) use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leaders close to producers, providing a similar benefit for ingress traffic. These features stack additional savings onto the replication cost savings of cloud topics and are useful for all topics in avoiding unnecessary infrastructure costs."
Description
Resolves https://redpandadata.atlassian.net/browse/DOC-1602
Review deadline:
DO NOT MERGE
Page previews
Cloud Topics << URL to share with beta users
Checks