Skip to content

Commit 55eff36

Browse files
Feediver1Feediver1
committed
DOC-1602
1 parent 8a9e883 commit 55eff36

File tree

1 file changed

+276
-0
lines changed

1 file changed

+276
-0
lines changed

‎modules/develop/pages/manage-topics/config-topics.adoc‎

Lines changed: 276 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,276 @@
1+
= Manage Topics
2+
:page-categories: Clients, Development
3+
:description: Learn how to create topics, update topic configurations, and delete topics or records.
4+
:page-aliases: develop:config-topics.adoc
5+
// tag::single-source[]
6+
7+
include::develop:partial$topic-defaults.adoc[]
8+
9+
== Create a topic
10+
11+
Creating a topic can be as simple as specifying a name for your topic on the command line. For example, to create a topic named `xyz`, run:
12+
13+
[,bash]
14+
----
15+
rpk topic create xyz
16+
----
17+
18+
ifndef::env-cloud[]
19+
This command creates a topic named `xyz` with one partition and one replica, because these are the default values set in the cluster configuration file. Replicas are copies of partitions that are distributed across different brokers, so if one broker goes down, other brokers still have a copy of the data.
20+
21+
endif::[]
22+
23+
ifdef::env-cloud[]
24+
This command creates a topic named `xyz` with one partition and three replicas, because these are the default values set in the cluster configuration file. Replicas are copies of partitions that are distributed across different brokers, so if one broker goes down, other brokers still have a copy of the data.
25+
26+
Redpanda Cloud supports 40,000 topics per cluster.
27+
28+
endif::[]
29+
30+
=== Choose the number of partitions
31+
32+
A partition acts as a log file where topic data is written. Dividing topics into partitions allows producers to write messages in parallel and consumers to read messages in parallel. The higher the number of partitions, the greater the throughput.
33+
34+
TIP: As a general rule, select a number of partitions that corresponds to the maximum number of consumers in any consumer group that will consume the data.
35+
36+
For example, suppose you plan to create a consumer group with 10 consumers. To create topic `xyz` with 10 partitions, run:
37+
38+
[,bash]
39+
----
40+
rpk topic create xyz -p 10
41+
----
42+
43+
ifndef::env-cloud[]
44+
=== Choose the replication factor
45+
46+
The default replication factor in the cluster configuration is set to 1. By choosing a replication factor greater than 1, you ensure that each partition has a copy of its data on at least one other broker. One replica acts as the leader, and the other replicas are followers.
47+
48+
To specify a replication factor of 3 for topic `xyz`, run:
49+
50+
[,bash]
51+
----
52+
rpk topic create xyz -r 3
53+
----
54+
55+
NOTE: The replication factor must be an odd number. Redpanda Data recommends a replication factor of 3 for most use cases. Administrators may set a minimum required replication factor for any new topic in the cluster through the cluster-level xref:reference:cluster-properties.adoc#minimum_topic_replications[`minimum_topic_replications`] property.
56+
57+
TIP: If you enable xref:manage:tiered-storage.adoc[Tiered Storage] on a topic, you can then use xref:manage:topic-recovery.adoc[topic recovery] to restore data for a deleted topic.
58+
59+
endif::[]
60+
61+
== Update topic configurations
62+
63+
After you create a topic, you can update the topic property settings for all new data written to it. For example, you can add partitions or change the cleanup policy.
64+
65+
=== Add partitions
66+
67+
You can assign a certain number of partitions when you create a topic, and add partitions later. For example, suppose you add brokers to your cluster, and you want to take advantage of the additional processing power. To increase the number of partitions for existing topics, run:
68+
69+
[,bash]
70+
----
71+
rpk topic add-partitions [TOPICS...] --num [#]
72+
----
73+
74+
Note that `--num <#>` is the number of partitions to _add_, not the total number of partitions.
75+
76+
include::develop:partial$balance-existing-topic-redistribution.adoc[]
77+
78+
ifndef::env-cloud[]
79+
=== Change the replication factor
80+
81+
Suppose you create a topic with the default replication factor of 1 (which is specified in the cluster properties configuration file). Now you want to change the replication factor to 3, so you can have two backups of topic data in case a broker goes down. To set the replication factor to 3, run:
82+
83+
[,bash]
84+
----
85+
rpk topic alter-config [TOPICS...] --set replication.factor=3
86+
----
87+
88+
NOTE: The replication factor can't exceed the number of Redpanda brokers. If you try to set a replication factor greater than the number of brokers, the request is rejected.
89+
90+
endif::[]
91+
92+
93+
=== Change the cleanup policy
94+
95+
The cleanup policy determines how to clean up the partition log files when they reach a certain size:
96+
97+
* `delete` deletes data based on age or log size. Topics retain all records until then.
98+
* `compact` compacts the data by only keeping the latest values for each KEY.
99+
* `compact,delete` combines both methods.
100+
101+
Unlike compacted topics, which keep only the most recent message for a given key, topics configured with a `delete` cleanup policy provide a running history of all changes for those topics.
102+
103+
include::develop:partial$topic-properties-warning.adoc[]
104+
105+
For example, to change a topic's policy to `compact`, run:
106+
107+
[,bash]
108+
----
109+
rpk topic alter-config [TOPICS…] —-set cleanup.policy=compact
110+
----
111+
112+
ifndef::env-cloud[]
113+
For details on compaction in Redpanda, see xref:manage:cluster-maintenance/compaction-settings.adoc[Compaction settings].
114+
115+
endif::[]
116+
117+
=== Configure write caching
118+
119+
Write caching is a relaxed mode of xref:develop:produce-data/configure-producers.adoc#acksall[`acks=all`] that provides better performance at the expense of durability. It acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write.
120+
121+
Write caching applies to user topics. It does not apply to transactions or consumer offsets: data written in the context of a transaction and consumer offset commits is always written to disk and fsynced before being acknowledged to the client.
122+
123+
ifndef::env-cloud[]
124+
NOTE: For clusters in xref:reference:rpk/rpk-redpanda/rpk-redpanda-mode.adoc#development-mode[development mode], write caching is enabled by default. For clusters in production mode, it is disabled by default.
125+
126+
endif::[]
127+
128+
Only enable write caching on workloads that can tolerate some data loss in the case of multiple, simultaneous broker failures. Leaving write caching disabled safeguards your data against complete data center or availability zone failures.
129+
130+
ifndef::env-cloud[]
131+
132+
==== Configure at cluster level
133+
134+
To enable write caching by default in all user topics, set the cluster-level property xref:reference:cluster-properties.adoc#write_caching_default[`write_caching_default`]:
135+
136+
`rpk cluster config set write_caching_default=true`
137+
138+
With `write_caching_default` set to true at the cluster level, Redpanda fsyncs to disk according to xref:reference:cluster-properties.adoc#raft_replica_max_pending_flush_bytes[`raft_replica_max_pending_flush_bytes`] and xref:reference:cluster-properties.adoc#raft_replica_max_flush_delay_ms[`raft_replica_max_flush_delay_ms`], whichever is reached first.
139+
140+
endif::[]
141+
142+
==== Configure at topic level
143+
144+
To override the cluster-level setting at the topic level, set the topic-level property `write.caching`:
145+
146+
`rpk topic alter-config my_topic --set write.caching=true`
147+
148+
With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to `flush.ms` and `flush.bytes`, whichever is reached first.
149+
150+
=== Remove a configuration setting
151+
152+
You can remove a configuration that overrides the default setting, and the setting will use the default value again. For example, suppose you altered the cleanup policy to use `compact` instead of the default, `delete`. Now you want to return the policy setting to the default. To remove the configuration setting `cleanup.policy=compact`, run `rpk topic alter-config` with the `--delete` flag:
153+
154+
[,bash]
155+
----
156+
rpk topic alter-config [TOPICS...] --delete cleanup.policy
157+
----
158+
159+
== List topic configuration settings
160+
161+
To display all the configuration settings for a topic, run:
162+
163+
[,bash]
164+
----
165+
rpk topic describe <topic-name> -c
166+
----
167+
168+
The `-c` flag limits the command output to just the topic configurations. This command is useful for checking the default configuration settings before you make any changes and for verifying changes after you make them.
169+
170+
The following command output displays after running `rpk topic describe test-topic`, where `test-topic` was created with default settings:
171+
172+
ifndef::env-cloud[]
173+
[,bash]
174+
----
175+
rpk topic describe test_topic
176+
SUMMARY
177+
=======
178+
NAME test_topic
179+
PARTITIONS 1
180+
REPLICAS 1
181+
182+
CONFIGS
183+
=======
184+
KEY VALUE SOURCE
185+
cleanup.policy delete DYNAMIC_TOPIC_CONFIG
186+
compression.type producer DEFAULT_CONFIG
187+
max.message.bytes 1048576 DEFAULT_CONFIG
188+
message.timestamp.type CreateTime DEFAULT_CONFIG
189+
redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG
190+
redpanda.remote.delete true DEFAULT_CONFIG
191+
redpanda.remote.read false DEFAULT_CONFIG
192+
redpanda.remote.write false DEFAULT_CONFIG
193+
retention.bytes -1 DEFAULT_CONFIG
194+
retention.local.target.bytes -1 DEFAULT_CONFIG
195+
retention.local.target.ms 86400000 DEFAULT_CONFIG
196+
retention.ms 604800000 DEFAULT_CONFIG
197+
segment.bytes 1073741824 DEFAULT_CONFIG
198+
----
199+
200+
Suppose you add two partitions, and increase the number of replicas to 3. The new command output confirms the changes in the `SUMMARY` section:
201+
202+
[.no-copy]
203+
----
204+
SUMMARY
205+
=======
206+
NAME test_topic
207+
PARTITIONS 3
208+
REPLICAS 3
209+
----
210+
211+
endif::[]
212+
213+
ifdef::env-cloud[]
214+
[,bash]
215+
----
216+
rpk topic describe test_topic
217+
SUMMARY
218+
=======
219+
NAME test_topic
220+
PARTITIONS 1
221+
REPLICAS 3
222+
223+
CONFIGS
224+
=======
225+
KEY VALUE SOURCE
226+
cleanup.policy delete DYNAMIC_TOPIC_CONFIG
227+
compression.type producer DEFAULT_CONFIG
228+
max.message.bytes 1048576 DEFAULT_CONFIG
229+
message.timestamp.type CreateTime DEFAULT_CONFIG
230+
redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG
231+
redpanda.remote.delete true DEFAULT_CONFIG
232+
redpanda.remote.read false DEFAULT_CONFIG
233+
redpanda.remote.write false DEFAULT_CONFIG
234+
retention.bytes -1 DEFAULT_CONFIG
235+
retention.local.target.bytes -1 DEFAULT_CONFIG
236+
retention.local.target.ms 86400000 DEFAULT_CONFIG
237+
retention.ms 604800000 DEFAULT_CONFIG
238+
segment.bytes 1073741824 DEFAULT_CONFIG
239+
----
240+
241+
endif::[]
242+
243+
== Delete a topic
244+
245+
To delete a topic, run:
246+
247+
[,bash]
248+
----
249+
rpk topic delete <topic-name>
250+
----
251+
252+
When a topic is deleted, its underlying data is deleted, too.
253+
254+
To delete multiple topics at a time, provide a space-separated list. For example, to delete two topics named `topic1` and `topic2`, run:
255+
256+
[,bash]
257+
----
258+
rpk topic delete topic1 topic2
259+
----
260+
261+
You can also use the `-r` flag to specify one or more regular expressions; then, any topic names that match the pattern you specify are deleted. For example, to delete topics with names that start with "`f`" and end with "`r`", run:
262+
263+
[,bash]
264+
----
265+
rpk topic delete -r '^f.*' '.*r$'
266+
----
267+
268+
Note that the first regular expression must start with the `^` symbol, and the last expression must end with the `$` symbol. This requirement helps prevent accidental deletions.
269+
270+
include::develop:partial$delete-topic-records.adoc[]
271+
272+
== Next steps
273+
274+
xref:develop:produce-data/configure-producers.adoc[]
275+
276+
// end::single-source[]

0 commit comments

Comments
 (0)