Restore issue for a topic with multiple partitions

[alessio-torelli]

Hi,

we're having issues in restoring topics with multiple partitions.

kafka version: 2.6.0

I'm backing up topic 2nd_topic using the docker version of the component using this command:

  docker run -d -v /path/to/backup_local_kafka/:/kafka-backup/ \
    itadventurer/kafka-backup:latest \
    backup-standalone.sh --bootstrap-server host.docker.internal:9092 \
    --target-dir /kafka-backup/ --topics '2nd_topic' 

The topic has 3 partitions.
Then I stop the container running the backup, delete the topic and run a restore using:

docker run -v /path/to/backup_local_kafka/:/kafka-backup/ \
   itadventurer/kafka-backup:latest \
   restore-standalone.sh --bootstrap-server host.docker.internal:9092 \
   --source-dir /kafka-backup/ --topics '2nd_topic'

The restore does not work and these are the logs:

[2021-04-06 14:51:46,206] ERROR WorkerSourceTask{id=backup-source-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter)
[2021-04-06 14:52:01,207] ERROR WorkerSourceTask{id=backup-source-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask)

The topic is recreated by the restore but does not have the right number of partitions:

bin/kafka-topics.sh --bootstrap-server=localhost:9092 --describe --topic 2nd_topic
Topic: 2nd_topic	PartitionCount: 1	ReplicationFactor: 1	Configs: segment.bytes=1073741824
	Topic: 2nd_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

The same process with topics with only one partition works and the restore container ends correctly.

Do you have suggestions on how to deal with this error?

[itadventurer]

Kafka Backup has no support for recreating the topics. You need to do it yourself. Afterwards, you can use Kafka Backup to fill the topics again with the data.
But by the way, that would be a nice feature to have