Do we want to have a functionality like clone topic to a file (backup/restore) ?

[gotha]

Hi,

just wanted to check if somebody is interested in backup/restore to/from file functionality?
Something like the existing clone functionality but with file as intermediate step.

I can potentially try to help implementing such a feature, but wanted to check if you think it is a good idea in the first place?

[d-rk]

Hi @gotha

to be honest, I don't see many people using such a feature.
Currently, you can already export the structure of a topic with

kafkactl describe topic my-topic -o json > my-topic.json

If you really want to implement it, I think it would be ok to add a --file parameter to create topic command
and then read the topic configuration from a json exported by the describe topic command.

Regards,
Dirk

[gotha]

hey @d-rk, thank you for the reply.
Maybe there was a little bit of a mixup probably to my misunderstanding what the "clone" functionality actually does.

The feature I am proposing is similar to what export and import in a database would do - being able to recreate a topic with the same settings and put a snapshot of all of its messages.

Do you think such a a feature makes sense for kafkactl ?

[d-rk]

hey @gotha
Then this issue is similar to #178

For both issues a topics and its content shall be cloned.
In this issue however, with the additional requirement to add an intermediate step to export everything to file and import from there.

I'm not sure if it is worth to implement it in one command, but doing it in multiple commands would require only smaller changes.

I'm thinking of something like this:

kafkactl describe topic my-topic -o json > my-topic-config.json
kafkactl consume my-topic --from-beginning --exit > my-topic-data
kafkactl create topic my-topic-clone --file my-topic-config.json
kafkactl produce my-topic-clone --file=my-topic-data

[gotha]

ok, looks like we are very close to this functionality already.

kafkactl consume my-topic --from-beginning --print-headers --print-keys --exit --output json > my-topic-data

seems to do exactly what I need with one minor issue.

The JSON output is formatted on multiple lines so when I try to execute

kafkactl produce my-topic-clone --file=my-topic-data

the messages are not properly created.

If I change the json.MarshalIndent to json.Marshal in internal/output/output.go, it works exactly as I would want for that use case.

I assume you would not like to change the default json output format, so would you recommend to add new format, probably something like --output=json-raw that would output every json object on a single line? Or maybe something else?

Thank you.

[gotha]

now that we have the feature to create a topic from file, the only thing left to be able to complete this functionality is to be able to export messages in a json file one message per line, so they can be read by kafkactl produce

@d-rk what would be your recommendation to proceed on this?
does #213 look good to you?