kcl

Contents

• Introduction
• Getting Started / Installation
• Stability Status
• Configuration
• Autocompletion
• Transactions
• Group Consuming
• API at a Glance
• Examples

Introduction

kcl is a complete, pure Go command line Kafka client. Think of it as your one stop shop to do anything you want to do with Kafka. Producing, consuming, transacting, administrating, and so on.

Unlike the small size of kafkacat, this binary is ~12M compiled. It is, however, still fast, has rich consuming and producing formatting options, and a complete Kafka administration interface.

Stability Status

I consider the current API relatively stable. Once this hits a 1.x release, the API will be even more stable. I would like to get some definitive broader usage of the client before deeming things unchanging.

I've spent a good amount of time integration testing my franz-go client that this program uses. The main thing I have currently been unable to test is closest replica fetching, which is only theoretically supported. It is worth it to read the stability status in the franz-go repo as well if using this client.

I would love confirmation that this program has been used more broadly, and would love to start a "Users" section below. With this confirmation, I will push a 1.x release.

Getting Started

If you have a go installation, you can simply

go install github.com/twmb/kcl@latest

This will install kcl from the latest release. You can optionally suffix the go get with @v#.#.# to install a specific version.

Otherwise, you can download a release from the releases page.

Configuration

kcl supports configuration through a config file, environment variables, and config flag overrides, with the config values being defined in that order. By default, kcl searches your OS's user config dir for a kcl directory and a config.toml file in that directory. The default path can be overridden. As well, multiple configs can easily swapped between with kcl myconfig.

The configuration supports TLS, SASL (currently PLAIN and SCRAM), seed brokers, and a timeout for requests that take timeouts.

To learn more about configuration, use kcl myconfig help.

Autocompletion

Thanks to cobra, autocompletion exists for bash, zsh, and powershell.

As an example of what to put in your .bashrc,

if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
    . /etc/bash_completion
    . <(kcl misc gen-autocomplete -kbash)
fi

Transactions

Transactions are supported by consuming a batch of records, executing a command and passing the records to the command's STDIN, reading the modified records via the command's STDOUT, and publishing those records.

Input to the program and output from the program is controlled through the same syntax as consuming and producing, and the --rw flag is a shortcut to say that the input and output will use the same format.

As an example, the following command:

kcl transact --rw '%V{b4}%v' -dtxn -g group -t foo -x mytxn -v ./command

reads topic foo in group group, executes ./command, writes all record values to it prefixed with the four byte big endian value length, reads back records in the same format, and produces to topic txn all using the transactional id mytxn.

Additionally, you can use the special command "mirror" to have poor man's mirrormarker:

kcl transact -x mytxn -g group -t srcTopic -d destTopic -v mirror

Group Consuming

Group consuming is supported with the -g or --group flag to kcl consume or kcl transact. The default balancer is the cooperative-sticky balancer, which was introduced with incremental rebalancing in Kafka 2.4.0. This balancer is incompatible with the previous eager balancers (roundrobin, range, sticky), thus if you are using kcl with existing groups that have members using eager balancing strategies, be sure to specify a different balancer.

API at a glance

Be sure to help any command before using it to understand the full syntax.

kcl
 consume                            -- consume records

 produce                            -- produce records

 transact                           -- transactional consuming & producing

 metadata                           -- print broker, cluster, and topic information

 group
   list                             -- list consumer groups
   describe                         -- describe consumer groups
   delete                           -- delete consumer groups
   offset-delete                    -- forcefully delete committed offsets a group (see KIP-496)

 topic
   create                         -- create topics
   delete                         -- delete topics
   add-partitions                 -- add partitions to topics

 misc
   api-versions                     -- print api versions for requests
   probe-version                    -- probe for the currently running Kafka version
   gen-autocomplete                 -- generate cli autocompletion
   errcode                          -- print the error name and desc for an error number
   errtext                          -- print the error name and desc for an error code / all errors
   raw-req                          -- issue a raw request from input JSON
   list-offsets                     -- list offsets for topics and partitions

 admin
   delete-records                   -- delete record deletion for partitions based off input offsets
   elect-leaders                    -- trigger leader elections

   acl
     create                         -- create ACLs
     describe                       -- describe ACLs
     delete                         -- delete ACLs

   client-quotas
     alter                          -- alter client quotas
     describe                       -- describe client quotas

   configs
     alter                          -- alter broker, topic, broker-logger, etc. configs
     describe                       -- describe broker, topic, broker-logger, etc. configs

   dtoken
     create                         -- create delegation tokens
     renew                          -- renew delegation tokens
     describe                       -- describe delegation tokens
     expire                         -- expire delegation tokens

   group                            -- duplicate of `kcl group` top level command

   logdirs
     alter                          -- alter log directories that partitions are in
     describe                       -- describe log directories that partitions are in

   partas
     alter                          -- alter partition assignments
     list                           -- list partition reassignments

   topic                            -- duplicate of `kcl topic` top level command

   user-scram
     alter                          -- alter user scram
     describe                       -- describe user scram

 myconfig
   unlink                           -- unlink the kcl config symlink
   link                             -- link a kcl config symlink
   dump                             -- dump the kcl configuration
   help                             -- print kcl configuration help
   ls                               -- list files in the kcl config directory

Examples

Consuming

...from topic foo, printing the values of records

kcl consume foo

...from topic foo with advanced printing

kcl consume foo -f "KEY=%k, VALUE=%v, HEADERS=%{%h{ '%k'='%v' }}\n"

...from topics foo and bar with group grup

kcl consume -g grup foo bar

Producing

...a newline delimited value to topic foo

echo fubar | kcl produce foo

...a bunch of newline delimited values in file baz to topic foo

kcl produce foo < baz

...key bar, value foo to topic foo in an obscurely formatted way

echo barfoo | kcl produce foo -f'%K{3}%V{3}%v%k\n'

...key bizzy, value bazzy to topic foo, pulling the key and value from a line

echo "key: bizzy, value: bazzy" | kcl produce foo  -f 'key: %k, value: %v\n'

...key k, value v, headers k1 v1 and k2 v2 to topic foo, with ascii length prefixed strings

echo "1 k 1 v 2 2 h1 2 v1 2 h2 2 v2 " | kcl produce foo -f '%K %k %V %v %H %h{%K %k %V %v }\n'