- Introduction
- Getting Started / Installation
- Stability Status
- Configuration
- Autocompletion
- Transactions
- Group Consuming
- API at a Glance
- Examples
kcl is a complete, pure Go command line Kafka client. Think of it as your one stop shop to do anything you want to do with Kafka. Producing, consuming, transacting, administrating, and so on.
Unlike the small size of kafkacat, this binary is ~12M compiled. It is, however, still fast, has rich consuming and producing formatting options, and a complete Kafka administration interface.
I consider the current API relatively stable. Once this hits a 1.x release, the API will be even more stable. I would like to get some definitive broader usage of the client before deeming things unchanging.
I've spent a good amount of time integration testing my franz-go client that this program uses. The main thing I have currently been unable to test is closest replica fetching, which is only theoretically supported. It is worth it to read the stability status in the franz-go repo as well if using this client.
I would love confirmation that this program has been used more broadly, and would love to start a "Users" section below. With this confirmation, I will push a 1.x release.
If you have a go installation, you can simply
go install github.com/twmb/kcl@latest
This will install kcl from the latest release. You can optionally suffix the
go get with @v#.#.# to install a specific version.
Otherwise, you can download a release from the releases page.
kcl supports configuration through a config file, environment variables, and
config flag overrides, with the config values being defined in that order.
By default, kcl searches your OS's user config dir for a kcl directory and
a config.toml file in that directory. The default path can be overridden.
As well, multiple configs can easily swapped between with kcl myconfig.
The configuration supports TLS, SASL (currently PLAIN and SCRAM), seed brokers, and a timeout for requests that take timeouts.
To learn more about configuration, use kcl myconfig help.
Thanks to cobra, autocompletion exists for bash, zsh, and powershell.
As an example of what to put in your .bashrc,
if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
. /etc/bash_completion
. <(kcl misc gen-autocomplete -kbash)
fiTransactions are supported by consuming a batch of records, executing a command and passing the records to the command's STDIN, reading the modified records via the command's STDOUT, and publishing those records.
Input to the program and output from the program is controlled through the same
syntax as consuming and producing, and the --rw flag is a shortcut to say that
the input and output will use the same format.
As an example, the following command:
kcl transact --rw '%V{b4}%v' -dtxn -g group -t foo -x mytxn -v ./command
reads topic foo in group group, executes ./command, writes all record
values to it prefixed with the four byte big endian value length, reads
back records in the same format, and produces to topic txn all using the
transactional id mytxn.
Additionally, you can use the special command "mirror" to have poor man's mirrormarker:
kcl transact -x mytxn -g group -t srcTopic -d destTopic -v mirror
Group consuming is supported with the -g or --group flag to kcl consume
or kcl transact. The default balancer is the cooperative-sticky balancer,
which was introduced with incremental rebalancing in Kafka 2.4.0. This balancer
is incompatible with the previous eager balancers (roundrobin, range, sticky),
thus if you are using kcl with existing groups that have members using eager
balancing strategies, be sure to specify a different balancer.
Be sure to help any command before using it to understand the full syntax.
kcl
consume -- consume records
produce -- produce records
transact -- transactional consuming & producing
metadata -- print broker, cluster, and topic information
group
list -- list consumer groups
describe -- describe consumer groups
delete -- delete consumer groups
offset-delete -- forcefully delete committed offsets a group (see KIP-496)
topic
create -- create topics
delete -- delete topics
add-partitions -- add partitions to topics
misc
api-versions -- print api versions for requests
probe-version -- probe for the currently running Kafka version
gen-autocomplete -- generate cli autocompletion
errcode -- print the error name and desc for an error number
errtext -- print the error name and desc for an error code / all errors
raw-req -- issue a raw request from input JSON
list-offsets -- list offsets for topics and partitions
admin
delete-records -- delete record deletion for partitions based off input offsets
elect-leaders -- trigger leader elections
acl
create -- create ACLs
describe -- describe ACLs
delete -- delete ACLs
client-quotas
alter -- alter client quotas
describe -- describe client quotas
configs
alter -- alter broker, topic, broker-logger, etc. configs
describe -- describe broker, topic, broker-logger, etc. configs
dtoken
create -- create delegation tokens
renew -- renew delegation tokens
describe -- describe delegation tokens
expire -- expire delegation tokens
group -- duplicate of `kcl group` top level command
logdirs
alter -- alter log directories that partitions are in
describe -- describe log directories that partitions are in
partas
alter -- alter partition assignments
list -- list partition reassignments
topic -- duplicate of `kcl topic` top level command
user-scram
alter -- alter user scram
describe -- describe user scram
myconfig
unlink -- unlink the kcl config symlink
link -- link a kcl config symlink
dump -- dump the kcl configuration
help -- print kcl configuration help
ls -- list files in the kcl config directory
kcl consume foo
kcl consume foo -f "KEY=%k, VALUE=%v, HEADERS=%{%h{ '%k'='%v' }}\n"
kcl consume -g grup foo bar
echo fubar | kcl produce foo
kcl produce foo < baz
echo barfoo | kcl produce foo -f'%K{3}%V{3}%v%k\n'
echo "key: bizzy, value: bazzy" | kcl produce foo -f 'key: %k, value: %v\n'
echo "1 k 1 v 2 2 h1 2 v1 2 h2 2 v2 " | kcl produce foo -f '%K %k %V %v %H %h{%K %k %V %v }\n'