Push/Pull on Kafka over HTTP
You can install kafka with pip:
pip install kafka
Alternatively, you can build and use the debian package.
To set up flasfka to talk to the Kafka cluster, the following variables
are available. Put this in a file (for example /etc/flasfka.conf.py):
# Hosts in the kafka cluster (list of ip:port) HOSTS=["localhost:9092"] # You probably don't need to change these # How long to wait when polling a topic for new messages CONSUMER_TIMEOUT=0.1 # Maximum of messages returned when consuming from a topic CONSUMER_LIMIT=100 # When you don't specify a group for listening to a topic, flasfka # will use this one DEFAULT_GROUP="flasfka"
The file will be read at flasfka startup, provided it is in the
environment variable FLASFKA_CONFIG:
export FLASFKA_CONFIG=/etc/flasfka.conf.py
Running flasfka is then just a matter of launching:
flasfka-serve
By default, it runs on the port 5000. Use flasfka-serve -h to
learn how to change that. The debian package wraps flasfka into a uwsgi
service, so you don't need to worry about this if you are using it.
Now, assuming flasfka is configured and running:
curl -X POST --data-binary '{"messages": ["msg1", "msg2"]}' "http://127.0.0.1:5000/my-topic/"
This pushes msg1 then msg2 to the topic my-topic.
curl -X POST --data-binary '{"messages": ["msg3"]}' "http://127.0.0.1:5000/my-topic/my-key/"
This pushes msg3 to the topic my-topic with the key my-key.
curl http://127.0.0.1:5000/my-topic/
This retrieves a maximum of 100 of the last messages posted to
my-topic, from the default group flasfka.
curl http://127.0.0.1:5000/my-topic/my-group/
This retrieves a maximum of 100 of the last messages posted to
my-topic, from the group my-group.
curl http://127.0.0.1:5000/my-topic/?limit=20
This retrieves a maximum of 20 of the last messages posted to
my-topic, from the default group flasfka.
Flafska only accepts json data, so your messages need to be in utf-8. If you want to pass arbitrary data, it is recommended to encode them to a suitable format before (e.g. base64).