RabbitIO is a tool to backup and restore RabbitMQ messages.
To learn more about common problems with queue management and our reasoning and design principles when building RabbitIO, please read this blog post.
A typical use case is to consume all messages from a RabbitMQ deadletter queue to your local dev machine, do some analysis of the messages, and possibly push (some of) them back to an exchange for reprocessing.
Pick your binary from Releases and download, in addition you'll need to set the binary to be executable.
Example with linux-amd64 and version v0.5.4:
wget https://github.com/meltwater/rabbitio/releases/download/v0.5.4/rabbitio-v0.5.4-linux-amd64 -O rabbitio
chmod 755 rabbitiogo get -u github.com/meltwater/rabbitioAfter installing rabbitio, you can quickly test out rabbitio by using docker-compose.
Included is a docker-compose file to set up local rabbitmq.
cd $GOPATH/src/github.com/meltwater
docker-compose up -dGo to your now running local rabbit and create example exchange rabbitio-exchange and queue rabbitio-queue. Then bind the queue to the exchange.
echo "My first message" > message # write a message into a file
tar cfz message.tgz message # create a tarball containing this message
rabbitio in -e rabbitio-exchange -q rabbitio-queue -f message.tgzThis will publish your first message into rabbitio-exchange and you'll see your message in the queue rabbitio-queue
$ mkdir data
$ rabbitio out -e rabbitio-exchange -q rabbitio-queue -d data/
2018/03/15 15:37:35 RabbitMQ connected: amqp://guest:guest@localhost:5672/
2018/03/15 15:37:35 Bind to Exchange: "rabbitio-exchange" and Queue: "rabbitio-queue", Messaging waiting: 1
^C Interruption, saving last memory bits..
2018/03/15 15:37:45 Wrote 203 bytes to data/1_messages_1.tgz
2018/03/15 15:37:45 tarball writer closingWe interrupt when the queue is empty by directly using a combination of CTRL + C once. This will save the last bits and ack the message.
$ rabbitio
Rabbit IO will help backup and restore your messages in RabbitMQ
Usage:
rabbitio [command]
Available Commands:
help Help about any command
in Publishes documents from tarballs into RabbitMQ exchange
out Consumes data out from RabbitMQ and stores to tarballs
version Prints the version of Rabbit IO
Flags:
-e, --exchange string Exchange to connect to
-h, --help help for rabbitio
-p, --prefetch int Prefetch for batches (default 100)
-q, --queue string Queue to connect to
-r, --routingkey string Routing Key, if specified will override tarball routing key configuration (default "#")
-t, --tag string AMQP Client Tag (default "Rabbit IO Connector ")
-u, --uri string AMQP URI, uri to for instance RabbitMQ (default "amqp://guest:guest@localhost:5672/")
Use "rabbitio [command] --help" for more information about a command.
When you read messages from a queue, the headers as well as the routing key will be saved as metadata in the tarballs, utilizing what in tar is called PAX Records. This is helpful if you one day want to replay the data back into the original queue, while keeping the attributes that belong to the message.
| Header Format | AMQP Headers | Tar PAX Records |
|---|---|---|
| Format Translation | map[String] Bool, Integer, String, Float | map[String] String |
| Body Type | Bytes | Bytes |
The tar metadata can be accessed using pax:
pax -r -zf 1_message_100.tgzThis will output the messages and in addition a PaxHeaders.0 directory containing identical filenames as the messages, enabling access of the metadata. Right now there is no way to selectively publish only certain messages containing a header value, this is a planned feature.
If you plan to work on rabbitio you will need Golang. PRs are welcome as well as implementation discussions.
Clone rabbitio
mkdir -p $GOPATH/src/github.com/meltwater
cd $GOPATH/src/github.com/meltwater
git clone [email protected]:meltwater/rabbitio.gitcd rabbitio
make && make buildFor any bug reports or change requests, please create a GitHub issue or submit a PR.
Also feel free to drop a line to the maintainers:
- Joel (@vorce)
- Stian (@stiangrindvoll)