Inter node IPC with bash

Inter-node IPC with bash

Keywords

Inter-node IPC, Distributed IPC, Message passing, bash

Introduction

Gsatellite uses a bash library that provides inter-node IPC when used on a shared file system like NFS or Lustre. This library provides message boxes (similar to message queues) and send and receive functions.

Usage

Include the library

To enable bash programs to use this feature you have to source the library in the respective programs:

[...]
. $_LIB/ipc/file.bashlib
[...]

In addition you also need a directory shared between the hosts or cluster you want to operate on that will be the base directory for message boxes. This is defined by the variable _MBOXES. Hence also add a definition for this variable:

[...]
readonly _MBOXES="$HOME/mboxes"

. $_LIB/ipc/file.bashlib
[...]

NOTICE: To prevent tampering, $_MBOXES should be owner accessible only (i.e. 0700)!

Create message boxes

Each process needs its own message box, so create one per process:

[...]
_self="$$"
_inboxName="$_self.inbox"

#  create inbox
_inbox=$( ipc/file/createMsgBox "$_inboxName" )
[...]

By default this should use the own PID as first part of the name plus a '.' and an arbitrary string afterwards. The example above shows a convenient naming. ipc/file/createMsgBox() will return the full path to the message box. This includes the $_MBOXES string as prefix and the fully qualified domain name (FQDN) of the host the process is running on as super directory, e.g. /home/you/mboxes/yourhost.domain/12345.inbox. If you want a special message box name for a specific process - maybe because you don't want to remember the PIDs of your processes, you can create an alias message box in addition:

[...]
#  create alias inbox
_aliasInboxName="MyProcessA"
_aliasInbox=$( ipc/file/createAliasMsgBox "$_inbox" "$_aliasInboxName" )
[...]

A process can then also be reached via its alias message box.

NOTICE: The name for an alias message box has to be unique per host, but not per cluster!

Send messages

After creating a message box you're ready to send and receive messages. This is done with the library function ipc/file/sendMsg(). The function has two arguments:

1. _messageBox: The destination message box
2. _message: The actual message

The _messageBox has to be specified as full path to the destination's message box. You're basically free to use an arbitrary string as _message, but to allow a receiver to detect from which sender a specific message came from, the proposal is to use a string formed like the following:

"<MESSAGE_TEXT>;<ANSWER_BOX>"

If the message could be delivered the call will return a status value equal to 0. If there were problems during delivery (e.g. the destination message box does not exist) the status value will be unequal to 0.

Example:

[...]
_destMsgBox="/home/you/mboxes/hostb.domain/MyProcessB"
_message="Hello World!;$_inbox"

ipc/file/sendMsg "$_destMsgBox" "$_message"
[...]

Receive messages

To receive messages you use the library function ipc/file/receiveMsg(). The function has one single argument:

• _messageBox: The message box to receive a message from

As with ipc/file/sendMsg() the _messageBox has to be specified as full path. The function blocks until it can receive a message. Therefore the library also provides a function - ipc/file/messageAvailable() - that will only return a status value equal to 0 if at least one message is available in the message box. The function has the same interface as ipc/file/receiveMsg(), i.e. one single argument, the message box to check for messages.

Example:

[...]
while [1]; do
        if ipc/file/messageAvailable "$_inbox"; then
                _message=$( ipc/file/receiveMsg "$_inbox" )
                processMsg "$_message"
                break
        else
                sleep 1
        fi
done

exit
[...]