SCALE

OpenDDS Scale Issues

Overview

Scalability of OpenDDS to transport data depends on
many interdependent factors and issues.  Ultimately all
limitations are a consequence of the hardware and system
resources available as presented to the DDS service via the
actual implementation of the service.  The resources which
will result in scaling issues include the following:

  1) Network data throughput;
  2) Host CPU throughput;
  3) Host Memory capacity;
  4) Host Disk capacity;
  5) Operating system network capacity (sockets).

Each of these resources is taxed by many different individual
components and policies of the specific DDS implementation.
We discuss each briefly below and how these are reflected
as scale limits in the DDS service.

Network Resources

Network bandwidth is required for both the payload as
well as the overhead information carried in DDS messages.
Other layers, such as an ESB, may add additional overhead to
the actual application payload.  There can be a significant
overhead for small payload messages from some layers.

The OpenDDS implementation attempts to minimize the individual
message overhead by using binary encoding of data and
variable length sample headers carrying only the minimum
amount of data required for each sample.  There is a small
amount of additional network overhead incurred by the OpenDDS
implementation during setup and association establishment
and teardown.  This overhead consists of messages managing
the service metadata and associations.

The topology of the nodes and network throughput will
determine the limits of how many nodes and how much traffic
can be carried for any given system.  This is independent
of a specific implementation used to transport data within
the system.

CPU Resources

The CPU capacity available will determine the amount of
processing that can be done on any given node.  The middleware
portion of processing represented by OpenDDS is minimal
compared to application processing in typical installations.
While there are many threads in the OpenDDS middleware, most
are blocked waiting for additional work to do.  They allow
low latency responses to be generated as work arrives and are
not consuming processing cycles while no work is present.
This means that there should be no discernible limitations
to scale introduced by the use of OpenDDS.

Memory Resources

The memory capacity available to any process will limit the
amount of DDS and OpenDDS artifacts that can be created at
any given time.  In addition to the middleware portion of
application processes, the OpenDDS implementation of the
DDS service includes a separate process providing metadata
to the service participants.  This metadata process will
have limitations on scale that are independent of any
limitations encountered within the application process
middleware components.

In the metadata process, the total number of DDS Entities
as well as the total number of Associations between DDS
Publications and Subscriptions will affect the amount of
memory usage by this metadata process.  An estimate of memory
usage for Entity metadata is on the order of kilobytes,
and for associations less than a kilobyte.  This means
that thousands of Entities will likely consume tens of
megabytes of process memory.  This is well within modern
system capabilities.  The actual point at which the number of
Entities can no longer be increased can be explored if needed.

In addition to memory usage limiting the number of Entities
which can be present at one time, an additional constraint is
imposed through the use of a finite identifier for Entities.
OpenDDS identifies Entities internally using a 16 byte value,
24 bits of which are dedicated to identification of Entites
within a single DomainParticipant.  While it is likely that
memory limitations will be reached long before the possible
16 million Entity identifier values have been exhausted, it is
also possible that dynamic long lived processes might exhaust
these values.  There is currently no collision avoidance
implementation included once the values reach the maximum and
wrap back to the start of the value range.  Careful analysis
of a systems expected usage will determine if this will be an
issue during deployment and whether adding collision avoidance
or replacing aged processes is the preferred solution.

Performance with a large number of Entities as well as number
of Associations will be slowed (degraded) by the search times
through the containers holding the metadata.  The containers
holding service metadata are implemented as red-black trees,
which have an O(log n) search time.  The comparisons done for
metadata searches are made between 16 byte values, with the
most likely differing bytes compared first.  Degradation of
speed will be slow, as the search time will double only after
the number of items is squared.  Also, searches are not a
large portion of the total processing done.  Any degradation
experienced in the metadata process will occur only during
association establishment, modification, and removal.

The middleware portion of application processes will encounter
memory limits in the amount of service Entities which can be
created as well as the amount of data which can be stored.
The number of DDS Publications and Subscriptions is the most
dynamic in typical applications.  These Entities require
storage for themselves as well as for data which they write
and read.  The dominant memory usage for these Entities is
data storage.  Data is stored within Publications for up to
the HISTORY QoS policy depth, or if set to KEEP_ALL, to the
limit of the RESOURCE_LIMITS QoS policy value.  These policy
values can be tuned to limit the storage for each Entity so
that more Entities can be created and used simultaneously.

Note that when a HISTORY policy depth is set, this amount
of memory is preallocated and used as a pool for storing
the publication data.  Data is stored within Subscriptions
similarly to that of Publications.  It is possible for the
application to remove data from Subscriptions by using the
'take()' operations on the DataReader Entities.  Since,
for fixed buffer sizes, memory is preallocated there is no
performance degradation caused by data storage.  As the number
of Publications and Subscriptions grow within the application
process, the containers holding them will take longer
to search.  This means a slight performance degradation.
The amount of searching done is minimal for these Entities
during typical processing scenarios.

In addition to the containers holding data, there are
additional containers related to DDS Entities as well as
internal elements containing information required for
processing.  Information about data instances within a
Subscription is held in a container that must be searched
linearly with each 'read()' operation performed.  In cases
where the number of instances (data with unique key values)
is large, this can result in degraded performance due
to this search.  This effect has only been observed to
date in pathological applications that created single use
instances without releasing the resources when complete.
Careful construction of application usage can avoid this
performance degradation.

In addition to the DDS service Entities, OpenDDS creates
Transport objects to provide the actual mechanism to send
and receive data.  These Transport objects create several
threads to manage active queues and reactors.  Each thread
created will result in memory being allocated for use as stack
storage for that thread.  The amount of storage dedicated
to stack storage can be large and some systems will exhaust
memory quickly if too many Transport objects are created.

Disk Resources

In addition to the memory used during processing, the
DDS service allows data to be stored on disk to provide
a persistence service.  This service is enabled by the
DURABILITY QoS policy setting of PERSISTENT.  The data
stored will require as much disk storage as the memory
used to store the data while the process is executing.
Systems with limited disk storage available might encounter
scale problems if these settings are enabled.

Operating System Resources

The operating system must provide the ability to establish
communications with other nodes.  When using the OpenDDS
implementation, either TCP or UDP sockets will be used for
this communication.  The OpenDDS transport implementation
attempts to multiplex all traffic between two TCP or UDP
endpoints through a single set of ports.  This is extended
when the TRANSPORT_PRIORITY QoS policy enables multiple
priority settings between different hosts.  Each priority
setting will establish its own TCP or UDP connection.
If there are a large number of priority policy values or if
there are a large number of hosts directly connected then
the system socket resources may limit the number of nodes
which can be added.  The simplest way to extend the scaling
if this limit is reached is to aggregate some traffic and
send only to the aggregators.

ESB Scale Issues

In addition to the scalability issues described above, use
of the OpenDDS JMS Provider within the JBoss ESB also
introduces some scalability issues of its own.

The ESB is exceptionally configuration-sensitive. Adjustments
to the configuration of the ESB, the JMS Provider, and even
JBoss AS itself will be required to achieve optimal
performance.

Typical memory and garbage collection issues will affect both
the ESB and JBoss AS; there are a number of documentation
sources on how to best tune and deal with these problems (and
are therefore out of scope for this document).

The ESB configuration, particularly the services and
providers configuration can affect the number of OpenDDS
entities which are initialized and used at runtime.
Mis-configuration of these attributes can create an in-ordinate
number of resource allocations and de-allocations over the
lifespan of processing a single message. Ideally, a single set
of resources is initialized at start up, and then re-used for
the lifetime of the service.

The OpenDDS JMS provider aggressively caches lower-level
entities such as DomainParticipants and Topics, however bad
configuration can result in an application creating (and
destroying) an arbitrary number of Subscribers, Publishers,
DataReaders, DataWriters, and Transports.