This article explains how you can configure shared memory in OpenSplice DDS. There may be occasions when you need to alter the default settings.
OpenSplice DDS can run in two different architectural modes. These are:
- Single process or standalone deployment
- Federated or shared memory mode
Shared Memory Mode
Each DDS application will then interface with the shared memory database. The data is only present once on any machine. This makes the reading and writing of data more efficient. The OpenSplice services are able to see all the data on the node. This allows them to make decisions about data delivery.
Starting OpenSplice in Shared Memory Mode
ospl [-e] [-f] start [URI]
ospl [-e] [[-d <domain> | -a] stop [URI]]
ospl [-e] list
ospl [-e] status [URI]
To start OpenSplice use the command:
What the different command line options do
The meaning of these commands is as follows:
-h – lists out the meaning of the different command line options
-v – shows the ospl version information
start – Start the identified system
The system is identified and configured by the URI which is defined by the environment variable OSPL_URI. This setting can be overruled with the command line URI definition. When none of the URI definitions is specified, a default system will be started. Upon exit an exit code will be provided to indicated the cause of termination. The following exit codes are supported:
- 0 : normal termination as result of OSPL stop.
- 1 : a recoverable error occurred. – The system has encountered a runtime error and has terminated. A restart of the system is possible. E.g., the system ran out of resources.
- 2 : an unrecoverable error occurred. – The system has encountered an error that cannot be resolved by a restart of the system. E.g., the configuration file contains errors or does not exist.
When the -f option is specified the ospl command will not return directly, but it will instead block until termination of the OpenSpliceDDS deamon. If the -f option is not specified the ospl command will return immediately after start up.
stop – Stop the identified system
The system to stop is identified by the URI which is defined by the environment variable OSPL_URI. This setting can be overruled by the command line URI definition or the domain name which is associated with the URI and specified via the -d option. When no domain is specified by the URI or by it’s name a default system is assumed. The -a options specifies to stop all running splice systems started by the current user.
Upon exit an exit code will be provided to indicated the cause of termination. The following exit codes are supported:
- 0 : normal termination as result of OSPL stop.
- 1 : Not Applicable
- 2 : an unrecoverable error occurred. – The system has encountered an error that cannot be resolved by a retry of the same command. E.g., A faulty URI was provided.
list – Show all systems started by the current user
status – Check whether the identified system is currently running The system is identified and configured by the URI which is defined by the environment variable OSPL_URI. This setting can be overruled with the command line URI definition. When none of the URI definitions is specified, a default system will be checked.
-e When the -e option is specified, then the exit codes of ospl will have extended values compared to 0/1/2:
- 0 : The domain is operational or normal/successful exit.
- 1 : The domain is already operational.
- 2 : The domain is present but is initializing.
- 3 : The domain is present but is terminating.
- 16 : The domain is non existent.
- 31 : Recoverable error, which a re-invoke/restart might fix.
- 32 : General tool failure, like invalid command line options or unexpected internal error.
- 33 : Unrecoverable error, which a re-invoke/restart will not fix.
- 34 : A configuration has been specified that is inappropriate for the command issued.
The domain as specified in the configuration.
Configuring Shared Memory
Configuration of OpenSplice is done with an xml file. Any changes to the configuration need to be made in the opsl.xml file that is in use in the system. Changes should be made with the OpenSplice Configuration Tool as this will check the xml file to ensure it is valid. Configuration of shared memory is done in the Database element in the Domain section of the xml file.
For shared memory configuration to work this section must exist in the xml file.
This section of the xml file specifies the size of the shared memory segment holding the database.
The default value is 10485760 bytes or 10 MB. You can increase this value by changing this setting. The value can be set in bytes. You can use values including K for Kilobytes, M for Megabytes or G for Gigabytes. For example 10M instead of 10485760.
If your system requires more memory than the default you can increase this value. You may need to configure your operating system to support the requested size. On most platforms you need ‘root’ privileges to set large sizes.
It is useful to use a Memory Mapper to help select the value for the memory address.
- Windows : VMmap (Sysinternals)
- Linux : pmap –x <pid>
OpenSplice stores absolute pointers in shared memory and not relative pointers, this being better for performance reasons. As a result the shared memory segment needs to be mapped into the same virtual memory address of all processes on that machine that use OpenSplice (otherwise pointers are corrupted). This means the virtual memory address range between the configured address and the configured address + configured shared memory size needs to be available in all processes that use OpenSplice. When (part of) the virtual memory range is already used for different purposes (for instance to map shared libraries) OpenSplice and/or any application(s) will not start as mapping in the shared memory will fail. One other well-known application of shared memory that uses the exact same way as OpenSplice is the Java Virtual Machine which also maps in a shared memory segment at start-up at a fixed virtual memory address.
When the OpenSplice service manager tool starts it obtains an address from the configuration file. This determines the start address in the process’ virtual address space where the shared memory segment is mapped. Each process that attaches to the current Domain gets mapped into this memory space.
There may be times when you need to change this memory address. For example you may see OpenSplice not being able to start and errors in the log file similar to this:
ERROR [0x000006e4] opensplice – Failure to open the domain, domain id = 0 and URI=”<NULL>” The most common cause of this error is that OpenSpliceDDS is not running (when using shared memory mode). Please make sure to start OpenSplice before creating a DomainParticipant.
ERROR [0x000006e4] opensplice – os_sharedMemoryAttach failed for domainId (0) and domainName (ospl_shmem_t3k)
ERROR [0x000006e4] opensplice – Can not Map View Of file: Attempt to access invalid address.
These errors are seen because the default address space is already in use on that node. The OpenSplice deamon is not able to use the address to create the shared administration and the database. The error can also occur after OpenSplice has been started. When a new application is started OpenSplice will map this into the database. If there is not enough free memory after the default address then this can fail. The problem can also be seen if a large database size has been specified causing the memory segment boundary to be crossed.
To change the start address of the virtual memory space you need to add the option
to the configuration xml file into the section
This address must be the same for each process communicating within a domain. The possible default values are platform dependent.
- 00×20000000 (Linux)
- 00×140000000 (Linux 64-bit)
- 00×40000000 (Windows)
- 00×140000000 (Windows 64-bit)
- 00xA0000000 (Solaris)
- 00xA0000000 (AIX5.3)
- 00×0 (VxWorks 5.5.1)
- 00×60000000 (VxWorks 6.x)
- 00×20000000 (Integrity)
- 00×20000000 (LynxOS)
- 00×140000000 (LynxOS 64-bit)
- 00×0 (PikeOS)
- 00×60000000 (QXN)
- 00×0 (RTEMS)
- 00×00220000 (Windows CE)
Unfortunately we cannot control what ranges of virtual memory are selected by any (instance of an) operating system (Linux/Windows/Solaris), so you might indeed need to configure the virtual memory address range differently on different nodes even when using the same operating system. This behavior is the same for Linux/Windows/Solaris but there are differences as noted below.
- Solaris Heap and Shared Memory (SHM). The heap pool requires contiguous addresses on Solaris. So the key is to allocate the shared memory at the highest address possible across all processes which will join the DDS domain. As well as the stack sizes of all running threads (and alignment/safe zones), you also need to consider the shared libraries which are mapped into each process, as they are also allocated to the top of memory.
- Linux/Windows Heap and SHM. On Linux and Windows you can use the space after the SHM address range for the heap, it does not need to be contiguous, so the shared memory address does not have to be set so precisely as the operating system will find heap space above the SHM range.
The Threshold value is set to indicate a value of shared memory that must be kept available. If the Threshold value is set the amount of available memory is checked before an update is done. If there is less free shared memory than the value indicated by the threshold then no new allocation will be allowed. The update will not occur and the error code RETCODE_OUT_OF_MEMORY will be thrown.
The Threshold is set in the xml file as follows:
It is advisable to use the OpenSplice configuration tool to modify the xml file as this will check the file for errors.
By default the Threshold is set to 1MB. This is about 10% of the default Database size. If you increase your database size, eg to 100MB, without modifying your Threshold, it only covers about 1% of the relative database size. This may be on the low side. As a general rule don’t make the Threshold lower than 1MB and leave it between 5 – 10% of your database size. When using big samples it will need to be higher, when using smaller samples it can be lower.
This element specifies the locking policy of the Database, indicating whether to lock pages in physical memory or not.
With the virtual memory architecture, the operating system decides when to swap memory pages from internal memory to disc. This results in execution delays for the corresponding code because it has to be paged back into main memory. The element Locking can be used to avoid such swapping for the shared memory where the database resides. The user needs the appropriate privileges from the underlying operating system to be able to use this option.
The possible values are:
- True: lock the pages in memory.
- False: don’t lock the pages in memory.
- Default: use the platform-dependent default value.
The default value is Default. This value can be set in the xml file using the following configuration: