Memory claim denied errors may appear in the OpenSplice log file. This is a symptom of running out of shared memory. These are the steps to follow to troubleshoot this problem.
Memory Claim Denied Error in ospl.error log
Report : ERROR
Date : Wed Apr 05 18:17:02 UTC 2000
Description : Memory claim denied for small object of 64 bytes: request exceeds available matching resources (approximately 0 bytes). To remedy this situation try to configure a bigger value for the OpenSplice/Domain/Database/Threshold value, most likely a database size is configured that is bigger than the default without adapting the default threshold value accordingly (hint: likely the solution if memory claim is denied while deleting large amounts of entities/data or when using very big topic sizes). Alternatively the value configured for the OpenSplice/Domain/Database/Size value has to be increased (hint: likely the solution if memory claim is denied while receiving a lot of data).
Node : cpu2-2
Process : Process1 <12549>
Thread : watchdog 7f7dfcbae700
Internals : V6.5.0p9/a3b348e/64b93f0/c_mmbase/c_mmbaseAdv.c/1564/0/954958622.466426306
Troubleshooting the Memory Claim Denied Error Message
Increase the amount of shared memory available
Check that OpenSplice has enough shared memory available to deal with the data sizes you are working with.
The size of the shared memory is set in the xml configuration file. Look for the Database element under the Domain section.
You may need to configure your operating system to support the requested size. On most platforms you need ‘root’ privileges to set large sizes.
Check your application is dealing with instances correctly
When adding instances into a system you need to remember that at some point these need cleaning up. If you keep adding instances into the system and do not do anything to get rid of the irrelevant ones you will eventually run out of memory. To avoid this you need to explicitly tell the system which instances should remain available and which are no longer relevant.
To ensure that OpenSplice can reclaim memory correctly the readers and writers should do the following:
- On the writer side you should unregister all instances which you no longer intend to update.
- Writers should also dispose and unregister all transient instances that are no longer relevant to the rest of the system.
- The reader needs to take all instance data that is marked DISPOSED or NO_WRITER. Please note that DISPOSED and/or UNREGISTERED instances don’t clean up themselves from your readers automatically. This has to be done manually by taking them out. You can also set the the auto_purge timers on the ReaderDataLifecycleQosPolicy to remove them automatically after some time interval elapses.
Further Troubleshooting for the Memory Claim Denied Error
If you have done above and are still seeing problems with the memory claim denied error message then you are probably allocating resources faster than you are releasing them. This means the total number of instances/samples in the system will be continually increasing. To see if this is the case you can use mmstat. This tool can provided details about the total number of instances and the total number of samples in the system.
If you do see that the number of instances is continually increasing then your writer may be updating samples at a faster rate than the readers can consume them. If you are using a HISTORY QOS of KEEP_LAST this may not be a problem since newer instances will overwrite older instances. If you have a HISTORY QOS of KEEP_ALL then all the instances are kept and the reader will get further and further behind.
There are a couple of things you can try:
- Slowing down the writer or increasing the speed that the reader is consuming the data
- Set instance bounds on the reader and writer side. This allows you to block a writer from creating additional instances until current resources are reclaimed. On the reader side it will prevent a reader from reading any more until the existing information has been dealt with. By default the reader will buffer as much as it wants. If the writer is writing faster than the reader is reading then you will not hit the writer bounds as the writer will only start buffering after the reader reaches its limit.
Setting a Memory Threshold
In normal use you should not see a Memory Claim Denied error. When they do occur they happen during the middle of an update. It is better to make an memory estimate before an update begins and prevent it happening if there is not enough memory available.
To do this you can set the OpenSplice/Domain/Database/Threshold value in the xml configuration file. This checks the available memory before doing an update. If enough memory is available the modification will take place successfully. If the modification will take more memory than there is available it will not happen The error code RETCODE_OUT_OF_MEMORY will be thrown. This can be caught by the application and the Memory Claim Denied error will not occur.
The Threshold is set in the xml file as follows:
It is advisable to use the OpenSplice configuration tool to modify the xml file as this will check the file for errors.
By default the Threshold is set to 1MB. This is about 10% of the default Database size and is a sensible value. However, if you increase your database size, eg to 100MB, without modifying your Threshold, it only covers about 1% of the relative database size. This may be on the low side. As a general rule don’t make the Threshold lower than 1MB and leave it between 5 – 10% of your database size. When using big samples it will need to be higher, when using smaller samples it can be lower.
You may also like to read Out of Memory: Unable to create message for Topic and OpenSplice Memory User questions.