Last Updated: August 12, 2013
Applies to GemStone/S 64 Bit 2.x and 3.x.

GemStone memory use is described in the System Administration Guide, starting on p293 for v3.1 and p329 for v2.4; and in the Programming Guide, starting on p319 for 3.1 and p312 in v2.4. The following information provides more specific details and instructions on what to do if you encounter an out of memory error in your application.

Overview of temporary memory use

GemStone gem sessions require memory to perform operations; any GemStone objects that are needed from disk or that are created, are stored in temporary object memory. The size of temporary object memory is configured using GEM_TEMPOBJ_CACHE_SIZE. This space must be large enough to hold persistent objects faulted into memory from disk, and newly created temporary objects, as well as classes, methods, and other system resources.

GemStone manages memory carefully to make maximum use of the configured size of memory. Each gem performs in-memory garbage collection periodically, as there is demand for memory. The garbage collector can remove from memory unmodified persistent objects, and temporary objects that are no longer referenced. However, persistent objects that have been modified, and referenced temporary objects, cannot be removed from memory. If there are too many objects that need to be in memory for the size of temporary object memory, you will encounter an out of memory error. Such errors terminate the gem process.

Avoiding out of memory errors requires that you both:

  • configure GEM_TEMPOBJ_CACHE_SIZE at an appropriate size for your application requirements, and
  • design and manage your application code to avoid excessive memory use.

Determining the correct balance between memory size and application structure may take some analysis and experimentation. The following information is intended to help you understand the general approach to out of memory errors. For specific details, you can enable diagnostics in your application, and send the information in for analysis by GemTalk Engineering.

What to do first if you encounter Out-Of-Memory errors

  1. Check your value for GEM_TEMPOBJ_CACHE_SIZE in the gem that is getting the error. Most applications will need to increase this from the default. Gems executing even fairly small operations may require a setting of 50-100MB. The maximum value is 1GB-64GB, depending on your server version.

    Be sure to check the printouts in the gem log to ensure that the gem is picking up the intended setting for GEM_TEMPOBJ_CACHE_SIZE, particularly if you use multiple configuration files or remote gems.

  2. Determine if your gem can commit more frequently. Uncommitted changes must be kept in memory until the changes are made persistent; making intermediate commits will reduce the peak memory usage.

    One way to setup your system to make such commits only as needed is to enable handling for the signal AlmostOutOfMemory (#rtSignalAlmostOutOfMemory in 2.x). If enabled, this asynchronous signal may be received any time memory use is greater than the threshold at the end of an in-memory garbage collection, and your application can be configured to commit (or take other action) as a result. See the Programming Guide for details.

  3. Review the code your gem is executing when it gets the error, for common causes of memory use. For example, besides long transaction, operations over very large collections (e.g. copying a very large collection) and creation of very many temporary objects are common issues.

    In 3.x, if GBS gems encounter out of memory error, and statmonitor data shows the gem has a very large ExportSet, you can configure the gem to remove from memory unmodified persistent objects that are in the ExportSet. To do this, execute the following in the gem:

       System gemConfigurationAt: #GemDropCommittedExportedObjs put: true

Further diagnosis consists of a deeper analysis of your application’s use of objects, to track down what exactly is causing the memory use. There are a number of tools to help with diagnosis. As part of error handling, memory details and stack traces are printed to stdout; and there are further configuration options for more detailed output of memory use and interactive debugging. Statmonitor data can be useful, and there are Smalltalk methods that report memory use.

For assistance with this, see below for instructions on what information to collect and send to GemTalk Technical Support.

Types of Out-Of-Memory errors

Not all out of memory error have the same cause. GemStone temporary memory is divided into a number of separate spaces. The spaces are used for different purposes, and the errors will vary depending on which space has become full.

The temporary objects spaces are

  • new and old — temporary objects
  • pom — Unmodified persistent objects that have been faulted in
  • perm — persistent (faulted-in) or newly created Classes and Metaclasses
  • code — recently executed instances of GsMethod/GsNMethod
  • mE — oopMap entries, mapping OOPs to objects in memory

Statmonitor statistics allow you to track the size of these spaces: OldGenSizeBytes, NewGenSizeBytes, PomGenSizeBytes, PermGenSizeBytes, CodeCacheSizeBytes, MeSpaceUsedBytes; as well as TempObjSpacePercentUsed and other related statistics.

The sizes of these spaces are calculated based on the gem’s configuraiton setting for GEM_TEMPOBJ_CACHE_SIZE. In some cases, for particularly large settings or for unusual usage patterns, you may need to adjust the relative sizes of the subspaces within temporary object memory. Contact GemStone Technical Support for assistance with this task.

While out of memory errors cause the gem to terminate, the actual cause will vary, and the error messages will be different.

Too many temporary objects

  • When you create too many temporary objects, new and old spaces will become full, and you will see an error such as “VM temporary object memory is full, old space overflow”.

Too many mark-sweeps

  • “VM temporary object memory is full, almost out of memory, too many markSweeps since last successful scavenge” occurs under various circumstances, when multiple in-memory garbage collection operations have failed to free up enough memory. This can occur when too many persistent objects have been modified, or when the ExportSet is large due to many persistent objects faulted to GBS. To avoid continuing to run with degraded performance, if in-memory GC operations are ineffective at freeing up space and would have to run continuously, this condition triggers an out of memory error.

Very many small temporary objects

  • If you are faulting in a very large number of very small committed objects, it is possible to run out of space in MESPACE before running out of space in other regions of temporary object memory. This results in the error “VM temporary object memory is full, ME-space overflow”

Too much space required for methods

  • If you have many large methods, you could encounter an error such as “VM temporary object memory is full, code space overflow”

Diagnosis of Out-Of-Memory errors

Provided your configuration is reasonable for your application’s requirements, diagnosis of out of memory errors requires that you identify what objects are unexpectedly filling up memory, and where in your applications these objects are being created or modified.

As the gem becomes close to running out of memory, lines similar to this are printed to stdout (the gem log or topaz -l output):

 vmGc MKSW softRefs: clear100% keepGoal 0, cleared 0 , live 0 nonNil 0 srThresh 0 
srUseCnt 1 
 vmGc   pom2:839K pom3:839K pom4:839K pom5:839K pom6:839K pom7:839K pom8:839K 
pom9:840K pom0:839K pom1:840K
 vmGc MKSW          66/ 304Knew  7495/ 7496Kold  8399Kpom   25Kperm   32Kcode 3440Kme 
agrStubbed 0 7594us  dirtyBytes 0 pomThresh 0 almOOM 0 DalmOOM 0 ---) 
 vmGc approaching OutOfMemory, requesting interpreter stack printout

These lines provide information about the size of the various memory sub-spaces. The gem also prints Smalltalk stacks, which may provide clues as to where object creation is occurring. Finally, it prints a list of the classes that have instances in memory, with instance counts and space required (divided by subspaces). This will identify what classes of objects are responsible forthe majority of memory usage.

For further assistance with diagnosis of out of memory issues, you can collected more detailed diagnostics and sent them to GemTalk Technical Support. GemTalk Engineering can provide more detailed assistance.

  1. Define the following environment variables in the environment of the gem process: 
        GS_DEBUG_VMGC_PRINT_MKSW=1
    export GS_DEBUG_VMGC_PRINT_MKSW
    GS_DEBUG_VMGC_PRINT_MKSW_MEMORY=1
    export GS_DEBUG_VMGC_PRINT_MKSW_MEMORY

    One way to do this is to use gemnetdebug, which is provided as an alternative to gemnetobject in the $GEMSTONE/sys directory. gemnetdebug includes a number of environment variables that can be uncommented to cause debug output to be written.

    Remove the comments on the above statements in gemnetobject, in the environment used by the gem that encounters the out of memory error. Then, specify gemnetdebug instead of gemnetobject in the login parameters of the gem.

    You can also set these variables in the environment where you start up a linked session; e.g. before executing topaz -1.

  2. Start statmonitor. The sample interval will depend on how long it takes to encounter your out of memory error.
  3. Run the code that reproduces the out-of-memory problem.

Send the gem log (or linked topaz output) and statmonitor output to GemTalk Technical Support. GemTalk Engineers will analyze the gem in-memory garbage collection diagnostic information, along with statmonitor data, and provide more specific advice for further investigation of your out of memory error.