The STLshm library makes it possible to create and use STL containers in a variety of shared memory implementations.  It works with standard STL implementations.  The really high level list of features includes:

 

 

The allocator template parameter of STL containers is often quoted as enabling containers in shared memory.  There are a number of implementations available, but many of them have limits I didn’t like.  This library aims to be a fairly full feature implementation of this concept.  Here’s the more detailed list of its capabilities:

 

1.    Minimizes the programming impact of using a container in shared memory.  The coding style matches what is done when a multi-threaded program is using an STL container, with the only other difference being in the way the initial pointer to the STL container is acquired.  (In a multi-threaded program, it would always be created sometime early in execution, but in a shared memory case, it could already exist before the program was started.)

 

2.    The shared memory region can grow while in use by multiple processes and/or threads.  We use the same paradigm that is typical of heap management.  I.e. A process initially creates a region of a given size, and gives that memory to the freelist manager (malloc / new).  When the freelist runs out of memory, additional shared memory is requested for the region from the operating system, and the new chunk is added to the free-list, and so on.  Any process using the region can trigger its growth.  Thus the shared region does not have to be pre-allocated to its maximum size.

 

3.    There is full built-in support for synchronizing the initial creation/attach of a shared region among multiple processes.

 

4.    Shared memory presents the interesting new problem that the data structure can already exist when the process attaches to the region. So we provide a ‘bootstrapping’ mechanism which is essentially an atomic “find() or create()” operation that processes can use when first creating/attaching to known objects in shared memory.

 

5.    A number of shared memory region implementations are provided by reusing the “Memory Pool” concept from the ACE library.  The options include memory maps, shared memory regions, Windows-specific anonymous page-file regions, and sbrk() allocated regions (useful for testing…)

 

6.    A couple of paradigms are supported by the shared memory allocator implementation, to suit different applications.

 

a.    The first approach provides the allocator with a fully functioning default constructor for the allocator that distinguishes when an allocator-aware object is being constructed in shared memory vs. heap, and automatically constructs the allocator to use the corresponding memory area.  I found this approach very useful with object like std::string, and for some constructs, like shm::map< int, shm::string>  (A container which contains some type which itself uses an allocator.)  Under this approach, an allocator member variable will automatically direct the object to allocate memory from the same region that the object itself is created in.  This means, for example, that when strings are copied to and from shared memory, the dynamic memory that they themselves allocate automatically comes from the same region that the object is constructed in.

 

b.    The second approach builds on (a) by allowing each thread/process to designate a particular shared memory region that will be used by all allocator objects constructed thereafter by that thread/process.  This allows more explicit control and is necessary with objects that make use of memory allocation optimizations like “copy on write”.  Many std::string classes include that optimization, and don’t work correctly using only the first approach.

 

c.   Finally, the allocator can be constructed with a pointer to the region that it should use, explicitly overriding (a) or (b).

 

7.    I’ve tried to provide for extensibility via the encapsulation of separate concepts, so that individual aspects of the design can be enhanced or replaced.  New free-list/allocation  algorithms, shared memory region implementations, locking mechanisms, etc. can all be added.  The library does not assume a one-size fits all, as there really doesn’t seem to be any one ‘ideal’ implementation of this capability.  Every design has its trade-offs.

 

 

If successful, this project will be absorbed into the ACE library, and I’ll also be able to use what I’ve learned from this effort to add similar capabilities to the Boost.interprocess libraries.

 

Originally, I had intended on building this project into a stand-alone library for using STL containers from shared memory.  I began this by making use of a small portion of the ACE library.  My intention was either to assist the ACE project in their modularization efforts so that the dependency would be on only a portion of that library, or to copy and refactor some elements of the ACE library into STLshm to create a stand-alone STLshm library.  I am going to offer this to ACE and see what comes of it.

 

In February 2008, I stumbled upon Boost.interprocess library, which at the time of this writing is not part of the standard Boost download, and not available in the Vault, but is available if you pull it directly out of CVS on sourceforge.   Boost.interprocess has a really nice API to it, complete with the scope-based locking approach that Boost.threads uses.  It lacks some of the capabilities that STLshm is providing:  The regions can’t grow and the allocators must be constructed with a clearly identified region (no default constructors).  But if I can lend my efforts to enabling those things in Boost.interprocess, it would be a good stand-alone library that addresses all of my needs.

 

So my intention now is to donate my work to the ACE library for use in ACE, and also contribute some work to Boost.interprocess to permit regions to grow under that library, and provide my Allocator variations to enable cases where you need the default constructor to work.