Re: Suspected Memory Leak in Multithread queue implmenetation
tikcireviva wrote:
I've done a mulithread queue implementation on stl<queue>, my
developement environment is on VC6 as well as FC3. Let's talks about
the win32 side.
The suspected memory leak is find after I've run through my unit test
cases.
Test Case:
1. start 2 threads, thread A and thread B.
2. thread A enqueue 10000 dummy objects, meanwhile thread B dequeue
10000 dummy objects (delete object at the same time), these operations
are called a "cycle".
3. there is a while loop to loop the cycle, in between each cycle,
there is a 5 seconds sleep time.
4. run this for 10 minutes
5. stop insert any dummy objects
I've run this test case for 15 mins, and monitor its private_bytes,
working_set and virtual_bytes with the windows performance monitor.
I'm not sure that this is really significative, especially on
such a small, short running application.
My concern after running this test case is that,
1. virtual_bytes increase double its size frequently (seems to me it is
due to the queue memory allocation), and the virtual_bytes never drops
into its original position (I mean at the beginning when the program is
started)
I'm not familiar with the Windows platform, but this is typical
in a lot of environments. Once a process acquires memory, it
keeps it; delete just returns it to be recycled within the
process. (With STL components, this can actually happen at
several levels: in the allocator, or in operator new/delete.)
2. both private_bytes and working_set are increasing as well, just the
fact that, duing the 5 seconds wait time, (step 3), these values drop
to nearly zeros, but after the 5 seconds wait time, these values
increase rapidly back to a state where it was dropped. and, of course
keep increasing.
The real question is: does the total memory use increase
significantly if, say, you double the number of cycles? If not,
the behavior is probably normal. If so, you may have a problem.
3. during the (step 5), I can tell virtual_bytes is at its max, and
private_bytes and working_set are set at a values (1/3 below the max
point) and won't be able to drop any more.
My questions are:
1. The stl::queue double its memory for storage when it hits the
maximun size,
std::queue is a wrapper for some other container. std::vector
will increase its memory use in an exponential fashion (not
necessarily doubling, but multiplying by a constant factor);
this is not true for std::deque or std::list, however.
is there any method to explicitly delete the allocated
memory from stl::queue? Since it consumes the virtual memory without
releaseing it.
The "standard" way of freeing all memory used by an STL
container is:
ContainerType().swap( containerToBeEmptied ) ;
Note, however, that this only ensures that the memory has been
returned to the associated "allocator". This allocator may or
may not return it to operator delete, and operator delete may or
may not return it to the OS.
2. Would you guys mind critique if I've done anything wrong on my
stl::queue multithread implemenation?
One obvious thing: you return references to the internal data in
certain cases. This makes any internal lock totally useless,
and means that your code probably isn't thread safe.
I'm not sure what your goal is. If it is to provide a wrapper
to standard containers, so that the user can use them from
several threads without locking, forget it. The interface is
designed in such a way that this simply cannot be done. (Note
that this is NOT a defect. There are many reasons for doing it
this way, and there is almost no reason for not requiring client
level synchronization.) If your goal is to implement a message
queue for communication between processes, you need to define
the interface of this first, rather than just try to assume the
interface of a standard container. My own interface just
implements send and receive, and uses std::auto_ptr to transfer
the data. This means that messages must be dynamically
allocated, but since I generally use the visitor pattern in my
messages, and thus need dynamic polymorphism, this is not a real
problem. If you plan on using value type objects as messages,
however, you should provide an interface which does a deep copy.
--
James Kanze (GABI Software) email:james.kanze@gmail.com
Conseils en informatique orientie objet/
Beratung in objektorientierter Datenverarbeitung
9 place Simard, 78210 St.-Cyr-l'Icole, France, +33 (0)1 30 23 00 34
--
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]