Re: malloc() size limit
"Dave Calkins" wrote:
In one of our apps, we're finding that we hit a wall in terms of
the size of memory block we can allocate with malloc(). It
appears to be around 1/2 Gb (specifically 503185324 bytes is the
highest we can allocate). Creating a new empty project doesn't
have this issue, presumably the limit is higher.
It seems you forgot that malloc attempts to allocate _contiguous_
chunk of memory. Of course there is a limit for contiguous free
chunk in a process' address space due to memory fragmentation.
Naturally new empty process has fewer things loaded in memory, so
it has larger contiguous free chunks.
At the time the failing malloc() is performed there is plenty of
system memory available, in fact, on the test machine we have
2GB of physical memory and just under 1GB is available.
The amount of physical memory has nothing to do with it. Windows
platform implements virtual memory mechanism
(http://en.wikipedia.org/wiki/Virtual_memory), which assigns total
4GB of address space for each process. 2GB is reserved for kernel
code of a process and 2GB is for user code.
So there don't appear to be any system limits being hit here,
but there's obviously something in our process limiting the size
of the malloc(). Any idea what controls this or how we can work
around this issue?
Memory fragmentation that's what limits you. There are other
things that loaded in user 2GB space: DLL's, stacks of threads,
your own allocations, etc.. You cannot expect that such huge
allocations as 1GB will succeed. There are two common approaches
for large memory consumption:
1. Reserve upfront large amount of virtual space, when it is still
not fragmented, then commit/decommit as neccesary. However, you'll
hit a limitation arount 1-1.5GB anyway. See `VirtualAlloc' et al
in MSDN.
2. Redesign the algorithm so it won't require such amounts of
contiguous memory. Make some kind of memory window and travel with
it over your data, which backed up by a file. Look for
`CreateFileMapping' in MSDN and memory mapped files.
Alternatively, upgrade to 64-bit Windows.
HTH
Alex