Re: Heap corruption in CThreadSlotData::SetValue()?

Hello there,

I???m looking for some suggestions in regard to the issue below. If this is
not appropriate group, please redirect me.

I???m trying to track down root cause for a fairly elusive problem that
occurs
once in awhile and only when full page heap is enabled. The problem is not
reproducible on demand when we enable full page heap, but symptoms suggest
a
heap corruption. Not sure if this is some type of timing or
synchronization
issue.

Basically, we get crash dumps from our customers that show:
# 0 Id: 1f0.810 Suspend: 1 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr Args to Child
0012fa64 7c90df5a 7c91b24b 00000064 00000000 ntdll!KiFastSystemCallRet
0012fa68 7c91b24b 00000064 00000000 00000000
ntdll!ZwWaitForSingleObject+0xc
0012faf0 7c901046 0197e178 7c912cce 7c97e178
ntdll!RtlpWaitForCriticalSection+0x132
0012faf8 7c912cce 7c97e178 00000000 006ff238
ntdll!RtlEnterCriticalSection+0x46
0012fb34 7c80b4af 00000001 00000000 0012fb70 ntdll!LdrLockLoaderLock+0xea
0012fb8c 7c80b5ad 00400000 02c5cdf8 00000104
kernel32!GetModuleFileNameW+0x89
0012fbb8 73e69cc8 00400000 0012fcdc 00000104
kernel32!GetModuleFileNameA+0x4b
0012fee8 73e69c67 006ff238 73ddcf57 00400000
mfc42!CWinApp::SetCurrentHandles+0x45
0012fef0 73ddcf57 00400000 00000000 00152ffc mfc42!AfxWinInit+0x4f
0012ff10 0065f893 00400000 00000000 00152ffc mfc42!AfxWinMain+0x2c
0012ff24 0056f6f2 00400000 00000000 00152ffc MyApp!WinMain+0x15
[appmodul.cpp @ 30]
0012ffc0 7c817077 80000001 0145e0c4 7ffdd000 MyApp!WinMainCRTStartup+0x134
0012fff0 00000000 0056f5be 00000000 78746341
kernel32!BaseProcessStart+0x23

  1 Id: 1f0.8c8 Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr Args to Child
02a0f650 7c90df4a 7c8648a2 00000002 02a0f838 ntdll!KiFastSystemCallRet
02a0f654 7c8648a2 00000002 02a0f838 00000001
ntdll!ZwWaitForMultipleObjects+0xc
02a0fa08 77c32f0f 02a0fa50 00000000 00000000
kernel32!UnhandledExceptionFilter+0x8b9
02a0fa24 77c3a3c7 00000000 02a0fa50 77c35cf5 msvcrt!_XcptFilter+0x161
02a0fa30 77c35cf5 02a0fa58 00000000 02a0fa58 msvcrt!_endthreadex+0xc0
02a0fa58 7c9032a8 02a0fb44 02a0ffa4 02a0fb60 msvcrt!_except_handler3+0x61
02a0fa7c 7c90327a 02a0fb44 02a0ffa4 02a0fb60 ntdll!ExecuteHandler2+0x26
02a0fb2c 7c90e48a 00000000 02a0fb60 02a0fb44 ntdll!ExecuteHandler+0x24
02a0fb2c 77c47631 00000000 02a0fb60 02a0fb44
ntdll!KiUserExceptionDispatcher+0xe
02a0fe2c 73e68a0e 02c72fe0 00000000 00000024 msvcrt!memset+0x41
02a0fe48 73e682ca 00000001 02c6eee8 00000000
mfc42!CThreadSlotData::SetValue+0xb9
02a0fe5c 73e6842a 73e688db 00000003 02171891
mfc42!CThreadLocalObject::GetData+0x55
02a0fe68 02171891 021752e8 00000003 80284006
mfc42!AFX_MAINTAIN_STATE2::AFX_MAINTAIN_STATE2+0x14
02a0fe80 02171a7b 02170000 00000003 00000000 MyDll!DllMain+0x102
[dllmodul.cpp @ 169]
02a0fea0 7c90118a 02170000 00000003 00000000 MyDll!_DllMainCRTStartup+0x50
02a0fec0 7c913a43 02171a2b 02170000 00000003
ntdll!LdrpCallInitRoutine+0x14
02a0ff38 7c80c136 7c968f03 02786f78 026fff78 ntdll!LdrShutdownThread+0xd7
02a0ff70 77c3a33b 00000000 02786f78 02a0ffb4 kernel32!ExitThread+0x3e
02a0ff80 77c3a3b5 00000000 7c968f03 02695000 msvcrt!_endthreadex+0x34
02a0ffb4 7c80b729 026fff78 7c968f03 02695000 msvcrt!_endthreadex+0xae

  2 Id: 1f0.8e8 Suspend: 1 Teb: 7ffdc000 Unfrozen
ChildEBP RetAddr Args to Child
0300fc0c 7c90df5a 7c91b24b 00000064 00000000 ntdll!KiFastSystemCallRet
0300fc10 7c91b24b 00000064 00000000 00000000
ntdll!ZwWaitForSingleObject+0xc
0300fc98 7c901046 0197e178 7c91e3b5 7c97e178
ntdll!RtlpWaitForCriticalSection+0x132
0300fca0 7c91e3b5 7c97e178 0300fd2c 00000004
ntdll!RtlEnterCriticalSection+0x46
0300fd18 7c90e457 0300fd2c 7c900000 00000000 ntdll!_LdrpInitialize+0xf0
00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0x7

All 3 threads were created about 30 seconds ago, but consumed hardly any
CPU
time. And it seems as the application is shutting down.
0:001> !runaway 7
User Mode Time
 Thread Time
  2:8e8 0 days 0:00:00.000
  1:8c8 0 days 0:00:00.000
  0:810 0 days 0:00:00.000
Kernel Mode Time
 Thread Time
  0:810 0 days 0:00:00.140
  1:8c8 0 days 0:00:00.031
  2:8e8 0 days 0:00:00.000
Elapsed Time
 Thread Time
  0:810 0 days 0:00:29.746
  1:8c8 0 days 0:00:29.590
  2:8e8 0 days 0:00:29.511

NT loader lock is owned by thread 1. Threads 0 and 2 are blocked on the NT
loader lock???s critical section.
0:001> !cs -l
-----------------------------------------
DebugInfo = 0x7c97e1a0
Critical section = 0x7c97e178 (ntdll!LdrpLoaderLock+0x0)
LOCKED
LockCount = 0x2
OwningThread = 0x000008c8
RecursionCount = 0x1
LockSemaphore = 0x64
SpinCount = 0x00000000

So, an access violation takes place within thread 1???s context. The access
violation occurs while setting memory block 02c72fe0 to 0 with size of 36
bytes. However, the memory block in question was allocated 4 bytes less
than
what memset is using, which is why we get access violation.

0:001> !heap -p -a 02c72fe0
   address 02c72fe0 found in
   _DPH_HEAP_ROOT @ 141000
   in busy allocation ( DPH_HEAP_BLOCK: UserAddr UserSize
- VirtAddr VirtSize)
                                 142b24: 2c72fe0 20
- 2c72000 2000
   7c918f21 ntdll!RtlAllocateHeap+0x00000e64
   7c809a7f kernel32!LocalAlloc+0x00000058
   73e689d7 mfc42!CThreadSlotData::SetValue+0x00000082
   02171a7b MyDll!_DllMainCRTStartup+0x00000050
   7c90118a ntdll!LdrpCallInitRoutine+0x00000014
   7c913a43 ntdll!LdrShutdownThread+0x000000d7
   7c80c136 kernel32!ExitThread+0x0000003e
   77c3a33b msvcrt!_endthreadex+0x00000034
   77c3a3b6 msvcrt!_endthreadex+0x000000af
   7c80b729 kernel32!BaseThreadStart+0x00000037

The disassembly below matches the code snippet from AFXTLS.cpp.
73e689f0 e8c0d2fbff call mfc42!AfxThrowMemoryException (73e25cb5)
73e689f5 8b4608 mov eax,dword ptr [esi+8]
73e689f8 8b4f0c mov ecx,dword ptr [edi+0Ch]
73e689fb 2bc8 sub ecx,eax
73e689fd c1e102 shl ecx,2
73e68a00 51 push ecx
73e68a01 8b4e0c mov ecx,dword ptr [esi+0Ch]
73e68a04 8d0481 lea eax,[ecx+eax*4]
73e68a07 53 push ebx
73e68a08 50 push eax
73e68a09 e8748ff6ff call mfc42!memset (73dd1982)

void CThreadSlotData::SetValue(int nSlot, void* pValue)
<snip>
                               if (pData->pData == NULL)
                                               AfxThrowMemoryException();

                               // initialize the newly allocated part
                               memset(pData->pData + pData->nCount, 0,
                                               (m_nMax - pData->nCount) *
sizeof(LPVOID));

So, either (pData->pData + pData->nCount) points to a wrong memory block
or
(m_nMax - pData->nCount) comes up with wrong byte count.

Does anyone have any suggestions on how to track down this issue?

Thank you,
Olegas