Reconstructing RtlCreateHeap: Assembly Optimization Patterns in Windows NTDLL
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
When you reconstruct C code from a Release build, the compiler has already gone through and “improved” your assembly. Understanding how it optimizes is what separates a clean decompilation from a confusing one. This post collects the common Release-build optimization patterns I ran into while reconstructing the Windows heap APIs RtlCreateHeap, RtlAllocateHeap, and RtlAllocateHeapInternal, and ends with the reconstructed RtlCreateHeap source.
Environment
- Analysis host: Windows 11
- Tools: WinDbg, IDA 7.5
- Target platform: Windows 10
- Target file:
C:\Windows\SysWow64\ntdll.dll(32-bit) - Version: 10.0.19041.5007 (WinBuild.160101.0800)
- SHA-1: 9C3A55D17C022D7B32EE558E8941C4C9938696CA
Outline
- 0x0 Common Release-build optimizations
- 0x1 Temporary-variable elimination
- 0x2 Zero-value register passing
- 0x3 Calling-convention optimization
- 0x4
ifpatterns and flattening (inverted branches) - 0x5
push/popregister assignment
0x0 Common Release-build optimizations
The Release-build optimizations relevant to this article are:
CPU pipeline optimization. The full theory is large and complicated, but for reading code you only care about the parts that affect analysis — chiefly out-of-order execution. Consider the classic three-stage pipeline (fetch → decode → execute), where three operations run concurrently on three different CPU components:
Pipeline optimization only reorders instructions that have no dependencies (mov eax, imm32 / mov ebx, imm32 and so on). A sequence of dependent instructions breaks the optimization at that point. For a three-stage pipeline the best case is three mutually independent instructions.
Once a group of code (call it group A) can be pipelined, its internal execution order is irrelevant, and group A may also be independent of another group (group B). The compiler can then hoist (or sink) group A past group B without changing semantics. The simplest case is __cdecl stack cleanup: in ntdll, when a called function is __cdecl, the caller’s add esp, imm can be interleaved with later code. When reconstructing, you can just skip over it.
Addition optimization (scale-factor optimization). Additions are often folded into lea; e.g. eax + eax + 1 becomes lea eax, [eax+eax+1]. This appeared rarely in my reconstruction.
0x1 Temporary-variable elimination
You’ll frequently see fragments like this:
If esi originally held some variable value but gets overwritten, then goes through a series of operations before being passed into a function or stored into a variable, the compiler likely eliminated a temporary. For example:
DWORD add(DWORD a, DWORD b)
{
DWORD dwRet = a + b;
return dwRet;
}Here dwRet may be optimized away into:
mov eax, [ebp + a]
add eax, [ebp + b]When you suspect this, check whether the register serves another purpose: if it’s an argument, reconstruct it as the expression passed in; if it’s a variable assignment, reconstruct it as the expression assigned to the variable.
0x2 Zero-value register passing
I found this while reconstructing RtlCreateHeap:
Here EBX is used as a constant zero throughout the whole function — for example, zeroing the return value esi on a failure path.
0x3 Calling-convention optimization
This is common enough to call out separately. If F5 in IDA shows a function with the wrong parameter count, it’s usually because the compiler optimized a __stdcall into a __fastcall:
When you see EDX and ECX being assigned before a call, step into the callee:
If the callee directly uses EDX or ECX, translate the function as __stdcall (or keep it as __fastcall), choosing whatever reads most clearly and is easiest to maintain.
0x4 if patterns and flattening (inverted branches)
Look at how VS2019 optimizes a simple if:
; if (a >= 0)
; {
; printf("hello world");
; }
mov edx, [a]
test edx, edx
jns xxxx ; the condition is inverted here
; this is the body
push str_xxx ; HelloWorld
call subxxxxx ; printf
add esp, 8 ; stack cleanup
xxxx:
; code that follows
A normal VS2019 Release build optimizes if into an inverted jump. But in ntdll, when a branch sits inside a flattened block, it should be translated without inverting the condition. If the flattened block after the jump is immediately followed by another condition, you can also merge it into a conditional-expression + if form.
There’s another case: the condition is folded into the flattened flow. Translate the condition without inverting, then move the flattened code into the conditional body. This usually involves two jumps — one in, one out of the flattened block; if there’s an else, it may jump out before the exit.
Here’s the outer layer first:
Then the inner:
ntdll is full of this. Only in short conditionals do you see the normal inverted-jump optimization. Look ahead and compare the jump targets against the flattened block’s internal jumps, and use context and readability to decide.
0x5 push/pop register assignment
This usually means the compiler ran out of registers, and is typically followed by a call or a more complex comparison:
Summary
Reconstructing Windows internals requires familiarity with the compiler’s assembly optimization patterns before you can read the code back cleanly. Below is the RtlCreateHeap reconstruction. I’m not publishing the other heap APIs here.
Note: this code is for learning purposes only, and is a rough version — some structures are omitted (the offsets are kept). Feel free to reconstruct the rest yourself.
typedef struct _SYSTEM_BASIC_INFORMATION {
ULONG Reserved; // reserved, usually 0
ULONG TimerResolution; // system timer resolution in 100-ns units
ULONG PageSize; // page size in bytes
ULONG NumberOfPhysicalPages; // number of physical pages
ULONG LowestPhysicalPageNumber; // lowest physical page number
ULONG HighestPhysicalPageNumber; // highest physical page number
ULONG AllocationGranularity; // allocation granularity
ULONG MinimumUserModeAddress; // minimum user-mode address
ULONG MaximumUserModeAddress; // maximum user-mode address
ULONG ActiveProcessorsAffinityMask; // active processor affinity mask
UCHAR NumberOfProcessors; // number of processors
} SYSTEM_BASIC_INFORMATION;
/*
* Flags : flags specifying optional heap attributes.
These options affect subsequent access to the new heap through the heap functions
(RtlAllocateHeap and RtlFreeHeap).
There are 3 values:
1. HEAP_GENERATE_EXCEPTIONS — the system raises an exception instead of returning NULL
(an exception heap)
2. HEAP_GROWABLE — a growable heap; must be specified if HeapBase is NULL
3. HEAP_NO_SERIALIZE — no mutex is used when allocating/freeing from this heap.
When not specified, heap access is serialized by default.
Serialized access lets two or more threads allocate and free
from the same heap concurrently.
* HeapBase: when non-NULL, the requested allocation address; when NULL, the system
allocates at a random address in the process address space.
* ReserveSize:
*
*
*/
ULONG g_initVar1ByInitalizeProccess = 0;
ULONG g_initVar2ByAvrfLoadDll = 0;
typedef int(__thiscall* PFN_4B3A32F4)(DWORD, DWORD, DWORD, DWORD, DWORD, PVOID);
PFN_4B3A32F4 g_4B3A32F4;
PVOID
RtlCreateHeap(
[in] ULONG Flags,
[in, optional] PVOID HeapBase,
[in, optional] SIZE_T ReserveSize,
[in, optional] SIZE_T CommitSize,
[in, optional] PVOID Lock,
[in, optional] PRTL_HEAP_PARAMETERS Parameters
)
{
DWORD var_12C;
DWORD var_120;
DWORD var_110[10];
PVOID pLock = Lock;
ULONG ulFlags = Flags;
PVOID pBase = HeapBase;
PVOID pBase2 = HeapBase;
SIZE_T stReserver = ReserveSize;
DWORD var_BC;
SIZE_T stCommit = CommitSize;
PVOID pLock2 = Lock;
DWORD var_CC;
DWORD CriticalSectionFlag;
DWORD var_D8;
DWORD BaseAddress;
BOOL ntGlobalFlag = ((PPEB)__readfsdword(0x30))->NtGlobalFlag;
ULONG ulUnknow1 = 0;
DWORD dwRegionSize = 0;
ULONG ulUnknow2 = 0;
ULONG ulUnknow3 = 0;
DWORD pHeapHandle2;
BYTE var_A8[0x30];
DWORD dwCommiteSize;
SYSTEM_BASIC_INFORMATION SystemInformation;
DWORD var_58;
CPPEH_RECORD ms_exc;
// register-simulation group
DWORD eax = 0;
DWORD ecx = 0;
DWORD edx = (DWORD)Flags;
DWORD ebx = 0;
DWORD esi = 0;
DWORD edi = (DWORD)Parameters;
// process default heap
if (g_initVar1ByInitalizeProccess != NULL
&& pBase == NULL
&& pLock == NULL)
{
// disallow the application from changing the policy
RtlpHpAppCompatDontChangePolicy();
// if the caller supplied a Commit pointer, get the heap result through it
esi = g_4B3A32F4(ulFlags, 0, stReserver, stCommit, 0, Parameters);
if (esi != 0)
{
goto RELEASE_SRC;
}
if (edi != 0xFFFFFFFF)
{
edi = (DWORD)pBase;
esi = 0;
goto RELEASE_SRC;
}
}
else
{
if (g_initVar2ByAvrfLoadDll != 0
&& (DWORD)Parameters == 1)
{
/*
mov eax, edx
and eax, 100h ; if ((eax & 100h) != 0)
; eax = 100h
; if ((eax & 100h) == 0)
; eax = 0
neg eax ; if ((eax & 100h) != 0)
; eax = 100h -> FFFF FEFF
; if ((eax & 100h) == 0)
; eax = 0 -> 0
sbb eax, eax ; if ((eax & 100h) != 0)
; eax = 100h -> FFFF FEFF -> -1
; if ((eax & 100h) == 0)
; eax = 0 -> 0 -> 0
not eax ; if ((eax & 100h) != 0)
; eax = 100h -> FFFF FEFF -> -1 -> 0
; if ((eax & 100h) == 0)
; eax = 0 -> 0 -> 0 -> -1
and edi, eax ; if ((eax & 100h) != 0)
; eax = 100h -> FFFF FEFF -> -1 -> 0 -> 0
; if ((eax & 100h) == 0)
; eax = 0 -> 0 -> 0 -> -1 -> edi
*/
edi = (edx & 100) == 0 ? 0 : edi;
}
}
//edx = edx & 0xF1FFFFFF;
ulFlags = ulFlags & 0xF1FFFFFF;
if ((ulFlags & 0x100) != 0)
{
if (!(LOWORD(ulFlags) & 2)
|| (DWORD)pBase != ebx
|| ecx
|| stCommit != ebx
|| (DWORD)pLock != ebx)
{
goto RELEASE_SRC;
}
if (edi == 0xFFFFFFFF)
{
edi = !g_initVar2ByAvrfLoadDll ? 0 : edi;
}
if (edi != 0)
{
esi = edi;
// parameter validation
if (!RtlpHpParametersVerify(edi))
{
// goto
}
edx = ulFlags;
}
else
{
esi = pHeapHandle2;
}
}
else if (_RtlpHpHeapFeatures != 1)
{
if (((((LOWORD(edx) & 2)
&& (DWORD)pBase == ebx))
// check whether the heap parameters are supported
&& RtlpHpParametersSupported(edi))
|| edi == 0
)
{
eax = 2;
if ((DWORD)pLock2 == ebx)
{
esi = (DWORD)&pHeapHandle2;
}
}
}
if (esi != 0)
{
eax = (DWORD)&pHeapHandle2;
if (esi == eax)
{
memset((PBYTE)esi, 0, 0x30u);
*(WORD*)esi = 2;
*(WORD*)((BYTE*)esi + 2) = 0x30;
*(DWORD*)((BYTE*)esi + 0xC) = 1;
*(DWORD*)((BYTE*)esi + 0x10) |= 0xFFFFFFFF;
}
if ((*((BYTE*)(esi + 0x4)) & 1))
{
if (g_initVar2ByAvrfLoadDll == 0)
{
esi = 0;
goto RELEASE_SRC;
}
RtlpHpAppCompatDontChangePolicy();
esi = g_4B3A32F4(ulFlags, 0, stReserver, stCommit, 0, Parameters);
goto RELEASE_SRC;
}
else
{
eax = (DWORD)RtlpHpEnvGetEnvHandleFromParams(esi);
edi = (DWORD)stReserver;
if (edi == 0) // if Reserve is 0, use the Commit size
{
edi = stCommit;
}
if (eax > edi)
{
eax = edi;
}
// create the heap via RtlpHpHeapCreate, then move it into the heap list
if (!RtlpHpHeapCreate(
RtlpHpConvertCreationFlags(ulFlags,
ntGlobalFlag),
edi,
eax,
ecx,
edx))
{
goto RELEASE_SRC;
}
// move between lists
RtlpMoveHeapBetweenLists(esi, edx, 1, ebx);
if (*(WORD*)(esi + 0x14) == LOWORD(ebx))
{
goto RELEASE_SRC;
}
RtlpHpHeapDestroy(esi);
esi = ebx;
goto RELEASE_SRC;
}
}
if (!(ulFlags & 0x10000000))
{
if (_RtlpHeapErrorHandlerThreshold >= 2
&& (ulFlags & 0xFFF80C00))
{
if (((PPEB)__readfsdword(0x30))->Ldr)
{
//_DbgPrint();
// normally this should be a structure rather than a cast + offset,
// but for schedule reasons it's translated as an offset for now
printf("HEAP[%wZ]: ", *(DWORD*)(*(DWORD*)(__readfsdword(0x30) + 0xc) + 0xc) + 0x2c);
}
else
{
// dbg p
printf("HEAP: ");
}
// dbg p
printf("!(CheckedFlags & ~HEAP_CREATE_VALID_MASK)");
//
if (byte_4B3A5DA8 == 0)
{
RtlpReportHeapFailure(2);
}
//edx = ulFlags;
}
/*if (edx & 0xFFF80C00)
{
edx &= 0x7F3FF;
}*/
if (ulFlags & 0xFFF80C00)
{
ulFlags &= 0x7F3FF;
}
}
// ebx is the eternal 0; infer that var_A8 is a structure.
// maybe initialized as STRUCT var_A8 { 0 };
// but given that Windows is written in C and it's unclear which language
// standard the Microsoft internal compiler supports,
// translate it as memset for now.
memset(var_A8, 0, 0x30);
if (edi != 0)
{
// set up SEH here
ms_exc.registration.TryLevel = 0;
if (edi == esi)
{
memcpy((PBYTE)edi, var_A8, 0xc);
}
ms_exc.registration.TryLevel = 0xFFFFFFFE;
}
// the assembly here is `test cl, 20h`, but ntGlobalFlag is 4 bytes.
// I'd rather not translate it as a cast; I'm hoping the compiler optimizes the low byte.
if (ntGlobalFlag & 0xff & 0x10)
{
ulFlags |= 0x20;
}
if (ntGlobalFlag & 0xff & 0x20)
{
ulFlags |= 0x40;
}
if (ntGlobalFlag & 0x200000)
{
ulFlags |= 0x80;
}
if (ntGlobalFlag & 0xff & 0x40)
{
ulFlags |= 0x40000000;
}
if (ntGlobalFlag & 0xff & 0x80)
{
ulFlags |= 0x20000000;
}
if (ntGlobalFlag & 0x1000)
{
ulFlags |= 0x8000000;
}
ecx = __readfsdword(0x30);
// var_a8 here should be a structure — later we can infer var_a8's members
// from the PEB members.
if (!var_A8[1])
{
var_A8[1] = *(DWORD*)(ecx + 0x78);
}
if (!var_A8[2])
{
var_A8[2] = *(DWORD*)(ecx + 0x7c);
}
if (!var_A8[3])
{
var_A8[3] = *(DWORD*)(ecx + 0x84);
}
if (!var_A8[4])
{
var_A8[4] = *(DWORD*)(ecx + 0x80);
}
if (!dword_4B3A6940)
{
dword_4B3A6944 = 0x10000;
// first query system information
if (NtQuerySystemInformation(SYSTEM_INFORMATION_CLASS::SystemBasicInformation,
&SystemInformation, 0x2c, 0) < 0)
{
goto RELEASE_SRC;
}
dword_4B3A6940 = var_58;
edx = 0x1000;
}
// next, page-segment handling.
// the -0x1000 appears to align the previous value to a page start.
if (!var_A8[5])
{
var_A8[5] = dword_4B3A6940 - dword_4B3A6944 - 0x1000;
}
// maximum is 0x7F000
if (!var_A8[6] || var_A8[6] > 0x7F000)
{
var_A8[6] = 0x7F000;
}
// some added value
if (stCommit)
{
stCommit = (stCommit + 0xFFF) & 0xFFFFF000;
}
/*edi = stReserver;
if (edi != 0)
{
ecx = edi + 0XFFF;
ecx &= 0xFFFFF000;
var_BC = ecx;
}*/
if (stReserver)
{
var_BC = (stReserver + 0xFFF) & 0xFFFFF000;
}
else
{
var_BC = (stCommit + 0xFFFF) & 0xFFFF0000;
}
/*
mov [ebp+var_BC], ecx ; ecx = var_BC
mov esi, edx ; edx = dwCommiteSize
; esi = dwCommiteSize
cmp edx, ecx
ja markif19
*/
// compute the maximum heap block size: it's actually 0x7f000, i.e. 0x80000 minus one page.
// the maximum block size is 0xfe00, with a granularity offset of 3.
if (stCommit > var_BC)
{
stCommit = var_BC;
esi = var_BC;
}
// get the supplied heap-parameters block, set it from the PEB,
// and set the heap block flags from PEB->NtGlobalFlag.
// ebx = 0
if (!(ulFlags & 0xff & 2) && pBase != NULL)
{
ntGlobalFlag = 0;
}
else
{
ntGlobalFlag = 0x1000;
ulUnknow1 = 2;
//eax = var_BC - 0x1000;
if (stCommit < var_BC - 0x1000)
{
var_BC = (var_BC + 0x10fff) & 0xffff0000;
}
}
if (!stCommit || !var_BC)
{
esi = 0;
goto RELEASE_SRC;
}
// return the debug heap
if (ulFlags & 0x61000000)
{
if (!(ulFlags & 0x10000000))
{
return (PVOID)RtlDebugCreateHeap(ulFlags, pBase, var_BC, stCommit, pLock2, &var_A8);
}
}
stReserver = 0x258;
if (!(ulFlags & 0xff & 1))
{
if (pLock2)
{
ulFlags |= 0x80000000;
}
CriticalSectionFlag = (DWORD)(pLock2 ? pLock2 : 0);
stReserver = !pLock2 ? 0x270 : 0x252;
}
else
{
if (pLock2 != 0)
{
esi = 0;
goto RELEASE_SRC;
}
}
// if the caller supplied a heap base address of 0, allocate memory from the
// memory manager via ZwAllocateVirtualMemory.
if (pBase == 0)
{
BaseAddress = 0;
if (var_A8[9] != (DWORD)pBase)
{
esi = 0;
goto RELEASE_SRC;
}
pLock = (PVOID)RtlpHeapGenerateRandomValue32();
var_BC = ebx;
// eax = RtlpSecMemFreeVirtualMemory(ecx, edx, &var_BC, 0x8000)
// | BaseAddress & 0x1f << 0x10;
var_D8 = RtlpHeapGenerateRandomValue32()
| BaseAddress & 0x1f << 0x10;
dwRegionSize = var_D8 + var_BC;
if (dwRegionSize < var_BC)
{
dwRegionSize = var_BC;
var_D8 = BaseAddress;
}
if (NtAllocateVirtualMemory(-1,
&BaseAddress,
0,
&dwRegionSize,
0x2000,
(ulFlags & 0x40000) == 0 ? 4 : 64) < 0)
{
esi = 0;
goto RELEASE_SRC;
}
pHeapHandle2 = BaseAddress;
var_BC = dwRegionSize;
if (var_D8 != 0)
{
RtlpSecMemFreeVirtualMemory(var_BC, &BaseAddress, &var_D8, 0x8000);
pHeapHandle2 = BaseAddress + var_D8;
var_BC = dwRegionSize - var_D8;
}
var_CC = BaseAddress + var_D8;
stCommit = BaseAddress + var_D8;
}
else
{
if (var_A8[0] != 0)
{
if (var_A8[6] == 0 || var_A8[7] == 0 || var_A8[6] > var_A8[7] || ulFlags & 0xff != 2)
{
esi = 0;
goto RELEASE_SRC;
}
var_CC = (DWORD)pBase;
stCommit = (DWORD)pBase + (DWORD)pLock2;
var_BC = var_A8[7];
memset(pBase, 0, 0x1000);
esi = ulFlags;
}
else
{
// pBase2 needs to be switched to a structure
if (NtQueryVirtualMemory(-1, pBase, 0, &var_110, 0x1c, 0) < 0
|| var_110[0] != stCommit
|| var_110[3] == 0x10000)
{
esi = 0;
goto RELEASE_SRC;
}
// set the reserved/committed pages of the heap block from the
// supplied ReserveSize and CommitSize.
var_CC = (DWORD)var_110;
if (var_110[3] == 0x1000)
{
var_BC = var_110[2];
if (stCommit > var_BC)
{
stCommit = var_BC;
}
if (stCommit < 0x1000)
{
esi = 0;
goto RELEASE_SRC;
}
esi = ulFlags;
}
else
{
if ((ulFlags & 0x40000))
{
if (var_110[4] & 0x40)
{
goto RELEASE_SRC;
}
}
// at this point we have a heap pointer, the committed base, the
// uncommitted base, the segment flags, the commit size and the
// reserve size.
// if the committed and uncommitted base addresses are equal, we need
// to call ZwAllocateVirtualMemory to commit the amount specified by CommitSize.
memset(var_110, 0, 0x1000);
if (NtQueryVirtualMemory(-1, pBase, 3, &var_12C, 0x1c, 0) < 0)
{
esi = 0;
goto RELEASE_SRC;
}
var_BC = var_120;
stCommit = var_110[2];
dwCommiteSize = stCommit + var_CC;
}
}
var_110[10] |= 1;
pHeapHandle2 = (DWORD)pBase;
ulFlags &= 0x40000;
ecx = stCommit;
eax = var_CC;
}
// edi = pBase
if (var_CC == stCommit)
{
if (NtAllocateVirtualMemory(-1, &var_CC, 0, &dwCommiteSize, 0x1000, (ulFlags & 40000) == 0 ? 4 : 64) < 0)
{
esi = 0;
goto RELEASE_SRC;
}
if (RtlGetCurrentServiceSessionId())
{
eax = __readfsdword(0x30);
ecx = *(DWORD*)(eax + 0x50);
ecx += 0x226;
}
else
{
// from here we enter ring 0
ecx = 0x7FFE0380;
}
if (*(BYTE*)ecx != 0)
{
if (*(BYTE*)(__readfsdword(0x30) + 0x240) != 1)
{
RtlpLogHeapCommit(pHeapHandle2, var_CC, dwCommiteSize, 1);
}
}
stCommit = stCommit + dwCommiteSize;
}
// edi = HeapHandle + 258h
if (*(DWORD*)(__readfsdword(0x30) + 0x68) != 0x800)
{
/*
;ecx = *(DWORD*)(pHeapHandle2 + 0x258 + 0x7) & 0xFFFFFFF8;
*(DWORD*)(pHeapHandle2 + 0xBC) = ecx;
stReserver = ecx;
edi = *(DWORD*)(pHeapHandle2 + 0xBC) + ecx;
ulFlags |= 0x4000000;
esi = ulFlags;
*/
*(DWORD*)(pHeapHandle2 + 0xBC) = *(DWORD*)(pHeapHandle2 + 0x258 + 0x7) & 0xFFFFFFF8;
stReserver = *(DWORD*)(pHeapHandle2 + 0x258 + 0x7) & 0xFFFFFFF8;
edi = *(DWORD*)(pHeapHandle2 + 0x258 + 0x7) & 0xFFFFFFF8;;
ulFlags |= 0x4000000;
esi = ulFlags;
}
var_110[9] = (stReserver + 7) & 0xFFFFFFF8;
// ecx = var_110[9] / 8;
*(WORD*)pHeapHandle2 = var_110[9] / 8;
*(WORD*)(pHeapHandle2 + 2) = 1;
*(WORD*)(pHeapHandle2 + 7) = 1;
*(DWORD*)(pHeapHandle2 + 0x60) = 0xEEFFEEFF;
*(DWORD*)(pHeapHandle2 + 0x40) = ulFlags & 0xEFFFFFFF;
*(DWORD*)(pHeapHandle2 + 0x58) = 0;
memset((BYTE*)pHeapHandle2 + 0x1F4, 0, 0x5C);
// actually the parameter here should be *_HEAP_HANDLE;
// this encodes the heap header.
RtlpCreateHeapEncoding((PVOID)pHeapHandle2);
*(DWORD*)(pHeapHandle2 + 0x234) = 1;
if (*(DWORD*)(pHeapHandle2 + 0x40) & 0x8000000)
{
// perhaps this fetches the interceptor index.
/*
based on this guess we give the following definitions
typedef DWORD(__stdcall* INTERCEPTOR_PFN)(DWORD, DWORD, DWORD, DWORD);
DWORD
NTAPI
RtlpGetHeapInterceptorIndex(INTERCEPTOR_PFN);
INTERCEPTOR_PFN RtlpStackTraceDatabaseLogPrefix；
*/
/*
mov ecx, offset _RtlpStackTraceDatabaseLogPrefix@16 ; RtlpStackTraceDatabaseLogPrefix(x,x,x,x)
call _RtlpGetHeapInterceptorIndex@4 ; RtlpGetHeapInterceptorIndex(x)
movzx eax, ax ; note the movzx takes the low bits, so consider that
// it returns a 4-byte structure rather than a DWORD, but we don't know
// the semantics, so LOWORD is used in place of the structure.
mov [esi+58h], eax
*/
*(DWORD*)(pHeapHandle2 + 0x58) = (DWORD)LOWORD(RtlpGetHeapInterceptorIndex(RtlpStackTraceDatabaseLogPrefix));
*(DWORD*)(pHeapHandle2 + 0x40) = 0xFFFFFFBF;
}
*(DWORD*)(pHeapHandle2 + 0x44) = ulFlags & 0x6001007D;
// this reconstruction is uncertain.
/*
edi provenance:
lea edi, [edx+258h] ; edx = heapHandle
...
mov ecx, edi ; edi = heapHandle + 0x258
mov eax, [ebp+HeapHandle]
sub ecx, eax ; ecx = 0x258
mov [eax+7Eh], cx
mov eax, [ebp+HeapHandle]
*/
*(WORD*)(pHeapHandle2 + 0x7E) = LOWORD(0x258);
// ebx is the eternal 0
*(DWORD*)(pHeapHandle2 + 0x80) = 0;
//eax = pHeapHandle2 + 0xc0;
//*(DWORD*)(eax + 4) = eax;
*(DWORD*)(pHeapHandle2 + 0xc4) = pHeapHandle2 + 0xc0;
*(DWORD*)(pHeapHandle2 + 0xc0) = pHeapHandle2 + 0xc0;
//eax = pHeapHandle2 + 0x9c;
//*(DWORD*)(eax + 4) = eax;
//*(DWORD*)(eax) = eax;
*(DWORD*)(pHeapHandle2 + 0x9c + 4) = pHeapHandle2 + 0x9c;
*(DWORD*)(pHeapHandle2 + 0x9c) = pHeapHandle2 + 0x9c;
*(DWORD*)(pHeapHandle2 + 0xA4 + 4) = pHeapHandle2 + 0xA4;
*(DWORD*)(pHeapHandle2 + 0xA4) = pHeapHandle2 + 0xA4;
*(DWORD*)(pHeapHandle2 + 0x8C + 4) = pHeapHandle2 + 0x8C;
*(DWORD*)(pHeapHandle2 + 0x8C) = pHeapHandle2 + 0x8C;
if (!CriticalSectionFlag && !(ulFlags & 0xff & 1))
{
//CriticalSectionFlag = edi;
CriticalSectionFlag = pHeapHandle2 + 0x258;
if (RtlInitializeCriticalSectionEx(CriticalSectionFlag, 0, 0x10000000) < 0)
{
//return 0;
esi = 0;
goto RELEASE_SRC;
}
edi += 0x18;
ecx = CriticalSectionFlag;
}
*(DWORD*)(pHeapHandle2 + 0xC8) = CriticalSectionFlag;
*(DWORD*)(pHeapHandle2 + 0x48) |= 0x80000000;
if (!(RtlpInitializeHeapSegment(pHeapHandle2,
pHeapHandle2,
var_110[9] + 0x238,
CriticalSectionFlag,
var_110[0xA],
var_CC,
stCommit,
var_CC - var_110[0xE] + var_BC) & 0xff))
{
esi = 0;
goto RELEASE_SRC;
}
esi = 0x80;
if (pBase != 0)
{
memset((BYTE*)pHeapHandle2 + 0x258 + 0x18, 0, 0x80);
}
*(DWORD*)(pHeapHandle2 + 0x258 + 0x4) = 0x80;
*(DWORD*)(pHeapHandle2 + 0x258 + 0x1c) = pHeapHandle2 + 0x258 + 0x24;
*(DWORD*)(pHeapHandle2 + 0x258 + 0x18) = pHeapHandle2 + 0xc0;
*(DWORD*)(pHeapHandle2 + 0x258 + 0x20) = pHeapHandle2 + 0x258 + 0x24 + 0x10;
RtlpPopulateListIndex(pHeapHandle2, pHeapHandle2 + 0x258);
*(WORD*)(pHeapHandle2 + 0x7c) = 0;
*(DWORD*)(pHeapHandle2 + 0x64) = var_A8[0];
*(DWORD*)(pHeapHandle2 + 0x68) = var_A8[1];
*(DWORD*)(pHeapHandle2 + 0x6c) = var_A8[2] / 8;
*(DWORD*)(pHeapHandle2 + 0x70) = var_A8[3] / 8;
*(DWORD*)(pHeapHandle2 + 0x78) = var_A8[4] / 8;
*(DWORD*)(pHeapHandle2 + 0x5c) = (var_A8[5] + 7) / 8;
*(DWORD*)(pHeapHandle2 + 0xcc) = (var_A8[8] ^ _RtlpHeapKey);
*(DWORD*)(pHeapHandle2 + 0x250) = 4;
*(DWORD*)(pHeapHandle2 + 0x254) = 0xFE000;
if (_RtlpDisableHeapLookaside != 1)
{
//test byte ptr _RtlpDisableHeapLookaside, dl
*(DWORD*)(pHeapHandle2 + 0x48) = 1;
}
if (!(ulFlags & 0x10000))
{
*(DWORD*)(pHeapHandle2 + 0x94) = 0xf;
*(DWORD*)(pHeapHandle2 + 0x98) = 0xFFFFFFF8;
}
if (*(DWORD*)(pHeapHandle2 + 0x40) & 0x20)
{
*(DWORD*)(pHeapHandle2 + 0x94) += 8;
}
*(DWORD*)(pHeapHandle2 + 0xE4) = 0;
*(WORD*)(pHeapHandle2 + 0xE8) = 0;
*(WORD*)(pHeapHandle2 + 0xEA) = 0;
*(WORD*)(pHeapHandle2 + 0xEB) = 0;
*(DWORD*)(pHeapHandle2 + 0xE8) = 0;
// initialize the block table according to the lookaside flags.
if (((ulFlags & 3) == 2) && ((_RtlpDisableHeapLookaside & 1) == 0))
{
*(DWORD*)(pHeapHandle2 + 0x104) = RtlAllocateHeap((PMY_HEAPSTRUCT)pHeapHandle2, 0x80000A, 0x100);
if (!(*(DWORD*)(pHeapHandle2 + 0xEC)))
{
goto RELEASE_SRC;
}
*(BYTE*)(pHeapHandle2 - 1) = 1;
*(BYTE*)(pHeapHandle2 + 0xf0) = 0x80;
}
// pass in a function pointer
/*
* temporarily given the following definition
typedef DWORD(__stdcall*RtlpProcessHeapsListLock_PFN)();
DWORD
NTAPI
RtlEnterCriticalSection(RtlpProcessHeapsListLock_PFN);
RtlpProcessHeapsListLock_PFN _RtlpProcessHeapsListLock;
*/
RtlEnterCriticalSection(_RtlpProcessHeapsListLock);
// add the heap to the list
RtlpAddHeapToUnprotectedList(pHeapHandle2);
// leave the critical section
RtlLeaveCriticalSection(_RtlpProcessHeapsListLock);
if (!*(DWORD*)(pHeapHandle2 + 0x7c))
{
esi = 0;
goto RELEASE_SRC;
}
if (RtlGetCurrentServiceSessionId())
{
eax = __readfsdword(0x30);
eax = *(DWORD*)(eax + 0x50);
eax += 0x226;
}
else
{
eax = 0x7FFE0380;
}
if (*(BYTE*)eax != 0 && (*(BYTE*)(__readfsdword(0x30) + 0x240) & 1))
{
if (RtlGetCurrentServiceSessionId())
{
esi = *(DWORD*)(__readfsdword(0x30) + 0x50) + 0x226;
}
RtlpLogHeapCreateEvent(pHeapHandle2, ulFlags, var_BC, dwCommiteSize, (DWORD) * (BYTE*)esi);
}
else
{
esi = ulFlags;
}
if (RtlGetCurrentServiceSessionId())
{
eax = __readfsdword(0x30);
eax = *(DWORD*)(eax + 0x50);
eax += 0x230;
}
else
{
eax = 0x7FFE0388;
}
if (*(BYTE*)eax != 0)
{
// log the created heap range
RtlpHeapLogRangeCreate(pHeapHandle2, var_BC, ulFlags);
}
*(DWORD*)(pHeapHandle2 + 0x48) &= 0x7FFFFFFF;
*(DWORD*)(pHeapHandle2 + 0xd0) = 0;
esi = pHeapHandle2; // return value
pHeapHandle2 = 0;
CriticalSectionFlag = 0;
/*
This is the unified exit used to release resources and return.
This function returns the HeapHandle.
*/
RELEASE_SRC:
// todo
edi = (DWORD)pBase;
edx = (DWORD)pLock2;
eax = ulUnknow3;
if (eax != 0
&& eax != edx)
{
RtlDeleteCriticalSection(eax);
}
if (ulUnknow2 != 0
&& edi == 0)
{
var_BC = 0;
// eax = RtlpSecMemFreeVirtualMemory(ecx, edx, &var_BC, 0x8000)
// | BaseAddress & 0x1f << 0x10;
var_D8 = RtlpSecMemFreeVirtualMemory(ecx, (PVOID)edx, &var_BC, 0x8000)
| BaseAddress & 0x1f << 0x10;
}
RET:
return (PVOID)esi;
}Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch