Fixing the VMProtect 3.5.1 Win32 Runtime Compilation Issue
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
I kept hitting this and never found a clean answer: when you build the VMProtect 3.5.1 win32 runtime yourself and generate the .inc file, the protected program refuses to run. Plenty of people following the build guides run into it — VS2010 or VS2008, it doesn’t matter. I’d asked around a couple of times and got nowhere, so eventually I sat down and traced it myself.
The cause is compiler optimization. (After the fix described below, build with VS2010 release/32-bit — I haven’t verified other configurations.)
The root cause
In loader.cc there are two consecutive tNtQueryInformationProcess calls:
- if (NT_SUCCESS(reinterpret_cast<tNtQueryInformationProcess *>(syscall | sc_query_information_process)(process, ProcessDebugPort, &debug_object, sizeof(debug_object), NULL)) && debug_object != 0) {
- LoaderMessage(mtDebuggerFound);
- return LOADER_ERROR;
- }
- debug_object = 0;
- if (NT_SUCCESS(reinterpret_cast<tNtQueryInformationProcess *>(syscall | sc_query_information_process)(process, ProcessDebugObjectHandle, &debug_object, sizeof(debug_object), reinterpret_cast<PULONG>(&debug_object)))
- || debug_object == 0) {
- LoaderMessage(mtDebuggerFound);
- return LOADER_ERROR;Even under VS2010/2008, optimization turns this into:
mov eax, fake_func_addr
mov [ebp+xxx], eax
...
call [ebp+xxx]These runtime addresses have to be re-filled with the real address in the final code. In intel.cc, IntelFunction::ReadFromBuffer handles that call replacement. VMProtect expects to see:
MOV EAX, 0XFACE_ID
CALL EAXWhen it instead encounters the optimized call [ebp+xxx], the pattern is misidentified and the replacement never happens — which is what breaks everything downstream:
else if (command->type() == cmCall) {
if (command->operand(0) == operand) {
command->Init(cmSyscall, command->operand(0));
/*
Because of compiler optimization, `call fake_syscall` causes IntelFunction::ReadFromBuffer to misidentify this FACE_SYSCALL. ReadFromBuffer only recognizes `reg = face_syscall | funcid` followed by `call reg`. The compiler optimized two consecutive such calls into `[mem] = face_syscall | funcid` then `call [mem]`, which is not recognized.
*/
static int fuckingcall =0;
printf("found FACE_SYSCALL call %d at last syscall | \n",fuckingcall++);
command->CompileToNative();
command->include_option(roNoNative);
syscall_found = true;
}
if (operand.type != otRegistr || operand.registr == regEAX)
is_end = true;
}So the call address at those two sites never gets rewritten.
Fix 1: split the consecutive calls
Restructure the logic so the two consecutive tNtQueryInformationProcess calls are no longer adjacent. That keeps the compiler from folding them into the [mem] form. (This is the approach I went with — I didn’t want to fight the optimizer.)
if (data.options() & LOADER_OPTION_CHECK_DEBUGGER) {
if(sc_query_information_process){
HANDLE debug_object = 0;
{
if (NT_SUCCESS(reinterpret_cast<tNtQueryInformationProcess *>(syscall | sc_query_information_process)(process, ProcessDebugObjectHandle, &debug_object, sizeof(debug_object), reinterpret_cast<PULONG>(&debug_object)))
|| debug_object == 0) {
LoaderMessage(mtDebuggerFound);
return LOADER_ERROR;
}
}
}
if (peb->BeingDebugged) {
LoaderMessage(mtDebuggerFound);
return LOADER_ERROR;
}
if (sc_query_information_process) {
// disable InstrumentationCallback
if (peb->OSMajorVersion > 6) {
// Windows 10
PROCESS_INSTRUMENTATION_CALLBACK_INFORMATION info;
info.Version = 0;
info.Reserved = 0;
info.Callback = NULL;
if (tNtSetInformationProcess *set_information_process = reinterpret_cast<tNtSetInformationProcess *>(LoaderGetProcAddress(ntdll, reinterpret_cast<const char *>(FACE_NT_SET_INFORMATION_PROCESS_NAME), true)))
set_information_process(process, ProcessInstrumentationCallback, &info, sizeof(info));
}
HANDLE debug_object;
/*
syscall|sc_query_information_process
*/
{
if (NT_SUCCESS(reinterpret_cast<tNtQueryInformationProcess *>(syscall | sc_query_information_process)(process, ProcessDebugPort, &debug_object, sizeof(debug_object), NULL)) && debug_object != 0) {
LoaderMessage(mtDebuggerFound);
return LOADER_ERROR;
}
}
} else if (tNtQueryInformationProcess *query_information_process = reinterpret_cast<tNtQueryInformationProcess *>(LoaderGetProcAddress(ntdll, reinterpret_cast<const char *>(FACE_NT_QUERY_INFORMATION_PROCESS_NAME), true))) {
HANDLE debug_object;
if (NT_SUCCESS(query_information_process(process, ProcessDebugPort, &debug_object, sizeof(debug_object), NULL)) && debug_object != 0) {
LoaderMessage(mtDebuggerFound);
return LOADER_ERROR;
}
if (NT_SUCCESS(query_information_process(process, ProcessDebugObjectHandle, &debug_object, sizeof(debug_object), NULL))) {
LoaderMessage(mtDebuggerFound);
return LOADER_ERROR;
}
}
// disable debugger
if (sc_set_information_thread)
reinterpret_cast<tNtSetInformationThread *>(syscall | sc_set_information_thread)(thread, ThreadHideFromDebugger, NULL, 0);
else if (tNtSetInformationThread *set_information_thread = reinterpret_cast<tNtSetInformationThread *>(LoaderGetProcAddress(ntdll, reinterpret_cast<const char *>(FACE_NT_SET_INFORMATION_THREAD_NAME), true)))
set_information_thread(thread, ThreadHideFromDebugger, NULL, 0);
}Fix 2: hardcode the call eax form in assembly
The cleaner, more robust option is to add a separate assembly object file that hardcodes the call as call eax, so you don’t depend on the compiler emitting the right pattern at all. This survives a change of compiler.
The lesson here is that the .inc/runtime build step isn’t a pure recompile — VMProtect’s post-processing depends on specific instruction patterns in the emitted code, and the optimizer will happily destroy those patterns if you let it. Once you know what IntelFunction::ReadFromBuffer is looking for, the failure mode is obvious.
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch