Analyzing the VMProtect 3.9.4 Virtual Machine: Backward Virtual Code and the Final Boss
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
This closes out the VMProtect virtual-machine series, from the early Delphi versions up to 3.9.4 — the “final boss.” As before, two instructions at the entry are virtualized (mov eax,0x12345678 and jmp 00415035), with recognizable register constants for tracking:
0040A5A8 > B8 AAAAAAAA mov eax,0xAAAAAAAA
0040A5AD BB BBBBBBBB mov ebx,0xBBBBBBBB
0040A5B2 B9 CCCCCCCC mov ecx,0xCCCCCCCC
0040A5B7 BA DDDDDDDD mov edx,0xDDDDDDDD
0040A5BC BE 51515151 mov esi,0x51515151
0040A5C1 BF D1D1D1D1 mov edi,0xD1D1D1D1
0040A5C6 B8 78563412 mov eax,0x12345678
0040A5CB E9 65AA0000 jmp 123.00415035
0040A5D0 C3 retn
0040A5D1 90 nopThe entry stub changed again — instead of a single jmp, the protected instruction is now replaced with a push/push/call/pop/jmp sequence:
0040A5C6 |. 57 push edi
0040A5C7 |. 53 push ebx
0040A5C8 |. E8 B4AB0B00 call 123_vmp.004C5181
0040A5CD |. 59 pop ecx
0040A5CE \.- FFE5 jmp ebpThe prologue still uses DSP (displacement) encryption from 3.8.1:
004C5181 9C pushfd
004C5182 BF 9C9C823B mov edi,0x3B829C9C
004C5187 BB 8C0F063D mov ebx,0x3D060F8C
004C518C 81F3 A36618C2 xor ebx,0xC21866A3
;flags change
004C5192 8BBC3C 70637DC4 mov edi,dword ptr ss:[esp+edi-0x3B829C90]
;== mov edi,[esp+0xC] -- dsp = 0xC, encrypted
;edi restored to D1D1D1D1
004C5199 C74424 0C 0895A>mov dword ptr ss:[esp+0xC],0x8BA49508
;[0019FF70] = 8BA49508
004C51A1 8B5C24 08 mov ebx,dword ptr ss:[esp+0x8]
;ebx restored to BBBBBBBB
004C51A5 E8 3294FFFF call 123_vmp.004BE5DCRegister roles and backward virtual code
After the pushes and initialization, the register assignments for 3.9.4 are:
| Register | Role |
|---|---|
ebx |
the decryption key |
esp |
virtual-register base |
ebp |
handler address |
esi |
virtual-code pointer — stored backwards |
edi |
stack read/write scratch (like a real esp) |
The stack layout after the pushes:
0019FF50 D1D1D1D1 ; edi
0019FF54 00000000 ; relocation
0019FF58 BBBBBBBB ; ebx
0019FF5C 00000246 ; eflags
0019FF60 51515151 ; esi
0019FF64 0019FF80 ; ebp
0019FF68 DDDDDDDD ; edx
0019FF6C CCCCCCCC ; ecx
0019FF70 AAAAAAAA ; eaxThe big new behavior in 3.9.4 is that the virtual code is stored backwards. esi starts at virtual_code + 4 and reads at negative offsets ([esi-0x4], [esi-0x6], [esi-0x9], …), decrementing as it goes. This also changes the first-round key: in 3.8.1 the initial key was esi’s starting value, but here it’s esi + 4. The decryption loop:
0047D8DA BD 3FCD892B mov ebp,0x2B89CD3F
0047D8DF 8BDE mov ebx,esi
;the virtual-code address into ebx (the key)
...
0047D8FA 8D2D F2D84700 lea ebp,dword ptr ds:[0x47D8F2]
;an in-code address into ebp, used to compute the next handler
...
0047D905 8DB486 0439FDFF lea esi,dword ptr ds:[esi+eax*4-0x2C6FC]
;== sub esi, 4
...
0047D90E 8B940E 424EFFFF mov edx,dword ptr ds:[esi+ecx-0xB1BE]
;== mov edx,[esi] -- read 4 bytes of virtual code
0047D919 33D3 xor edx,ebx ;decrypt the handler-address addend
...The first handler: filling vm_edi, vm_relocation, vm_ebx
0050AA79 8B4E FC mov ecx,dword ptr ds:[esi-0x4]
;read 4 bytes at esi-4 -- consistent with backward storage
0050AA7C B8 879910BE mov eax,0xBE109987
0050AA81 0FB6D0 movzx edx,al
0050AA84 33CB xor ecx,ebx ;decrypt handler-address addend
0050AA86 92 xchg eax,edx
0050AA87 F7D1 not ecx
0050AA89 0FB3C2 btr edx,eax
0050AA8C D1C1 rol ecx,1
0050AA8E c0e2 e6 shl dl,0xe6
0050AA91 66:c1e2 8c shl dx,0x8c
0050AA95 c0e0 e3 shl al,0xe3
0050AA98 49 dec ecx
0050AA99 0AC0 or al,al
0050AA9B F7D9 neg ecx ;decrypt done
0050AA9D 33D9 xor ebx,ecx ;new key
0050AA9F 03E9 add ebp,ecx ;next handler address
0050AAA1 66:0FA3D2 bt dx,dx
0050AAA5 8B5407 C8 mov edx,dword ptr ds:[edi+eax-0x38]
;read edi's value from the stack
...
0050AAAF 0FB68C06 FB1FFF>movzx ecx,byte ptr ds:[esi+eax-0xE005]
;read one byte of virtual code
0050AAB7 32CB xor cl,bl ;decrypt the virtual-register offset
...
0050AACD FEC9 dec cl ;offset decryption done
...
0050AAD9 03CC add ecx,esp
;address of virtual register at offset 0x4
0050AADE 891421 mov dword ptr ds:[ecx],edx
;store edi's value into vm_edi (offset 0x4)The handler continues through several chained code blocks (the jmp edx connections from 3.8.1) and fills two more registers — vm_relocation (offset 0x2c) and vm_ebx (offset 0x38) — before retn 0xC dispatches the next handler. The second and third handlers fill the rest, yielding the full virtual-register map:
19fe8c unused edi eflags eax
19fe9c unused unused unused esi
19feac ecx unused unused vm_relocation
19febc unused unused ebx edx
19fecc ebp unused unused unusedSo the offsets are: vm_edi=0x4, vm_relocation=0x2c, vm_ebx=0x38, vm_eflags=0x8, vm_esi=0x1c, vm_ebp=0x40, vm_edx=0x3c, vm_ecx=0x20, vm_eax=0xC. Note vm_ebp at 0x40 again confirms the frame is larger than the old 0x40 total.
Handling the two protected instructions
The fourth handler decrypts the immediate (0x12345678) and stores it to virtual register offset 0x10:
004A5D91 8B4E FC mov ecx,dword ptr ds:[esi-0x4]
;read 4 bytes of virtual code
...
004A5D9F 33CB xor ecx,ebx
;decrypt the immediate (0x12345678)
004A5DA1 E9 CB560000 jmp 123_vmp.004AB471004AB471 c0fa c2 sar dl,0xc2
004AB474 C1C9 02 ror ecx,0x2
004AB477 FEC8 dec al
004AB479 0AC2 or al,dl
004AB47B 0FC9 bswap ecx
004AB47D 41 inc ecx
004AB47E D1C1 rol ecx,1 ;immediate decryption done
...
004AB484 898C17 1280FFFF mov dword ptr ds:[edi+edx-0x7FEE],ecx
;store the immediate at 0019FF70The fifth handler decrypts the jmp displacement (0x00415035) and stores it at 0019FF70 as well:
004D24DD B9 BD5E15C1 mov ecx,0xC1155EBD
004D24E2 E8 F73CF9FF call 123_vmp.004661DE004661DE 8D048D A43E85C5 lea eax,dword ptr ds:[ecx*4-0x3A7AC15C]
004661E5 81EE 04000000 sub esi,0x4
...
004661FC 8B8416 6898638E mov eax,dword ptr ds:[esi+edx-0x719C6798]
;read 4 bytes of virtual code
00466203 49 dec ecx
00466204 33C3 xor eax,ebx
;decrypt the jmp displacement (00415035)
00466206 C1C8 02 ror eax,0x2
...
00466224 D1C0 rol eax,1 ;displacement decryption done
00466226 33D8 xor ebx,eax ;new key
...
0046622F 8984CF F07F8A9F mov dword ptr ds:[edi+ecx*8-0x60758010],eax
;store the displacement at 19ff70Dumping virtual registers to the stack
The sixth through tenth handlers pop the virtual registers onto the stack in reverse order. They’re structurally identical to each other (same decryption chain, differing only in which virtual register they read), so I’ll summarize:
| Stack address | Value | Source |
|---|---|---|
0019FF50 |
D1D1D1D1 |
vm_edi |
0019FF54 |
BBBBBBBB |
vm_ebx |
0019FF58 |
00000246 |
vm_eflags |
0019FF5C |
51515151 |
vm_esi |
0019FF60 |
0019FF80 |
vm_ebp |
0019FF64 |
DDDDDDDD |
vm_edx |
0019FF68 |
CCCCCCCC |
vm_ecx |
0019FF6C |
12345678 |
the virtualized mov eax,0x12345678 |
0019FF70 |
00415035 |
the virtualized jmp 00415035 |
Exiting the VM
The eleventh and final handler restores the real registers and executes both protected instructions:
0049AC92 8BE7 mov esp,edi ;prepare to restore registers
0049AC94 5F pop edi ;restore edi
0049AC95 5B pop ebx ;restore ebx
0049AC96 B8 8ECD941D mov eax,0x1D94CD8E
0049AC9B 9D popfd ;restore eflags
0049AC9C 5E pop esi ;restore esi
0049AC9D 0FB6C8 movzx ecx,al
0049ACA0 98 cwde
0049ACA1 5D pop ebp ;restore ebp
0049ACA2 8D14CD 3CE01F73 lea edx,dword ptr ds:[ecx*8+0x731FE03C]
0049ACA9 E9 C8340400 jmp 123_vmp.004DE176004DE176 87CA xchg edx,ecx
004DE178 5A pop edx ;restore edx
004DE179 51 push ecx
004DE17A 8B8C0C 581BE08C mov ecx,dword ptr ss:[esp+ecx-0x731FE4A8] ;restore ecx
004DE181 864424 01 xchg byte ptr ss:[esp+0x1],al
004DE185 874424 00 xchg dword ptr ss:[esp],eax
004DE189 ^ E9 C37BF6FF jmp 123_vmp.00445D5100445D51 8B8404 5C71E08C mov eax,dword ptr ss:[esp+eax-0x731F8EA4]
;eax = 12345678 -- executes the first protected instruction
00445D58 68 B5BFAB16 push 0x16ABBFB5
00445D5D 8D6424 10 lea esp,dword ptr ss:[esp+0x10]
00445D61 C3 retn
;leave the VM -- executes the second protected instruction (jmp 00415035)Summary
3.9.4’s material change over 3.8.1 is the backward storage of the virtual code (esi reads at negative offsets and the initial key shifts by 4). Everything else — the DSP-encrypted displacements, the double-encrypted virtual code, the key in ebx, the fragmented handlers chained with computed jumps — carries forward from 3.8.1, with the register roles shuffled once more.
This wraps the full 1.x → 3.9.4 virtualization branch. It’s still a long way from “fully analyzed” (instruction mutation, anti-debug, and memory protection remain), but the execution framework is now mapped end to end — which is the foundation everything else builds on.
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch