Analyzing the VMProtect 2.12.3 Virtual Machine: Handler Flow and Virtual-Register Mapping
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
This continues a series on VMProtect’s virtual machine. 2.12.3 is the last widely available 2.x build written in Delphi (3.0 rewrote it in C++), and — since the analysis plugins only officially support up to 2.08 — it’s the bridge into the 3.x line. The target is a VMP-protected HelloASM.vmp1, compiled with “fastest speed” and 5 nop bytes at the entry encrypted (a 12 KB exe). Everything below is done in OllyDbg.
VM entry: what changed since 1.64
For context, here’s the 1.64 VM entry (the push-register prologue):
004050E6 |$ 56 push esi
004050E7 |. 53 push ebx
004050E8 |. 50 push eax
004050E9 |. 53 push ebx
004050EA |. 51 push ecx
004050EB |. 9C pushfd
004050EC |. 52 push edx
004050ED |. 57 push edi
004050EE |. 55 push ebp
004050EF |. 68 00000000 push 0x0
004050F4 |. 8B7424 2C mov esi,dword ptr ss:[esp+0x2C]
004050F8 |. 89E5 mov ebp,esp
004050FA |. 81EC C0000000 sub esp,0xC0Which left the stack like this at EIP = 004050FA:
0019FF44 00000000 ;1 push 0x0
0019FF48 0019FF80 ;2 push ebp
0019FF4C 00401000 ;3 push edi
0019FF50 00401000 ;4 push edx
0019FF54 00000246 ;5 pushfd
0019FF58 00401000 ;6 push ecx
0019FF5C 003C9000 ;7 push ebx
0019FF60 0019FFCC ;8 push eax
0019FF64 003C9000 ;9 push ebx
0019FF68 00401000 ;10 push esi
0019FF6C 0040578C ;return address from the preceding CALL
0019FF70 00405790 ;the virtual code addressIn 2.12.3 the entry is no longer push A; call B. It opens with a jmp:
00401000 > $- E9 8E480000 jmp HelloASM.0040589300405893 9C pushfd
00405894 E9 73010000 jmp HelloASM.00405A0CIn 1.6, A was the encrypted virtual-code address and pushfd filled a slot. At 00405894 the eight GPRs and flags are:
EAX 0019FFCC
ECX 00401000
EDX 00401000
EBX 00228000
ESP 0019FF70
EBP 0019FF80
ESI 00401000
EDI 00401000
Eflag 00000246Then the prologue, now laced with junk instructions:
00405A0C E8 00000000 call HelloASM.00405A11
00405A11 C74424 04 05594>mov dword ptr ss:[esp+0x4], 00405905
00405A19 882C24 mov byte ptr ss:[esp],ch ; junk
00405A1C 66:C70424 EE70 mov word ptr ss:[esp],0x70EE ; junk
00405A22 880424 mov byte ptr ss:[esp],al ; junk
00405A25 C70424 2F1D0185 mov dword ptr ss:[esp],0x85011D2F; junk
00405A2C 9C pushfd
00405A2D 8D6424 04 lea esp,dword ptr ss:[esp+0x4]
00405A31 ^ E9 C1E9FFFF jmp HelloASM.004043F7The call + mov [esp+4], 00405905 pair is the same “push A, call B” trick as before; pushfd + lea esp,[esp+4] cancels to nothing (junk). Then the register pushes:
004043F7 60 pushad
004043F8 9C pushfd
004043F9 885C24 04 mov byte ptr ss:[esp+0x4],bl
004043FD FF7424 14 push dword ptr ss:[esp+0x14] ; push ebx
00404401 f3:9c rep pushfd ; pushfd
00404403 8F4424 24 pop dword ptr ss:[esp+0x24]
00404407 9C pushfd
00404408 C60424 2B mov byte ptr ss:[esp],0x2B
0040440C 9C pushfd
0040440D 9C pushfd
0040440E 8D6424 30 lea esp,dword ptr ss:[esp+0x30]
00404412 E9 C1110000 jmp HelloASM.004055D8This does the same job as the 1.64 prologue but is far harder to read. The trick: rather than single-step through it, set a breakpoint at 004050B2 (after the stack frame is built), dump the stack, and compare it against the saved registers — the layout comes back:
0019FF3C 0019FF33
0019FF40 00000000 ; push 0
0019FF44 00000000
0019FF48 0019FF80 ; push ebp
0019FF4C 00401000 ; push ecx
0019FF50 00401000 ; push edx
0019FF54 00228000 ; push ebx
0019FF58 00401000 ; push esi
0019FF5C 0019FFCC ; push eax
0019FF60 00401000 ; push edi
0019FF64 00401000 ; push duplicate
0019FF68 00000246 ; pushfd
0019FF6C 85011D2F ; the CALL return-address slot
0019FF70 00405905 ; the virtual code addressAt this point esp = 0019FF3C, ebp = 0019FF40 (the extra unannotated slots are almost certainly there to complicate virtual-code execution). Then the virtual-machine frame:
004050B2 81EC BC000000 sub esp,0xBC ; build the frame
004050B8 66:0FBAF9 07 btc cx,0x7
004050BD D2D8 rcr al,cl
004050BF 89E7 mov edi,esp ; edi = virtual-register base
004050C1 D2E0 shl al,cl
004050C3 66:0FBDC9 bsr cx,cx
004050C7 F5 cmc
004050C8 0375 00 add esi,dword ptr ss:[ebp] ; relocation-relatedHandler dispatch
The dispatch loop fetches a virtual-code byte, uses it to index a handler table, and decrypts the resulting address:
0040463E 8A06 mov al,byte ptr ds:[esi] ; fetch virtual code
00404640 66:0FACC9 02 shrd cx,cx,0x2
00404645 F6D1 not cl
00404647 D2E9 shr cl,cl
00404649 0FB6C0 movzx eax,al
0040464C 80F1 55 xor cl,0x55
0040464F 8B0C85 1A4C4000 mov ecx,dword ptr ds:[eax*4+0x404C1A] ; handler addr (still encrypted)
00404656 E9 81050000 jmp HelloASM.00404BDC00404BDC 83EE FF sub esi,-0x1 ; == inc esi00404BF7 81F1 CE66B527 xor ecx,0x27B566CE ; decrypt handler address00404C02 894C24 24 mov dword ptr ss:[esp+0x24],ecx ; store handler addr
00404C06 C60424 0E mov byte ptr ss:[esp],0xE
00404C0A FF7424 24 push dword ptr ss:[esp+0x24] ; push handler addr
00404C0E C2 2800 retn 0x28 ; jump to handlerMechanically this is the same as 1.6/1.2. The difference: in this version the table entry [eax*4 + handler_base] is encrypted, and it’s the handler that’s decrypted — so the virtual-register offset that used to be recovered from the same byte now needs a second bytecode byte (the dispatch reads two bytes: 09 and 3C, where 3C is the virtual-register offset).
The first handler: filling a virtual register
00404125 F5 cmc
00404126 0FC0C2 xadd dl,al
00404129 66:29CA sub dx,cx
0040412C 66:0FBDD3 bsr dx,bx
00404130 8A06 mov al,byte ptr ds:[esi] ; fetch virtual-register offset
00404132 66:81C2 F096 add dx,0x96F0
00404137 8B55 00 mov edx,dword ptr ss:[ebp] ; previously-pushed register value into edx
0040413A F5 cmc
0040413B E8 13010000 call HelloASM.0040425300404253 8D76 01 lea esi,dword ptr ds:[esi+0x1] ; == inc esi
00404256 84FF test bh,bh
00404258 83C5 04 add ebp,0x4
0040425B 9C pushfd
0040425C 60 pushad
0040425D 891438 mov dword ptr ds:[eax+edi],edx ; write the virtual register
00404260 FF7424 08 push dword ptr ss:[esp+0x8]
00404264 68 58E5DA5D push 0x5DDAE558
00404269 9C pushfd
0040426A 8D6424 34 lea esp,dword ptr ss:[esp+0x34]
0040426E E9 580E0000 jmp HelloASM.004050CB004050CB 20E8 and al,ch
004050CD E8 6BF5FFFF call HelloASM.0040463D ; back to the dispatch loopThat’s the first handler: still a “fill virtual register” handler that writes 0 (from 0019FF40) into the virtual register. Note the missing and al, 0x3C from earlier versions — the virtual-register offset is now carried in its own bytecode byte.
A second handler is a decoy: it reads 4 bytes from virtual code and writes them to [ebp], which just subtracts 4 from ebp — its only real effect, and the third handler can be skipped. The author deliberately salted the flow with junk handlers and conditional jumps that are actually unconditional (e.g. cmp ebp,eax where both are constant, so ja always jumps).
The virtual-register fill sequence
The next sixteen handlers each fill one virtual register. The mapping between the two-byte virtual code (09 3C, 04 24, …) and the pushed registers is:
| Virtual-register offset | Register pushed at entry |
|---|---|
0x3C |
push 0 |
0x0C |
(junk) |
0x24 |
push ebp |
0x08 |
push ecx |
0x34 |
push edx |
0x28 |
push ebx |
0x00 |
push esi |
0x10 |
push eax |
0x04 |
push edi |
0x14 |
push duplicate |
0x38 |
push eflag |
0x2C |
0x85011D2F |
0x30 |
the virtual code address |
After the fill, the virtual-register frame looks like:
0019FE80 esi edi ecx junk
0019FE90 eax dup unfilled unfilled
0019FEA0 unfilled ebp ebx 85011D2F
0019FEB0 code addr edx eflag relocation 0The “junk” and “unfilled” virtual-register slots (and possibly more) can be treated as scratch — the rest are only used to hold computed results while emulating the protected code.
Virtual-register shuffling
This is the big change from earlier versions: virtual registers get swapped within a single flow block (in ≤1.6 they only differed across blocks). The eflag register, for example, starts at offset 0x38, moves to 0x2C, then to 0x34. A representative handler:
0040521D 80CE FE or dh,0xFE
00405220 8A06 mov al,byte ptr ds:[esi]
00405222 D0CE ror dh,1
00405224 21D2 and edx,edx
00405226 8B1438 mov edx,dword ptr ds:[eax+edi] ; read eflag into edx
00405229 66:0FBAE0 09 bt ax,0x9
0040522E 83ED 04 sub ebp,0x4 ; ebp = 0019FF70
00405231 F5 cmc
00405232 60 pushad
00405233 E8 23000000 call HelloASM.0040525B0040525B 8955 00 mov dword ptr ss:[ebp],edx ; [0019FF70] = eflag
0040525E F8 clc
0040525F 60 pushad
00405260 886424 04 mov byte ptr ss:[esp+0x4],ah
00405264 83EE FF sub esi,-0x1 ; == inc esi
00405267 66:C74424 08 0C>mov word ptr ss:[esp+0x8],0xAB0C ; junk
0040526E 885C24 08 mov byte ptr ss:[esp+0x8],bl ; junk
00405272 C64424 14 E6 mov byte ptr ss:[esp+0x14],0xE6 ; junk
00405277 8D6424 44 lea esp,dword ptr ss:[esp+0x44] ; junk
0040527B E9 97020000 jmp HelloASM.00405517That’s also junk — the shuffle is the real work. From pseudo-code address 0040597B onward, the virtual registers are effectively copied onto the stack and then freely re-mapped to different offsets.
The modified-ESI trick
One handler deserves a closer look because it rewrites esi (the virtual instruction pointer) — a computed jump:
004047EE 66:0FCE bswap si ; junk
004047F1 66:0FBCF7 bsf si,di ; junk
004047F5 9C pushfd
004047F6 66:0FABD6 bts si,dx ; junk
004047FA 8B75 00 mov esi,dword ptr ss:[ebp] ; esi = 00405899
004047FD 66:0FBAE1 0D bt cx,0xD
00404802 84DB test bl,bl
00404804 F9 stc
00404805 66:0FBAE0 08 bt ax,0x8
0040480A 83C5 04 add ebp,0x4
0040480D 68 E70B3726 push 0x26370BE7
00404812 8D6424 08 lea esp,dword ptr ss:[esp+0x8]
00404816 E9 A6080000 jmp HelloASM.004050C1004050C1 D2E0 shl al,cl
004050C3 66:0FBDC9 bsr cx,cx
004050C7 F5 cmc
004050C8 0375 00 add esi,dword ptr ss:[ebp] ; relocation: ebp holds offset 0x3C (=0)
004050CB 20E8 and al,ch
004050CD E8 6BF5FFFF call HelloASM.0040463DAfter this, esi = 0x00405899. Notably, the computed value for esi was encrypted and decrypted through multiple passes — yet the final exit address (0x401005) is stored unencrypted in the virtual code, unlike ≤1.6 where the exit was also encrypted.
Exiting the VM
Finally the virtual registers are dumped back to the stack and popped into the real registers:
004056DD 66:D3FF sar di,cl
004056E0 66:0FBAF3 06 btr bx,0x6
004056E5 38DE cmp dh,bl
004056E7 89EC mov esp,ebp ; esp = 0019FF44
004056E9 66:11E1 adc cx,sp
004056EC FC cld
004056ED 59 pop ecx ; ecx = AD7B4B1E (only advances esp)
004056EE 9F lahf
004056EF 5B pop ebx ; ebx = 00401000 (only advances esp)
004056F0 66:0FC1EE xadd si,bp
004056F4 D5 B9 aad 0xB9
004056F6 66:0FC1D6 xadd si,dx
004056FA 5D pop ebp ; ebp = 0019FF80
004056FB D4 7A aam 0x7A
004056FD 5E pop esi ; esi = 00401000
004056FE 3F aas
004056FF D2D4 rcl ah,cl
00405701 E8 F3F2FFFF call HelloASM.004049F9004049F9 66:21DF and di,bx
004049FC 8B7C24 04 mov edi,dword ptr ss:[esp+0x4] ; junk
00404A00 99 cdq
00404A01 8B5C24 08 mov ebx,dword ptr ss:[esp+0x8] ; ebx = 00228000
00404A05 F5 cmc
00404A06 80FA 7A cmp dl,0x7A
00404A09 8B7C24 0C mov edi,dword ptr ss:[esp+0xC] ; edi = 00401000
00404A0D 80E5 E5 and ch,0xE5
00404A10 0FBAF8 0A btc eax,0xA
00404A14 66:0FB6C1 movzx ax,cl
00404A18 8B4424 10 mov eax,dword ptr ss:[esp+0x10] ; eax = 0019FFCC
00404A1C 66:81D1 6348 adc cx,0x4863
00404A21 8B5424 14 mov edx,dword ptr ss:[esp+0x14] ; edx = 00401000
00404A25 F5 cmc
00404A26 9C pushfd
00404A27 8B4C24 1C mov ecx,dword ptr ss:[esp+0x1C] ; ecx = 00401000
00404A2B 9C pushfd
00404A2C F5 cmc
00404A2D FF7424 24 push dword ptr ss:[esp+0x24]
00404A31 9D popfd
00404A32 66:C74424 24 3E>mov word ptr ss:[esp+0x24],0xAA3E
00404A39 886424 0C mov byte ptr ss:[esp+0xC],ah
00404A3D FF7424 28 push dword ptr ss:[esp+0x28]
00404A41 C2 2C00 retn 0x2C ; return to 0x401005The final stack layout as the registers restore:
0019FF4C 0019FF80 ebp
0019FF50 00401000 esi
0019FF54 461BFFD3
0019FF58 002A6000 ebx
0019FF5C 00401000 edi
0019FF60 0019FFCC eax
0019FF64 00401000 edx
0019FF68 00401000 ecx
0019FF6C 00000246 eflagSummary
Compared to 1.6, 2.12.3 is a real step up:
- virtual registers are swapped within a single flow block, not just across blocks
- the computed
esivalue is encrypted, but the exit address is now plaintext in the virtual code - virtual-register read/write pseudo-code is split into two bytes instead of one
- the handler table entries are encrypted, requiring the extra bytecode byte to carry the virtual-register offset
The overall execution framework is unchanged, which is what makes it predictable once you’ve mapped one version — but each of these changes materially raises the cost of writing an analysis plugin.
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch