Analyzing the VMProtect 3.0.0beta Virtual Machine: Handler Chaining and Double Encryption
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
This continues the VMProtect virtual-machine series, moving from the last Delphi build (2.12.3) into the 3.x line. Starting with 3.0, the whole thing was rewritten in C++ (with a Qt UI), and the encrypted output ballooned — this sample grows from 179 KB to 590 KB under protection. The target is the 3.0.0beta. Two instructions at the entry are virtualized (mov eax,0x12345678 and jmp 00415035), with the other registers set to recognizable constants so their movement through the VM is easy to track:
0040A5A8 > B8 AAAAAAAA mov eax,0xAAAAAAAA
0040A5AD BB BBBBBBBB mov ebx,0xBBBBBBBB
0040A5B2 B9 CCCCCCCC mov ecx,0xCCCCCCCC
0040A5B7 BA DDDDDDDD mov edx,0xDDDDDDDD
0040A5BC BE 51515151 mov esi,0x51515151
0040A5C1 BF D1D1D1D1 mov edi,0xD1D1D1D1
0040A5C6 B8 78563412 mov eax,0x12345678
0040A5CB E9 65AA0000 jmp 123.00415035
0040A5D0 C3 retn
0040A5D1 90 nopAfter protection, the two virtualized instructions become:
0040A5C6 .- E9 CCC20800 jmp 123_vmp.00496897
0040A5CB >^ FFE7 jmp edi
0040A5CD . 57 push edi
0040A5CE . C3 retn
0040A5CF . 4F dec edi
0040A5D0 . C3 retn
0040A5D1 90 nop0040A5C6 is replaced with a jmp into the VM; 0040A5CB becomes what looks like a handler fragment. Stepping into the entry:
00496897 68 5AFA2B26 push 0x262BFA5A
0049689C E8 05BFFBFF call 123_vmp.004527A6The same push A; call B opener as 1.6. But the prologue is where things diverge. The key region, with junk elided:
004527A6 9C pushfd
------省略------
004527CD 8B7424 28 mov esi,dword ptr ss:[esp+0x28]
------省略------
00452811 8BEC mov ebp,esp
------省略------
0045281C 8DA424 40FFFFFF lea esp,dword ptr ss:[esp-0xC0]
------省略------
0045282B > 8BDE mov ebx,esi;ebx存放esi的值作为密钥
------省略------
0045283B 8D3D 3B284500 lea edi,dword ptr ds:[0x45283B]
00452841 8B06 mov eax,dword ptr ds:[esi]
00452843 F9 stc
00452844 8DB6 04000000 lea esi,dword ptr ds:[esi+0x4]
0045284A 33C3 xor eax,ebx;解密
0045284C 48 dec eax解密
0045284D 66:3D 8A58 cmp ax,0x588A
00452851 66:85D7 test di,dx
00452854 0FC8 bswap eax解密
00452856 2D 6D7E3B00 sub eax,0x3B7E6D解密
0045285B 35 412CD300 xor eax,0xD32C41解密
00452860 F6C6 C6 test dh,0xC6
00452863 33D8 xor ebx,eax;生成下一次解密的密钥
00452865 F8 clc
00452866 03F8 add edi,eax ;算出handler的地址
00452868 57 push edi
00452869 C3 retn ;执行handlerComparing against the 1.64 entry:
- At
004527CDthe duplicated register slot from earlier versions is gone —esiis still encrypted at that point, and only gets decrypted by00452811. 0045281Cis justsub esp, 0xC0.0045283Bis new:ediis no longer the virtual-register base (that role moves toesp).edinow holds the handler address.00452841reads 4 bytes of virtual code, not 1.
The handler lookup is no longer the [eax*4 + base] vtable-style table of earlier versions. Instead the handler is chained: the current handler’s address plus a decrypted 4-byte value yields the next handler.
Double encryption of the virtual code
The per-handler pseudocode is now double-encrypted. The outer layer is a sequence of arithmetic ops (not, inc, bswap, dec, neg, …); the inner layer xors/adds/subtracts with a running key that is itself the pseudocode address. The key lives in ebx (initialized from esi on the first round), and every round it’s updated by xoring with the decrypted value:
xor ebx,eax ; generate the key for the next round
add edi,eax ; edi = decrypt(pseudocode) + edi -> next handler addressThe pseudocode length can be 4, 2, or 1 byte(s); 2- and 1-byte variants update only bx or bl of the key. The decryption order is the exact reverse of encryption: first the inner xor/add/sub using the pseudocode address as key, then the arithmetic sequence, then the new key is computed.
The practical consequence: to enumerate every handler address you must know the address of the previous handler. The author’s entire redesign goal was to prevent dumping the full handler table and translating all virtual code back into normal instructions.
The first handler
Following the retn at 00452869 lands on the first handler:
00447687 0FB606 movzx eax,byte ptr ds:[esi];取出一个字节的伪代码
0044768A 66:D3C9 ror cx,cl
0044768D 81C6 01000000 add esi,0x1;;伪代码指针加1
00447693 03CF add ecx,edi
00447695 81F1 7B41E822 xor ecx,0x22E8417B
0044769B 66:0FBAF1 41 btr cx,0x41
004476A0 32C3 xor al,bl ;解密伪代码
004476A2 0FBFCC movsx ecx,sp
004476A5 F6D0 not al ;解密伪代码
004476A7 0f94c5 sete ch
004476AA FEC8 dec al ;解密伪代码
004476AC D3F9 sar ecx,cl
004476AE 0FB7CB movzx ecx,bx
004476B1 F6D0 not al ;解密伪代码
004476B3 0FBDCE bsr ecx,esi
004476B6 F6D8 neg al ;解密伪代码
004476B8 0BCF or ecx,edi
004476BA 66:3BC6 cmp ax,si
004476BD 32D8 xor bl,al;生成下次解密的密钥
004476BF c1d1 f3 rcl ecx,0xf3
004476C2 86ED xchg ch,ch
004476C4 66:13CE adc cx,si
004476C7 8B4C25 00 mov ecx,dword ptr ss:[ebp];取出堆栈的值（对应push 0）
004476CB A8 8D test al,0x8D
004476CD 81C5 04000000 add ebp,0x4;ebp指向下一个堆栈的值（对应push esi）
004476D3 66:F7C7 5744 test di,0x4457
004476D8 890C04 mov dword ptr ss:[esp+eax],ecx ;存入虚拟寄存器（对应Vm_Relocation）
004476DB 66:0FBAF8 99 btc ax,0x99
004476E0 03C7 add eax,edi
004476E2 8B06 mov eax,dword ptr ds:[esi];从esi读取4字节
004476E4 81C6 04000000 add esi,0x4;伪代码指针加4
004476EA F8 clc
004476EB 33C3 xor eax,ebx ;解密伪代码
004476ED 48 dec eax ;解密伪代码
004476EE F8 clc
004476EF F5 cmc
004476F0 0FC8 bswap eax;解密伪代码
004476F2 66:F7C2 FD44 test dx,0x44FD
004476F7 8D80 9381C4FF lea eax,dword ptr ds:[eax-0x3B7E6D] ;解密伪代码
004476FD 35 412CD300 xor eax,0xD32C41;解密伪代码
00447702 33D8 xor ebx,eax;生成下次解密的密钥
00447704 85DA test edx,ebx
00447706 03F8 add edi,eax;生成handler的地址
00447708 ^ FFE7 jmp edi;等价于push retn组合This is still a “fill virtual register” handler, but it differs structurally from 2.12.3. The handler now has two parts: part one performs the handler’s function (here, mov [esp+eax], ecx — writing the pushed register value into a virtual register, with esp as the virtual-register base and eax carrying the decoded offset); part two decrypts the next 4 bytes and computes the next handler address.
Using vm_<reg> to denote each virtual register, the fill handlers map like this:
| Virtual-register offset | Register |
|---|---|
0x2c |
vm_relocation (always 0) |
0x30 |
vm_esi |
0x24 |
vm_ebp |
0x04 |
vm_ebx |
0x34 |
vm_eax |
0x18 |
vm_edi |
0x00 |
vm_ecx |
0x1c |
vm_edx |
0x3c |
vm_eflags |
0x20 |
vm_retaddr (the return from the call at 0x49689c) |
0x28 |
vm_push (the value pushed at 0x496897) |
The full virtual-register frame is 0x40 bytes — 16 registers of 4 bytes each, same as the 1.2 era, with the spare slots used for computation and confusion:
0019FE88 vm_ecx vm_ebx 空闲 空闲
0019FE98 空闲 空闲 vm_edi vm_edx
0019FEA8 vm_retaddr vm_ebp vm_push vm_relocation
0019FEB8 vm_esi vm_eax 空闲 vm_eflagsDumping virtual registers to the stack
After the virtualized mov eax,0x12345678, the handlers pop virtual registers back onto the stack. The next-handler computation is occasionally obfuscated with a decoy jump:
004840A4 03F8 add edi,eax ;算出下一条handler地址
004840A6 ^ E9 1319FCFF jmp 123_vmp.004459BE;这条代码不是push edi，或者jmp edi。004459BE 8D4424 60 lea eax,dword ptr ss:[esp+0x60]
004459C2 3BE8 cmp ebp,eax
004459C4 ^ 0F87 89AFFFFF ja 123_vmp.00440953 ;混淆，永远都是跳走00440953 ^\FFE7 jmp ediThe cmp ebp,eax; ja is a fake conditional — both operands are constant, so it always jumps. After the dump phase, the stack layout is:
0019FF50 51515151 ;vm_esi
0019FF54 0019FF80 ;vm_ebp
0019FF58 BBBBBBBB ;vm_ebx
0019FF5C 12345678 ;0x12345678
0019FF60 D1D1D1D1 ;vm_edi
0019FF64 CCCCCCCC ;vm_ecx
0019FF68 DDDDDDDD ;vm_edx
0019FF6C 00000246 ;vm_eflags
0019FF70 00415035 ;00415035Note there’s no vm_eax in that table — 0019FF5C is vm_eax, which now holds 0x12345678 because the mov eax,0x12345678 was virtualized. And 0019FF70 (00415035) is the virtualized jmp 00415035.
Exiting the VM
The final handler restores the real registers and performs the virtualized jump:
0047F585 8BE5 mov esp,ebp ;修改esp，配合后面的出栈指令
0047F587 0FBFF6 movsx esi,si
0047F58A 5E pop esi ;恢复esi
0047F58B 9F lahf
0047F58C 0FBFDA movsx ebx,dx
0047F58F F7C5 A23F0045 test ebp,0x45003FA2
0047F595 5D pop ebp ;恢复ebp
0047F596 5B pop ebx ;恢复ebx
0047F597 98 cwde
0047F598 66:0FBAFA C4 btc dx,0xC4
0047F59D 66:98 cbw
0047F59F 58 pop eax ;恢复eax
0047F5A0 c0f9 28 sar cl,0x28
0047F5A3 5F pop edi ;恢复edi
0047F5A4 66:0FB6D0 movzx dx,al
0047F5A8 59 pop ecx ;恢复ecx
0047F5A9 5A pop edx ;恢复edx
0047F5AA F6C3 95 test bl,0x95
0047F5AD 9D popfd ;恢复eflags
0047F5AE C3 retn ;相当于jmp 00415035At 0047F59F the top of the stack is 0x12345678, so pop eax executes the virtualized mov eax,0x12345678; the retn at the end jumps to 00415035, executing the virtualized jmp.
Summary
3.0.0beta is a large file that executes relatively little code. The core changes versus 2.12.3:
- Handler hiding: the vtable-style
[eax*4 + base]lookup is gone.edinow carries the handler address (the virtual-register base moves toesp), and each handler computes its successor asedi = decrypt(pseudocode) + edi. - 4-byte pseudocode reads instead of 1-byte.
- Double encryption of the virtual code: an arithmetic-op layer plus an xor/add/sub keyed by the pseudocode address.
- The virtual-register frame itself stays 0x40 bytes / 16 registers.
It’s still a beta — mostly functionality with comparatively little confusion — so despite the bigger file it’s actually less involved to trace than 2.12.3. But the handler-chaining and double encryption are the foundation the later 3.x builds build their obfuscation on.
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch