About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

When the VMProtect 3.5.1 source code leaked and its last critical files were made public, the obvious prediction was that VMProtect would be laid bare to every reverse engineer. What actually happened was the opposite: VMProtect came back stronger. This article breaks down the 3.8-series overhaul — a rewrite so large it effectively amounts to a new VMProtect — and walks through the seven major obfuscation strategies it introduced, with concrete disassembly for each.

Background: the 3.8 rewrite

The 3.5.1 source leak became fully public (including compiled-build tutorials on various forums) around December 2023. But by then VMProtect had already moved on. The official changelog (https://vmpsoft.com/news/) shows the 3.8 release landing in early 2023:

VMProtect changelog (1)

VMProtect changelog (2)

The options added in 3.8 are not minor tweaks — each one is a serious anti-analysis measure. The timing (3.8 landing right around the leak) is worth discussing: it looks like a direct response. The update is not incremental; it’s close to a full rewrite, adding a large number of aggressive obfuscation strategies and changing huge swaths of the engine. Here they are, one by one.

Strategy 1: full instruction mutation

Nearly every instruction that has an alternative form has been mutated. The only instruction left untouched is the XOR-decryption instruction itself; everything else has one or more variants.

Pseudo-code (bytecode) decryption:

00422864 | xor al,byte ptr ss:[ebp+edx-0x7]    // the pseudo-code read instruction has four variants
00422868 | xor al,bl         // cannot be mutated, kept as-is
0042286A | jmp vmp.42BE65
0042BE65 | add dword ptr ss:[esp+edx*2+0x2],0xC8169CA3
0042BE6D | inc al
0042BE6F | xchg word ptr ss:[esp+edx+0x1],dx
0042BE74 | mov dword ptr ss:[esp+edx*8-0x729D0],edx
0042BE7B | shl dword ptr ss:[esp+edx-0xE53B],0x57
0042BE83 | xor al,0xA6
0042BE85 | lea edx,dword ptr ds:[edx*2+0x2FBCE804]
0042BE8C | inc dl
0042BE8E | sete dl
0042BE91 | dec al
0042BE93 | jmp vmp.5088D5
005088D5 | call vmp.5074A0
005074A0 | neg al
005074A2 | pop edx
005074A3 | add edx,0xFFF7AC3A
005074A9 | jmp edx
00483514 | mov edx,0x119DB08A
00483519 | xchg byte ptr ss:[esp+edx*4-0x4676C225],dl
00483520 | add al,0x9E
00483522 | lea edx,dword ptr ds:[edx+edx*4+0x11B81E34]
00483529 | xadd word ptr ss:[esp+edx-0x69CC901C],dx
00483532 | xor bl,al          // cannot be mutated, kept as-is

The five possible forms of the pseudo-code read instruction:

mov eax,dword ptr ss:[esi] // original

xor eax,eax                // new leading zero-clear (can be another clear instruction)
add eax,dword ptr ss:[esi] // variant 1
xor eax,dword ptr ss:[esi] // variant 2
or  eax,dword ptr ss:[esi] // variant 3
adc eax,dword ptr ss:[esi] // variant 4

Computing a handler address:

004C9172 | mov eax,dword ptr ss:[ebp-0x4]     // read pseudo-code
004C9176 | jno vmp.46E996
0046E996 | mov edx,0x581983A
0046E99B | xor eax,ebx
0046E99D | ror eax,0x1
0046E99F | add edx,edx
0046E9A1 | bswap eax
0046E9A3 | push edx
0046E9A4 | shl byte ptr ss:[esp+edx*8-0x5819839E],0x84
0046E9AC | inc eax
0046E9AD | jmp vmp.490ABF
00490ABF | or word ptr ss:[esp+edx*8-0x581983A0],dx
00490AC7 | movsx ecx,dx
00490ACA | rol eax,0x2
00490ACD | shl dword ptr ss:[esp+edx-0xB033074],cl
00490AD4 | xor ebx,eax                           // decrypted addend complete
00490AD6 | or cl,byte ptr ss:[esp+edx-0xB033071] // added leading CF-clear; after this, CF = 0
00490ADD | adc edi,eax      // variant of add

Strategy 2: memory-address constant encryption

For any instruction that has a memory operand, the operand almost always involves a memory-address constant:

constant-only: mov eax, dword ptr ss:[0x123]
base register only: mov dword ptr ss:[esp], 0x123
base + index + constant: mov dword ptr ss:[esp+edi+0x123], 0x123

If you count the constant 0 as a constant, then every memory-operand instruction contains an address constant — and that constant can be encrypted.

// here the pseudo-code register is ebp and ecx is 0x07400000,
// so the memory-address constant is actually -5;
// -5 plus ebp is the current pseudo-code pointer
00433C60 | movzx eax,byte ptr ds:[ecx+ebp-0x7400005]
00433C68 | ror ecx,0x77
00433C6B | not cx
00433C6E | and cl,cl
00433C70 | xor al,bl

Someone might ask what the point is beyond making the code uglier. The answer: every one of these changes is aimed at analysis-plugin authors. The author’s team clearly has a department that builds analysis plugins, and each update collects and patches against every analysis technique in circulation. The most visible consequence: the pre-3.8 approach of matching handlers by instruction-text signatures is now completely dead.

Strategy 3: handler mutation

This mutation only exists in handlers that compute a virtual-register address.

PushVR32 handler:

004BF2B6 | movzx eax,byte ptr ss:[ebp-0xB]           // read pseudo-code
004BF2BB | xor al,bl
004BF2BD | mov ecx,dword ptr ss:[esp+0xC]
004BF2C1 | add ecx,0xFFFEE8B1
004BF2C7 | jmp ecx
0044D3C2 | neg al
0044D3C4 | inc al
0044D3C6 | neg al
0044D3C8 | mov edx,dword ptr ss:[esp+0x2D]
0044D3CC | not edx
0044D3CE | inc al
0044D3D0 | movzx ecx,dx
0044D3D3 | xor bl,al                       // decrypt the offset
0044D3D5 | add ecx,0xD3A68C98
0044D3DB | lea eax,dword ptr ss:[esp+eax+0x3C]    // compute the virtual-register address
0044D3DF | or edx,edx
0044D3E1 | xchg dword ptr ss:[esp+0x1C],edx
0044D3E5 | adc edx,0xFFFD3506
0044D3EB | jmp edx
0042075C | mov edx,dword ptr ds:[eax]        // load the virtual-register value
0042075E | mov dword ptr ds:[esi-0x1C],edx   // write the value into the virtual stack

Mutated form:

00420766 | lea ecx,dword ptr ss:[esp+ecx-0x2C596EE3]  // compute the virtual-register address
0042076D | mov eax,dword ptr ds:[ecx]         // load the virtual-register value
0042076F | mov dword ptr ds:[esi-0x20],eax    // write the value into the virtual stack

Strategy 4: handler level-1 fusion

Normally, when one handler finishes it jumps to the next handler to execute, and each handler contains a block that computes the next handler’s address — a block that reads pseudo-code. After fusion, the pseudo-code that computed the next handler address is discarded, and the next-handler-address block is gone too. Two handlers are joined by a jump (or directly connected) into a single handler, and the process repeats — like a human centipede.

// first handler: a PushVR32 variant
00479089 | mov dword ptr ss:[esp+edx*2-0x2],edx
0047908D | sub dword ptr ss:[esp+edx*2+0x16],0x8B38041C
00479095 | movzx eax,byte ptr ss:[esp+edx+0xA]
0047909A | lea ecx,dword ptr ss:[esp+edx+0x55]     // compute the virtual-register address
0047909E | mov eax,dword ptr ds:[ecx+edx-0x3]      // load the virtual-register value
004790A2 | shl dx,0x68
004790A6 | mov dword ptr ds:[edi+edx*8-0x1808],eax    // write the value into the virtual stack
// second handler: a PushVR32 variant
004790AD | mov eax,edx
004790AF | xadd byte ptr ss:[esp+edx-0x2F9],dl
004790B7 | call vmp.4B01CD
004B01CD | lea ecx,dword ptr ss:[esp+edx*4-0xB80]    // compute the virtual-register address
004B01D4 | jo vmp.520FD5
004B01DA | mov eax,dword ptr ds:[ecx+edx*8-0x1800]  // load the virtual-register value
004B01E1 | lea ecx,dword ptr ds:[edx+edx*2+0x6BAEF303]
004B01E8 | mov dword ptr ds:[edi+edx*4-0xC0C],eax     // write the value into the virtual stack
// third handler: a PushVR32 variant
004B01EF | lea ecx,dword ptr ds:[edx+edx-0x50DA8F6D]
004B01F6 | or dl,byte ptr ss:[esp+edx-0x2FB]
004B01FD | neg edx
004B01FF | lea eax,dword ptr ss:[esp+0x44]     // compute the virtual-register address
004B0203 | ror dword ptr ss:[esp+0xF],cl
004B0207 | pop ecx
004B0208 | add ecx,0xFFFDC76A
004B020E | jmp ecx
00455826 | pop edx
00455827 | mov edx,dword ptr ds:[eax]     // load the virtual-register value
00455829 | call vmp.44DAE9
0044DAE9 | mov dword ptr ds:[edi-0x10],edx      // write the value into the virtual stack

Strategy 5: handler level-2 fusion

Building on level-1 fusion is level-2 fusion, seen in handler combinations that emulate an arithmetic operation. Take a simulated addition: the first handler reads the addend constant from pseudo-code, the second reads the augend, the third performs the addition. Previously, the first two handlers pushed the addend and augend onto the virtual stack, and the third popped them and executed ADD. Here, the push-to-virtual-stack instructions are discarded and the values are bound directly to registers — the first handler leaves its value in eax, the second in ecx, and the third just executes add eax, ecx. The result stays in eax and can feed directly into a fourth handler without touching the virtual stack.

// PushVR8 variant
0043B239 | lea edx,dword ptr ss:[esp+0x50]     // compute the virtual-register address
0043B23D | mov ecx,0x489D36B2
0043B242 | sar ecx,0x45
0043B245 | ror cl,0x67
0043B248 | movzx cx,byte ptr ds:[ecx+edx-0x244E96B] // load the virtual-register value into cx
0043B251 | jb vmp.48BAC5
0043B257 | mov eax,0xFF9BE60E
0043B25C | mov dx,cx     // move cx into dx (itself a notable obfuscation point)
0043B25F | shl eax,0xB1  // up to here the loaded value has NOT been stored into the virtual stack
// PopVR8
0043B262 | movzx ecx,byte ptr ss:[ebp-0x6]       // read pseudo-code
0043B267 | pop eax
0043B268 | add eax,0xD15B
0043B26D | jmp eax
00499668 | xor cl,bl
0049966A | mov eax,0x822BE31F
0049966F | shl al,0xC6
00499672 | sub ax,0x923B
00499676 | xor cl,0x3E
00499679 | not cl
0049967B | jmp vmp.4570F0
004570F0 | push eax
004570F1 | add cl,0xA3
004570F4 | xor ax,ax
004570F7 | ror cl,0x1
004570F9 | shl eax,0x4D
004570FC | sar eax,0xB7
004570FF | xor bl,cl       // decrypt the offset
00457101 | neg al
00457103 | sub ax,0xEC15
00457107 | sbb eax,eax
00457109 | lea ecx,dword ptr ss:[esp+ecx+0x4]    // compute the virtual-register address
0045710D | ror al,0xC1
00457110 | mov byte ptr ds:[ecx],dl     // write dl into the virtual register

Strategy 6: register release

The three rotating registers ebp, esi, edi each have their role. Near the end of a flow block — right before the VM-exit handler — when a handler will no longer read pseudo-code or compute the next handler address, its corresponding registers are released: the pseudo-code register or dispatcher register is overwritten with an arbitrary value, or reused as the encrypted memory-address-constant register. Superficially it looks meaningless, but for a plugin author it forces major logic rewrites.

// jump to the next handler
0042E08C | jmp ebp
// PushVR32
004DF21B | jmp vmp.4E7299
004E7299 | mov ecx,0xEC3505BA
004E729E | movzx eax,byte ptr ds:[esi-0x1]   // read pseudo-code
004E72A2 | movsx ebp,cl                      // dispatcher register ebp released
004E72A5 | xor al,bl
004E72A7 | ror al,0x1
004E72A9 | jmp vmp.4F5E38
004F5E38 | movzx edx,cl
004F5E3B | add al,0x98
004F5E3D | movsx esi,dl                     // dispatcher register released (again)
004F5E40 | push edx
004F5E41 | not word ptr ss:[esp+edx*4-0x2E8]
004F5E49 | not al
004F5E4B | rol al,0x1
004F5E4D | dec dl
004F5E4F | and ecx,dword ptr ss:[esp+edx*8-0x5C8]
004F5E56 | jge vmp.4AD5DC
004AD5DC | dec al
004AD5DE | xor bl,al    // pseudo-code decryption complete
004AD5E0 | not cx
004AD5E3 | rol word ptr ss:[esp+ecx-0xFAFE],0xA7
004AD5EC | lea eax,dword ptr ss:[esp+eax+0x4]  // compute the virtual register
004AD5F0 | shl dx,0xEE
004AD5F4 | sbb ecx,0x39B0CEB2
004AD5FA | mov edx,dword ptr ds:[eax+edx*4-0x10000] // load the virtual-register value
004AD601 | mov dword ptr ds:[edi-0x4],edx // store into the virtual stack
// PushVR32 variant
004AD604 | lea edx,dword ptr ss:[esp+0x4C]  // compute the virtual register
004AD608 | mov ecx,dword ptr ds:[edx]      // load the virtual-register value
004AD60A | sub esi,ebp
004AD60C | movsx eax,byte ptr ss:[esp+esi*8]
004AD610 | push eax
004AD611 | mov dword ptr ds:[edi+esi-0x8],ecx // store into the virtual stack
// VMExit - leaving the virtual machine
004AD615 | mov edx,ebp
004AD617 | lea esp,dword ptr ss:[esp+0x8]
004AD61B | lea esp,dword ptr ds:[edi+esi-0x8] // virtual-stack pointer restored to esp
004AD61F | lea ebx,dword ptr ds:[eax-0x3B72475A]
004AD625 | pop ebp        // restore the real ebp
004AD626 | shr bl,0x41
004AD629 | pop ecx        // restore the real ecx
004AD62A | setnp bl
004AD62D | shl dl,0x47
004AD630 | not si
004AD633 | popfd     // restore the real eflags
004AD634 | bswap esi
004AD636 | pop eax  // restore the real eax
004AD637 | pop edi  // restore the real edi
004AD638 | pop esi  // restore the real esi
004AD639 | pop edx  // restore the real edx
004AD63A | pop ebx  // restore the real ebx
004AD63B | call vmp.495550
00495550 | lea esp,dword ptr ss:[esp+0x4]
00495554 | ret    // leave the VM and return to real instructions

Strategy 7: swapping the rotating register and the decryption register

I found this while analyzing the universal-gate handler — one of the more sensitive handlers in the VM, and the key handler that JCC-patching attacks usually target, so the author hardened it heavily.

The trick: the virtual-stack register ebp’s value is copied into eax, making eax an alias of ebp. Now determining whether a given instruction reads virtual-stack memory becomes much harder. The pseudo-code register may also be swapped this way, though I haven’t found that yet.

// PushVStack - the handler that stores the virtual-stack register into the virtual stack
004F61DF | mov ecx,esi   // the current virtual-stack register esi is copied into ecx
004F61E1 | ror dl,0xC5
004F61E4 | jae vmp.556582
004F61EA | mov dword ptr ds:[esi+eax*4-0x8AA4860],ecx  // store ecx into the virtual stack
// Dispatcher - compute the next handler address
004F61F1 | or al,ah
004F61F3 | mov eax,dword ptr ss:[ebp+eax*2-0x4552532]  // read pseudo-code
004F61FA | push edx
004F61FB | lea edx,dword ptr ds:[edx+edx+0x4A284788]
004F6202 | lea ebp,dword ptr ss:[ebp+edx-0x4A2848FC]
004F6209 | movsx ecx,dl
004F620C | jne vmp.431ABA
00431ABA | xor eax,ebx
00431ABC | sub eax,0x91B19C05
00431AC1 | call vmp.4A467A
004A467A | add dx,cx
004A467D | not eax
004A467F | mov dword ptr ss:[esp+edx-0x4A2848F0],ecx
004A4686 | xor eax,0x4122C6BE
004A468B | xor cl,0x27
004A468E | sub edx,0xF40FEF8C
004A4694 | dec eax
004A4695 | call vmp.43DAFA
0043DAFA | shl ecx,0x96
0043DAFD | or edx,edx
0043DAFF | xor ebx,eax         // pseudo-code decryption complete
0043DB01 | add edi,eax         // edi is the dispatcher register; compute the next handler address
// Nor32 - a universal-gate handler
0043DB03 | push ecx
0043DB04 | sub cx,word ptr ss:[esp+edx-0x56185964]
0043DB0C | inc dx
0043DB0F | mov eax,dword ptr ds:[esi+edx-0x56185969] // pop the previously pushed virtual-stack pointer from the virtual stack
0043DB16 | pop ecx
0043DB17 | mov edx,dword ptr ss:[eax+edx-0x56185965] // equivalent to reading argument 1 from the virtual stack
0043DB1F | jo vmp.449C0D
0043DB25 | inc cl
0043DB27 | mov eax,edx                             // move argument 1 from eax to edx
0043DB29 | pop ecx
0043DB2A | adc ecx,0x2C648
0043DB30 | jmp ecx
004D0CE2 | jmp vmp.4F19A1
004F19A1 | mov edx,dword ptr ds:[esi]            // read argument 2 from the virtual stack
004F19A4 | mov ecx,0xD886303E
004F19A9 | not eax                                // negate argument 1
004F19AB | dec cx
004F19AE | ror cl,0x27
004F19B1 | not edx                               // negate argument 2
004F19B3 | shl byte ptr ss:[esp+0x2],0x42
004F19B8 | or ecx,ecx                            // perform the OR
004F19BA | and eax,edx
004F19BC | mov dword ptr ds:[esi],eax            // store the result into the virtual stack

Summary

VMProtect 3.8 is not a small revision — it’s a defensive rewrite that specifically targets the analysis-plugin techniques that worked against 3.5.1 and earlier. The seven strategies above — full instruction mutation, encrypted memory-address constants, handler mutation, two levels of handler fusion, register release, and rotating/decryption-register swapping — are each individually annoying, and together they make text-signature-based handler recognition and naive plugin logic largely obsolete. Understanding why each mutation exists (defeating a specific analysis assumption) is the key to writing tooling that survives it.

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch