VMProtect 1.81 Under the Hood: VM Structure and De-virtualization

About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

Preface

This article walks through the fundamental structure of VMProtect by analyzing a program protected with the VMProtect 1.81 Demo, and attempts to work out what the virtualized code does and how to de-virtualize it.

Virtual machine structure

Virtual-machine (VM) protection converts a program’s executable code into custom intermediate opcodes (one or more bytes each) to protect the original program from reverse engineering and tampering. These opcodes are interpreted and executed by a virtual machine, which reproduces the original behavior. To reverse a protected program you then have to reverse the whole VM structure, understand what it does, and additionally analyze the opcodes — making the whole reverse-engineering effort very tedious. A general VM structure looks like this:

General VM structure

The VM’s interpretation loop works as follows:

Push the real execution context onto the stack and initialize the environment the VM needs (for example, initialize VIP and VSP).
Fetch the next virtual instruction’s opcode via VIP, and jump to the corresponding VM handler.
Execute the VM handler and update the virtual instruction pointer.
Check the virtual stack to make sure there is enough space.
Repeat steps 2–4 until all virtual bytecode has executed.
Restore the real context and exit the VM.

The VM-related data structures:

VIP — virtual instruction pointer, the analog of x86-64’s RIP. Points to the address of the next virtual instruction to execute. Usually kept in the RSI register.
VSP — virtual stack pointer, the analog of x86-64’s RSP. Points to the top of the virtual stack. Usually kept in RBP.
VM Stack — the virtual stack. A stack-based VM performs every instruction through the virtual stack; an add, for example, requires pushing both operands first and then operating on the stack.
VM Registers / VM Context — the virtual register file, or the VM’s context. Usually pointed to by RDI.
VM Instruction — a virtual instruction, aka virtual bytecode: the bytes the VM interprets and executes. Each is one or more bytes.
VM Opcode — the opcode of a virtual instruction; the first byte of each virtual instruction.
IMM — immediates within a virtual instruction. They may be constants, virtual-register indices, etc.
VM Handler — a snippet of real code that implements one virtual operation (one ADD opcode maps to one handler). The handler for an opcode is chosen by that opcode and interprets its specific semantics.
VM Handler Table — a table of offsets to all VM handlers.
VM Opcode Table — a table of virtual-instruction bytecode.

Case study

The sample in this case comes from a beginner tutorial on analyzing the VMProtect 1.81 Demo. The sample’s assembly is just adding and subtracting the same constant:

.text:00401000 sub_401000      proc near               ; CODE XREF: start↓p
.text:00401000                 mov     eax, dword_403000; value is 0deadbeefh
.text:00401005                 add     eax, 12345678h
.text:0040100A                 sub     eax, 12345678h
.text:0040100F                 mov     dword_403000, eax
.text:00401014                 retn
.text:00401014 sub_401000      endp

After VMProtect protection it becomes:

.text:00401000 ; void sub_401000()
.text:00401000 sub_401000      proc near               ; CODE XREF: start↓p
.text:00401000                 push    offset byte_404781
.text:00401005                 call    sub_40472C
.text:0040100A
.text:0040100A loc_40100A:                             ; CODE XREF: sub_40472C+2B↓j
.text:0040100A                                         ; DATA XREF: .vmp0:jpt_404757↓o
.text:0040100A                 mov     esi, [ebp+0]    ; jumptable 00404757 cases 126,128,146,149,246,255
.text:0040100D                 add     ebp, 4
.text:00401010                 jmp     loc_40474E
.text:00401010 sub_401000      end

There’s a classic VMProtect entry signature here: push + call. The push pushes the address of the VM opcode table, then call sub_40472C enters the VM entry. byte_404781:

.vmp0:00404781 byte_404781     db 0Eh, 0E8h, 81h       ; DATA XREF: sub_401000↑o
.vmp0:00404784                 dd 5C765DA9h, 3E0A1A1Eh, 3A36262Eh, 602122Ah, 3000E822h
.vmp0:00404798                 dd 16790040h, 34567869h, 1E7A1412h, 5678E836h, 97341234h
.vmp0:004047AC                 dd 5C066B4Dh, 1121973Eh, 4C3C0622h, 0EB161121h, 1E11F7EAh
.vmp0:004047C0                 dd 326B2020h, 78081577h, 36325C32h, 30006904h, 0ED0040h
.vmp0:004047D4                 dd 4382010h, 8342C24h, 0E1001Ch, 8 dup(0)
.vmp0:00404800                 dd 200h dup(?)

VM Entry analysis

.vmp0:0040472C                 push    esi
.vmp0:0040472D                 push    edi
.vmp0:0040472E                 push    esp
.vmp0:0040472F                 push    ebx
.vmp0:00404730                 push    eax
.vmp0:00404731                 push    edx
.vmp0:00404732                 push    ebp
.vmp0:00404733                 pushf
.vmp0:00404734                 push    ecx

The first classic signature: when VMProtect builds the VM environment, it pushes the real register values onto the stack to save them.

.vmp0:00404735                 push    ds:dword_404649
.vmp0:0040473B                 push    0

These two instructions push the data to be operated on. This is a trait of stack-based VMs: operations happen on the stack. So this is the virtual stack.

.vmp0:00404740                 mov     esi, [esp+2Ch+arg_0] ;arg_0 = dword ptr  4

Initialize esi. Drawing a stack diagram shows that esi becomes byte_404781, the base of the VM opcode table.

.vmp0:00404744                 mov     ebp, esp
.vmp0:00404746                 sub     esp, 0C0h
.vmp0:0040474C                 mov     edi, esp

Initialize ebp to esp (the virtual stack top), then create a new stack frame used as the virtual registers; edi points to the virtual-register base.

.vmp0:0040474E loc_40474E:
.vmp0:0040474E                 add     esi, [ebp+0]        ;index of vm opcode

Next it pops the virtual-stack top — the offset of the virtual opcode — and adds the opcode-table base, so esi now points to the first virtual opcode to execute.

VM Dispatcher analysis

mov     al, [esi]        ;get vm opcode
movzx   eax, al
inc     esi             ;index + 1
jmp     ds:jpt_404757[eax*4] ; jmp to vm handler

Jump to the VM handler for the opcode. jpt_404757 is a function-address table holding the offsets of the VM handlers.

VM handler table

VM Handler analysis

ADD — addition

ADDB — add two bytes

loc_40476F:
mov     al, [ebp+0]
sub     ebp, 2
add     [ebp+4], al
pushf
pop     dword ptr [ebp+0]
jmp     loc_40400F

Adds the two bytes at the top of the virtual stack, stores the result into the latter’s slot, pushes the flags onto the real stack, then stores the top real-stack element (the flags) into the virtual-stack top.

The addition uses [src ebp+2] and [src ebp], not [src ebp+1] and [src ebp] — probably because the virtual stack is 2-byte aligned.

ADDW — add two words

loc_4045FC:
mov     ax, [ebp+0]
sub     ebp, 2
add     [ebp+4], ax
pushf
pop     dword ptr [ebp+0]
jmp     loc_40400F

Adds the two 2-byte values at the virtual-stack top. The virtual stack then rises two bytes; the first four bytes at the new top are the flags produced by the operation, and the next two bytes are the result value.

ADDDW — add two dwords

loc_404000:
mov     eax, [ebp+0]
add     [ebp+4], eax
pushf
pop     dword ptr [ebp+0]
jmp     loc_404751

Adds the two dwords at the virtual-stack top. The virtual-stack top stays in place.

SHR — logical shift right

SHRB — logical shift right a byte

loc_404680:
mov     al, [ebp+0]
mov     cl, [ebp+2]
sub     ebp, 2
shr     al, cl
mov     [ebp+4], ax
pushf
pop     dword ptr [ebp+0]
jmp     loc_40400F

Takes two bytes from the virtual-stack top and shifts right. The result is zero-extended to a word (the result goes into al, but ax is pushed). The virtual stack rises two bytes.

SHRW — logical shift right a word

loc_40454E:
mov     ax, [ebp+0]
mov     cl, [ebp+2]
sub     ebp, 2
shr     ax, cl
mov     [ebp+4], ax
pushf
pop     dword ptr [ebp+0]
jmp     loc_40400F

Takes a word and a byte (the shift count) from the virtual stack and shifts right. The virtual stack rises two bytes.

SHRDW — logical shift right a dword

loc_404077:
mov     eax, [ebp+0]
mov     cl, [ebp+4]
sub     ebp, 2
shr     eax, cl
mov     [ebp+4], eax
pushf
pop     dword ptr [ebp+0]
jmp     loc_40400F

Takes a dword and a byte (the shift count) from the virtual-stack top and shifts right. The virtual stack rises two bytes.

SHRD — double-precision logical shift right

SHRDDW — double-precision shift right a dword

loc_404610:
mov     eax, [ebp+0]
mov     edx, [ebp+4]
mov     cl, [ebp+8]
add     ebp, 2
shrd    eax, edx, cl
mov     [ebp+4], eax
pushf
pop     dword ptr [ebp+0]
jmp     loc_404751

Takes two dwords and a byte from the virtual-stack top, performs a double-precision shift right, and stores only the high part eax. The virtual stack shrinks two bytes.

SHL — logical shift left

SHLB — logical shift left a byte

loc_4045C0:
mov     al, [ebp+0]
mov     cl, [ebp+2]
sub     ebp, 2
shl     al, cl
mov     [ebp+4], ax
pushf
pop     dword ptr [ebp+0]
jmp     loc_40400F

Takes two bytes from the virtual stack and shifts left, zero-extending the result to a word (result in al, but ax is pushed). The virtual stack rises two bytes.

SHLW — logical shift left a word

loc_404698:
mov     ax, [ebp+0]
mov     cl, [ebp+2]
sub     ebp, 2
shl     ax, cl
mov     [ebp+4], ax
pushf
pop     dword ptr [ebp+0]
jmp     loc_40400F

Takes a word and a byte (shift count) from the virtual-stack top and shifts left. The virtual stack rises two bytes.

SHLDW — logical shift left a dword

loc_4046EF:
mov     eax, [ebp+0]
mov     cl, [ebp+4]
sub     ebp, 2
shl     eax, cl
mov     [ebp+4], eax
pushf
pop     dword ptr [ebp+0]
jmp     loc_40400F

Takes a dword and a byte (shift count) from the virtual-stack top and shifts left. The virtual stack rises two bytes.

SHLD — double-precision logical shift left

SHLDDW — double-precision shift left a dword

loc_4046BF:
mov     eax, [ebp+0]
mov     edx, [ebp+4]
mov     cl, [ebp+8]
add     ebp, 2
shld    eax, edx, cl
mov     [ebp+4], eax
pushf
pop     dword ptr [ebp+0]
jmp     loc_404751

Takes two dwords and a byte from the virtual-stack top and performs a double-precision shift left. The virtual stack shrinks two bytes.

NAnd — NOT-then-AND

NAndB — NOT-then-AND two bytes

loc_404591:
mov     ax, [ebp+0]
mov     dx, [ebp+2]
not     al
not     dl
sub     ebp, 2
and     al, dl
mov     [ebp+4], ax
pushf
pop     dword ptr [ebp+0]
jmp     loc_40400F

Takes two words from the virtual stack; actually the low byte of each is NOTed, then ANDed. The virtual stack rises two bytes.

NAndW — NOT-then-AND two words

loc_404041:
not     dword ptr [ebp+0]
mov     ax, [ebp+0]
sub     ebp, 2
and     [ebp+4], ax
pushf
pop     dword ptr [ebp+0]
jmp     loc_40400F

NOTs the dword at the virtual-stack top, then splits it into two words and ANDs them. The virtual stack rises two bytes.

NAndDW — NOT-then-AND two dwords

loc_40451A:
mov     eax, [ebp+0]
mov     edx, [ebp+4]
not     eax
not     edx
and     eax, edx
mov     [ebp+4], eax
pushf
pop     dword ptr [ebp+0]
jmp     loc_404751

Takes two dwords from the virtual-stack top, NOTs each, then ANDs them. The virtual-stack top stays in place.

PushConst — push an immediate from the virtual instruction

PushConstB — push a one-byte immediate

loc_4044E4:
movzx   eax, byte ptr [esi]
sub     ebp, 2
mov     [ebp+0], ax
add     esi, 1
jmp     loc_40400F

Takes the operand (one byte) from the virtual instruction, zero-extends it to a word, and pushes it. The virtual stack rises two bytes.

PushConstW — push a two-byte immediate

loc_40464D:
mov     ax, [esi]
lea     esi, [esi+2]
sub     ebp, 2
mov     [ebp+0], ax
jmp     loc_40400F

Takes the operand (two bytes) from the virtual instruction and pushes it. The virtual stack rises two bytes.

PushConstDW — push a four-byte immediate

loc_40462B:
mov     eax, [esi]
sub     ebp, 4
lea     esi, [esi+4]
mov     [ebp+0], eax
jmp     loc_40400F

Takes the operand (four bytes) from the virtual instruction and pushes it. The virtual stack rises four bytes.

PushConstCBDE — sign-extend a one-byte immediate to dword and push

loc_40453B:
mov     al, [esi]
cbw
cwde
sub     ebp, 4
lea     esi, [esi+1]
mov     [ebp+0], eax
jmp     loc_40400F

Takes the operand (one byte) from the virtual instruction, sign-extends it to a dword, and pushes it. The virtual stack rises four bytes.

PushConstCWDE — sign-extend a two-byte immediate to dword and push

loc_4044BE:
mov     ax, [esi]
cwde
add     esi, 2
sub     ebp, 4
mov     [ebp+0], eax
jmp     loc_40400F

Takes the operand (two bytes) from the virtual instruction, sign-extends it to a dword, and pushes it. The virtual stack rises four bytes.

PushReg — push the value of a given virtual register

PushRegB — push the byte value of a virtual register

loc_4044D0:
mov     al, [esi]
mov     al, [edi+eax]
sub     ebp, 2
sub     esi, 0FFFFFFFFh
mov     [ebp+0], ax
jmp     loc_40400F

Takes the operand (one byte) from the virtual instruction as a virtual-register index, reads the corresponding virtual register’s byte, and pushes it. The virtual stack rises two bytes.

sub esi, 0FFFFFFFFh equals add esi, 1.

PushRegW — push the word value of a virtual register

loc_4045E9:
mov     al, [esi]
inc     esi
mov     ax, [edi+eax]
sub     ebp, 2
mov     [ebp+0], ax
jmp     loc_40400F

Takes the operand (one byte) from the virtual instruction as a virtual-register index, reads the corresponding virtual register’s word, and pushes it. The virtual stack rises two bytes.

PushRegDW — push the dword value of a virtual register

loc_4045AF:
and     al, 3Ch
mov     edx, [edi+eax]
sub     ebp, 4
mov     [ebp+0], edx
jmp     loc_40400F

ANDs the virtual instruction’s opcode byte with 0x3C to form the virtual-register index, reads the corresponding dword, and pushes it. The virtual stack rises four bytes.

PopReg — pop the top of the stack into a virtual register

PopRegB — pop a byte into a virtual register

loc_404568:
mov     al, [esi]
mov     dx, [ebp+0]
sub     esi, 0FFFFFFFFh
add     ebp, 2
mov     [edi+eax], dl
jmp     loc_404751

Takes a one-byte operand from the virtual instruction as a virtual-register index, takes the two-byte value at the virtual-stack top, and stores its low byte into the register. The virtual stack shrinks two bytes.

PopRegW — pop a word into a virtual register

loc_4046DA:
mov     al, [esi]
add     esi, 1
mov     dx, [ebp+0]
add     ebp, 2
mov     [edi+eax], dx
jmp     loc_404751

Takes a one-byte operand from the virtual instruction as a virtual-register index, then takes the two-byte value at the virtual-stack top and stores it into the register. The virtual stack shrinks two bytes.

PopRegDW — pop a dword into a virtual register

loc_404058:
and     al, 3Ch
mov     edx, [ebp+0]
add     ebp, 4
mov     [edi+eax], edx
jmp     loc_404751

ANDs the virtual instruction’s opcode byte with 0x3C, takes the dword at the virtual-stack top, and stores it into the virtual register. The virtual stack shrinks four bytes.

PushVSP — push the virtual stack pointer

PushVSPW — push the low word of the virtual stack pointer

loc_40463B:
mov     eax, ebp
sub     ebp, 2
mov     [ebp+0], ax
jmp     loc_40400F

Pushes the virtual-stack pointer (low two bytes). The virtual stack rises two bytes.

PushVSPDW — push the virtual stack pointer (dword)

loc_4046B2:
mov     eax, ebp
sub     ebp, 4
mov     [ebp+0], eax
jmp     loc_40400F

Pushes the virtual-stack pointer (dword). The virtual stack rises four bytes.

PopVSP — load the virtual stack pointer from the top of the stack

PopVSPW — load a word into the virtual stack pointer

loc_404532:
mov     bp, [ebp+0]
jmp     loc_40400F

Loads the two-byte value at the virtual-stack top into the virtual stack pointer. The resulting stack top is unknown.

PopVSPDW — load a dword into the virtual stack pointer

loc_40457C:
mov     ebp, [ebp+0]
jmp     loc_40400F

Loads the dword at the virtual-stack top into the virtual stack pointer. The resulting stack top is unknown.

Store — write to memory

StoreB — write a byte to memory

loc_4044AE:
mov     eax, [ebp+0]
mov     dl, [ebp+4]
add     ebp, 6
mov     [eax], dl
jmp     loc_404751

Takes a dword (the address) and a byte (the value) from the virtual stack, and stores the byte at that address. The virtual stack shrinks six bytes.