Manually De-virtualizing a VMP 3.9.4 cmp/jne Handler Sequence
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
After months of studying VMProtect without ever doing a full de-virtualization by hand, I finally sat down and did one. It took a full day, but it was worth it.
Naming conventions
I hadn’t settled on a convention yet — whether virtual-register names should carry q/d/w/b suffixes to indicate width, whether operation widths should be in bytes (1/2/4/8) or bits (8/16/32/64), whether the register that points at the bytecode should be vpc or vip. In the end it doesn’t matter much; pick what feels natural and stay consistent.
Here I name the bytecode pointer vpc, the virtual-stack pointer vsp, the current-handler base vbase, and the rolling key vkey. The register used to address virtual registers is rsp — no rename needed there, but if I had to, I’d call it vreg.
The code being protected
r10d is our input. If it’s 3735928559 (0xdeadbeef), rax becomes 0xdeadbeef.
Protection options
One observation from experience: at 10% complexity a composite handler contains at most two simple handlers; at 20%, at most three. My sample size isn’t huge, so I won’t swear by the 10% figure, but in the final analysis a 10%-complexity composite handler never exceeded two simple handlers.
At 100% complexity it goes up to ten. Here’s a handler sequence from another 100%-complexity sample I analyzed:
[0] vPopReg64 VR22; vPopReg64 VR20; vPopReg64 VR5; vPopReg64 VR18; vPopReg64 VR24; vPopReg64 VR1; vPopReg64 VR14
[1] vPopReg64 VR12; vPopReg64 VR0; vPopReg64 VR21; vPopReg64 VR4; vPopReg64 VR6; vPopReg64 VR19; vPopReg64 VR17; vPopReg64 VR15; vPopReg64 VR2; vPopReg64 VR13
[2] vPushImm64 0x140001188; vPushReg64 VR4; vAdd64; vPopReg64 VR25; vPushVsp64
[3] vPopReg64 VR26; vPushReg64 VR15; vPopReg64 VR27; vPushReg64 VR6; vPopReg64 VR28; vPushReg64 VR24; vPopReg64 VR29; vPushReg64 VR21; vPopReg64 VR30; vPushImm64 0x140000000
[4] vPushReg64 VR4; vAdd64; vReadMemDs64; vPopReg64 VR16
[5] vPushImm64 0x1400011c0 ...Collecting traces
After packing, trace the VM’d code with x64dbg. The correct input produces success.trace64; the wrong one produces failed.trace64.
Keeping the meaningful instructions
Analyzing handlers directly from the raw trace is dizzying — there’s too much obfuscation. We need to keep only what’s meaningful.
First, split the trace into handlers:
handlers: list[list[Supertrace.InstructionRecord]] = []
handler: list[Supertrace.InstructionRecord] = []
for i, ins in enumerate(record):
if (i + 1 >= len(record)): nextIns = None
else: nextIns = record[i + 1]
ttins = triton.Instruction()
ttins.setAddress(ins.ins_address)
ttins.setOpcode(ins.bytes)
ctx.disassembly(ttins)
ins.ttins = ttins
for memAcc in ins.mem_accs:
if (memAcc.type == Supertrace.AccessType.READ and (vmpsec_begin <= memAcc.acc_address <= vmpsec_end)):
ctx.taintMemory(triton.MemoryAccess(memAcc.acc_address, memAcc.acc_size)) # taint bytecode from the VM section (so obfuscated indirect jumps aren't picked up too)
handler.append(ins)
compatibleProcessing(ctx, ttins, ins, nextIns, True, False) # execute the instruction
if (checkIndirectIns(ttins) and (nextIns is not None) and ttins.isTainted()):
handlers.append(handler.copy())
handler.clear()Then the key code — the rest of the work is driven by this script’s output:
def getMask(n):
if (n == 1):
return 0xff
elif (n == 2):
return 0xffff
elif (n == 4):
return 0xffffffff
elif (n == 8):
return 0xffffffffffffffff
else:
raise ValueError
for i, handler in enumerate(handlers):
firstIns = handler[0]
tmpctx = triton.TritonContext()
tmpctx.setArchitecture(triton.ARCH.X86_64)
tmpctx.setMode(triton.MODE.ALIGNED_MEMORY, True)
tmpctx.setMode(triton.MODE.AST_OPTIMIZATIONS, True)
tmpctx.setMode(triton.MODE.CONSTANT_FOLDING, True)
tmpctx.setMode(triton.MODE.TAINT_THROUGH_POINTERS, True)
initTritonCtxEnv(tmpctx, firstIns, main_thread.teb)
# you need to identify the dedicated registers yourself ahead of time;
# if they change, adjust accordingly
if (firstIns.dbg_id >= 0x35e9):
continue
vspRegName = "r8"
vpcRegName = "rdi"
vkeyRegName = "r9"
vbaseRegName = "rbx"
# vspRegName = "rsi"
# vpcRegName = "rbx"
# vkeyRegName = "rdi"
# vbaseRegName = "rbp"
vspReg = getattr(tmpctx.registers, vspRegName)
vpcReg = getattr(tmpctx.registers, vpcRegName)
vkeyReg = getattr(tmpctx.registers, vkeyRegName)
vbaseReg = getattr(tmpctx.registers, vbaseRegName)
tmpctx.taintRegister(vspReg)
tmpctx.taintRegister(vpcReg)
tmpctx.taintRegister(vkeyReg)
tmpctx.taintRegister(vbaseReg)
initVsp = tmpctx.getConcreteRegisterValue(vspReg)
initRsp = firstIns.reg_dump64.regcontext.csp
print(f"--[{i}]---------- {hex(firstIns.dbg_id)} addr: {hex(firstIns.ins_address)} --------------")
print(f"|\tvsp[{vspRegName}]: {hex(tmpctx.getConcreteRegisterValue(vspReg))[2:]} vpc[{vpcRegName}]: {hex(tmpctx.getConcreteRegisterValue(vpcReg))[2:]} rsp: {hex(firstIns.reg_dump64.regcontext.csp)[2:]} vkey[{vkeyRegName}]: {hex(tmpctx.getConcreteRegisterValue(vkeyReg))[2:]} vbase[{vbaseRegName}]: {hex(tmpctx.getConcreteRegisterValue(vbaseReg))[2:]}")
memInfo = []
for ii, ins in enumerate(handler):
if (ii + 1 >= len(handler)): nextIns = None
else: nextIns = handler[ii + 1]
ttins = triton.Instruction()
ttins.setAddress(ins.ins_address)
ttins.setOpcode(ins.bytes)
for memAcc in ins.mem_accs:
if (memAcc.type != Supertrace.AccessType.READ):
continue
for i in range(memAcc.acc_size):
memi = triton.MemoryAccess(memAcc.acc_address + i, triton.CPUSIZE.BYTE)
if (not tmpctx.isConcreteMemoryValueDefined(memi)):
oldby = (memAcc.old_data >> (i * 8)) & 0xFF
tmpctx.setConcreteMemoryValue(memi, oldby)
tmpctx.processing(ttins)
insType = ttins.getType()
if (insType == triton.OPCODE.X86.CPUID):
tmpctx.taintRegister(tmpctx.registers.rax)
tmpctx.taintRegister(tmpctx.registers.rbx)
tmpctx.taintRegister(tmpctx.registers.rcx)
tmpctx.taintRegister(tmpctx.registers.rdx)
elif (insType == triton.OPCODE.X86.RDTSC):
tmpctx.taintRegister(tmpctx.registers.rax)
tmpctx.taintRegister(tmpctx.registers.rdx)
elif (insType == triton.OPCODE.X86.RDTSCP):
tmpctx.taintRegister(tmpctx.registers.rax)
tmpctx.taintRegister(tmpctx.registers.rcx)
tmpctx.taintRegister(tmpctx.registers.rdx)
elif (insType == triton.OPCODE.X86.RDSEED or insType == triton.OPCODE.X86.RDRAND):
opreg: triton.Register = ttins.getOperands()[0]
tmpctx.taintRegister(opreg)
show = False
if (insType == triton.OPCODE.X86.LEA and not ttins.isTainted()): # lea instructions that compute addresses related to rsp
readRegs = ttins.getReadRegisters()
for ttreg, ast in readRegs:
if (tmpctx.getParentRegister(ttreg).getId() == triton.REG.X86_64.RSP):
regop: triton.Register = ttins.getOperands()[0]
show = True
if (tmpctx.getParentRegister(regop).getId() != triton.REG.X86_64.RSP): # e.g. lea rdx, qword ptr ss:[rsp+0x20]
tmpctx.taintRegister(regop)
break
if (ttins.isTainted() or show or insType == triton.OPCODE.X86.CPUID or insType == triton.OPCODE.X86.RDTSC or insType == triton.OPCODE.X86.RDTSCP or insType == triton.OPCODE.X86.RDSEED or insType == triton.OPCODE.X86.RDRAND):
asm = str(ttins)
asm = asm.replace(" + riz", "") # drop the 'riz' token from memory operands
for memacc in ins.mem_accs: # blank line before instructions that read VM bytecode
if (vmpsec_begin <= memacc.acc_address <= vmpsec_end):
print("|")
changed_vspOrvpc = " "
for ttreg, astNode in ttins.getWrittenRegisters(): # mark with * when vsp/vpc/vbase/vkey is written
if (tmpctx.getParentRegister(ttreg).getName() in [vspRegName, vpcRegName, vbaseRegName, vkeyRegName]):
changed_vspOrvpc = "*"
# blank line for the jump-to-next-handler part
if (ii == len(handler) - 1 and ttins.getType() == triton.OPCODE.X86.JMP): # "jmp reg" form
print(f"|")
elif (ii == len(handler) - 2 and ttins.getType() == triton.OPCODE.X86.MOV and handler[ii + 1].ttins.getType() == triton.OPCODE.X86.RET): # "mov [rsp], reg; ret" form
print(f"|")
print(f"|{changed_vspOrvpc}{hex(ins.dbg_id)} {asm}")
for ttreg, astNode in ttins.getWrittenRegisters(): # blank line after writing vkey
if (tmpctx.getParentRegister(ttreg).getName() in [vkeyRegName]):
print("|")
for memacc in ins.mem_accs:
mask = getMask(memacc.acc_size)
vrInfo = ""
VRLIMIT = 0x100 # this value comes from the corrected ecx in checkVsp
if ((initRsp + VRLIMIT > memacc.acc_address >= initRsp) and (memacc.acc_address & 7) == 0): # possible virtual-register addressing
off = (memacc.acc_address - initRsp)
vrN = off // 8
vrInfo = f" [+{hex(off)} = VR{vrN}]"
vspInfo = ""
# check whether the address is between the virtual registers and the stack bottom
# (so addresses like 0x10000 aren't mistaken for virtual-stack accesses)
if ((stackMemArea.addr + stackMemArea.size) >= memacc.acc_address >= (initRsp + VRLIMIT)): # possible virtual-stack addressing
off = memacc.acc_address - initVsp # offsets are computed from the vsp at the start of the handler
if (off == 0):
vspInfo = f" vsp[+0x0]"
elif (off > 0):
vspInfo = f" vsp[+{hex(off)}]"
else: # off < 0
vspInfo = f" vsp[{hex(off)}]"
if (memacc.type == Supertrace.AccessType.READ):
memInfo.append(f"| {hex(ins.dbg_id)} [R]{vrInfo}{vspInfo} {hex(memacc.acc_address)[2:]}:\t{hex(memacc.new_data & mask)[2:]}")
elif (memacc.type == Supertrace.AccessType.WRITE):
memInfo.append(f"| {hex(ins.dbg_id)} [W]{vrInfo}{vspInfo} {hex(memacc.acc_address)[2:]}:\t{hex(memacc.old_data & mask)[2:]} -> {hex(memacc.new_data & mask)[2:]}")
if (len(memInfo) > 0):
print("--------------------------")
for memI in memInfo:
print(memI)
print()
print()The idea is to taint everything tied to the VM machinery — the dedicated registers (vpc, vsp, vkey, vbase) plus special instructions like cpuid and rdtsc. The taint rules (which instructions to keep, which obfuscation to drop) can be tuned later. After my final pass, the output came out very clean — only a handful of handlers still dragged obfuscation in, and it didn’t matter.
Don’t taint
rspitself — that pulls in too many obfuscation instructions. Only taint instructions that address offrsp, e.g.lea rdx, qword ptr ss:[rsp+0x20]— taint justrdx.
This script doesn’t yet handle dedicated-register changes gracefully, so trace them manually first. Before index
0x35e9the(vsp, vpc, vkey, vbase)mapping is(rsi, rbx, rdi, rbp); after0x35e9it becomes(r8, rdi, r9, rbx).
VRLIMIT is the total size copied by rep movsb inside checkVsp (i.e. the value of ecx, since rep movsb copies ecx bytes):
We also find this value manually in the trace. It tells us the exact virtual-register boundary: checkVsp re-raises the virtual stack, and to do that it copies data to protect the virtual registers — so watching how many bytes it copies reveals the boundary. In this sample VRLIMIT is 0x100, and (0x100 // 8) = 0x20 = 32, so the virtual-register range is VR0 ~ VR31.
Here’s the output for the first handler:
--[0]---------- 0x80 addr: 0x1402257c3 --------------
| vsp[r8]: 64f492f688 vpc[rdi]: 14000651c rsp: 64f492f480 vkey[r9]: 1400f4f42 vbase[rbx]: 1402257c3
|
| 0x81 0x1402257c8: mov ecx, dword ptr [rdi - 4]
| 0x83 0x1402257d1: xor ecx, r9d
| 0x87 0x1402257df: neg ecx
| 0x8b 0x1402257ef: lea ecx, [rdx + rcx - 0x5fc1bee2]
| 0x8c 0x1402257f6: not ecx
| 0x90 0x14011659c: lea ecx, [rcx + rbp*2 + 0x54309f19]
| 0x93 0x1401165a8: ror ecx, 3
| 0x96 0x1401165b5: bswap ecx
| 0x99 0x1401165cb: mov qword ptr [rsp + rbp - 0xc02], r9
| 0x9a 0x1401165d3: xor dword ptr [rsp + rax - 0x9fffff8], ecx
|*0x9c 0x1401165de: mov r9, qword ptr [rsp + rax*2 - 0x1407fff8]
|
| 0x9e 0x1401165ee: movsxd rcx, ecx
|*0xa0 0x1401165f5: adc rbx, rcx
| 0xa3 0x140116604: mov rbp, qword ptr [r8 + rax*2 - 0x140860fa]
|
| 0xa7 0x140116620: movzx eax, byte ptr [rdi + rsi*2 - 0x764ef967]
| 0xab 0x14011663a: xor al, r9b
| 0xad 0x140116640: dec al
| 0xb1 0x1400c32bb: not al
| 0xb2 0x1400c32bd: sbb al, 0x37
| 0xb6 0x14018384c: rol al, 1
| 0xb8 0x140183851: dec al
|*0xba 0x140183857: xor r9b, al
|
| 0xbc 0x14018385f: lea rax, [rsp + rax + 0x18]
| 0xbe 0x140183866: mov qword ptr [rax + r11*2], rbp
| 0xc1 0x140084b20: mov rcx, qword ptr [r8 + rsi*2 - 0x764ef95a]
|*0xc3 0x140084b2c: lea r8, [r8 + r11*4 + 0x10]
|
| 0xc7 0x140084b40: xor bpl, byte ptr [rsi + rdi - 0x3b277cb7]
|*0xca 0x140084b4d: lea rdi, [rdi + rsi - 0x3b277cb7]
| 0xcb 0x140084b55: xor bpl, r9b
| 0xcc 0x140084b58: sub bpl, r10b
| 0xcf 0x140084b63: rol bpl, 1
| 0xd0 0x140084b66: xor bpl, 0x1d
| 0xd1 0x140084b6a: add bpl, r10b
| 0xd2 0x140084b6d: btr edx, r10d
| 0xd3 0x140084b71: ror bpl, 1
|*0xd5 0x140084b75: xor r9b, bpl
|
| 0xd8 0x140084b88: lea rbp, [rsp + rbp + 0x18]
| 0xdb 0x140084b95: mov qword ptr [rbp + rsi*2 - 0x764ef962], rcx
|
| 0xde 0x140084ba6: mov qword ptr [rsp + rdx - 0x20d51f31], rbx
| 0xdf 0x140084bae: ret 0x10
--------------------------
| 0x81 [R] 140006518: 629fecfd
| 0x99 [W] 64f492f478: c53800c09 -> 1400f4f42
| 0x9a [W] 64f492f478: 400f4f42 -> 400f4f42
| 0x9c [R] 64f492f478: 1400f4f42
| 0xa3 [R] vsp[+0x0] 64f492f688: 7ffffffffffffffc
| 0xa7 [R] 140006517: 7b
| 0xbe [W] [+0x20 = VR4] 64f492f4a0: 0 -> 7ffffffffffffffc
| 0xc1 [R] vsp[+0x8] 64f492f690: 21c3885b3e0
| 0xc7 [R] 140006516: 0
| 0xdb [W] [+0x70 = VR14] 64f492f4f0: 64f492f570 -> 21c3885b3e0
| 0xde [W] 64f492f468: b1 -> 1402257c3
| 0xdf [R] 64f492f468: 1402257c3(The remaining handlers follow the same structure; I’ve omitted them for brevity.)
From here it’s a long manual analysis. Handing the trace to Claude or GPT works well for this stage.
The final de-virtualization
Most handlers are composite; I use ; to separate the simple handlers inside a composite one.
VR4 = r8
VR14 = rbx
VR9 = r13
VR0 = rflags
VR19 = rax
VR20 = rcx
VR7 = Relocation Address
VR21 = r11
VR13 = rsi
VR18 = r15
VR15 = r14
VR17 = r10
VR1 = rdi
VR5 = rbp
VR8 = r12
VR11 = rdx
VR22 = r9
=================bytecode region[0x1400060f9 - 0x14000651c]=================
loc_0: // first entry into the VM
[0] vPopReg64 VR4; vPopReg64 VR14
[1] vPopReg64 VR9; vPopReg64 VR0
[2] vPopReg64 VR19; vPopReg64 VR20
[3] vPopReg64 VR7; vPopReg64 VR21
[4] vPopReg64 VR13; vPopReg64 VR18
[5] vPopReg64 VR15; vPopReg64 VR17
[6] vPopReg64 VR1; vPopReg64 VR5
[7] vPopReg64 VR8; vPopReg64 VR11
[8] vPopReg64 VR22; vPushImm64 0x140001188
[9] vPushReg64 VR7; vAdd64
[10] vPopReg64 VR24
[11] vPushVsp64; vPopReg64 VR25
[12] vPushReg64 VR5
[13] vPopReg64 VR26
[14] vPushReg64 VR13
[15] vPopReg64 VR27; vPushReg64 VR1
[16] vPopReg64 VR28; vPushReg64 VR14
[17] vPopReg64 VR29
[18] vPushImm32 0xdeadbeef
[19] vPopReg32 VR16
[20] vPushReg32 VR16; vPushReg32 VR17
[21] vPopReg32 VR2
[22] vPushReg32 VR2; vPushReg32 VR2
[23] vNand32; vAdd32 // nand(P, P) = not(P)
[24] vPushVsp64
[25] vReadMem32; vNand32
[26] vPopReg32 VR6
[27] vPushImm64 0x140006176 // candidate VPC1
[28] vPushImm64 0x1400061f3 // candidate VPC2
[29] vPushVsp64
[30] vPushImm16 0x19; vPushImm32 0x0
[31] vPushImm32 0x7fffffff; vPushReg32 VR6
[32] vAdd32; vPushReg32 VR6
[33] vNor32
[34] vPushImm32 0x80000000; vNand32
[35] vPushVsp64; vReadMem32
[36] vNand32
[37] vShr64; vPopReg64 VR10
[38] vPushImm16 0x3; vPushReg64 VR10
[39] vShr64; vPushImm64 0xfffffffffffffff7
[40] vNor64
[41] vAdd64; vReadMem64 // got the next vpc value
[42] vPopReg64 VR12; vPopReg64 VR23 // VR12 = next vpc value
[43] vPopReg64 VR23; vPushReg64 VR12
[44] vPushReg64 VR7; vAdd64
[45] vPopReg64 VR12; vPushReg64 VR5
[46] vPushReg64 VR14
[47] vPushReg64 VR21; vPushReg64 VR4
[48] vPushReg64 VR7
[49] vPushImm16 0x14; vPushImm32 0x0
[50] vPushReg32 VR2; vPushReg32 VR2
[51] vNand32; vPushReg32 VR16
[52] vPushReg32 VR16
[53] vNor32; vNor32
[54] vPushReg32 VR2; vPushReg32 VR16
[55] vNor32; vNor32
[56] vPushReg32 VR2; vPushReg32 VR2
[57] vNor32
[58] vPushReg32 VR6; vPushReg32 VR6
[59] vNor32; vNor32
[60] vPushReg32 VR2; vPushReg32 VR6
[61] vNor32; vNor32
[62] vNand32; vPushVsp64
[63] vReadMem32
[64] vNor32
[65] vPushImm32 0x80000000; vNand32
[66] vPushVsp64; vReadMem32
[67] vNor32
[68] vShr64; vPushImm16 0x18 // of
[69] vPushImm32 0x0
[70] vPushReg32 VR6; vPushImm32 0x80000000
[71] vNand32; vPushVsp64
[72] vReadMem32; vNand32
[73] vShr64; vAdd64 // sf
[74] vPushImm16 0x19; vPushImm32 0x0
[75] vPushImm32 0x7fffffff; vPushReg32 VR6
[76] vAdd32; vPushReg32 VR6
[77] vNor32
[78] vPushImm32 0x80000000; vNand32
[79] vPushVsp64; vReadMem32
[80] vNor32; vShr64
[81] vAdd64; vPushImm32 0x0 // zf
[82] vPushReg32 VR2; vPushReg32 VR2
[83] vNand32; vPushReg32 VR16
[84] vPushReg32 VR16; vNor32
[85] vNor32; vPushReg32 VR2
[86] vPushReg32 VR16; vNor32
[87] vNor32
[88] vPopReg32 VR10
[89] vPushReg32 VR10; vPushReg32 VR10
[90] vNor32; vPushReg32 VR6
[91] vPushReg32 VR6
[92] vNand32; vNor32
[93] vPushReg32 VR10; vPushReg32 VR6
[94] vNor32; vNor32
[95] vPushImm32 0x10
[96] vNand32; vPushVsp64
[97] vReadMem32; vNand32
[98] vAdd64 // af
[99] vPushImm16 0x4; vPushReg8 VR6
[100] vShr8
[101] vPopReg8 VR10; vPushImm16 0x2
[102] vPushImm16 0x0; vPushImm32 0x0
[103] vPushReg8 VR6
[104] vPushReg8 VR6; vNand8
[105] vPushReg8 VR10
[106] vNand8
[107] vPushReg8 VR6
[108] vPushReg8 VR10; vPushReg8 VR10
[109] vNand8; vNand8
[110] vNand8
[111] vPushImm16 0xf; vNand16
[112] vPushVsp64; vReadMem16
[113] vNor16
[114] vPushImm16 0x9669; vShr16
[115] vPushImm16 0x1; vNand16
[116] vPushVsp64; vReadMem16
[117] vNor16
[118] vSal64
[119] vAdd64 // pf
[120] vPushImm16 0x1f; vPushImm32 0x0
[121] vPushReg32 VR2; vPushReg32 VR2
[122] vNor32; vPushReg32 VR16
[123] vNand32; vPushVsp64
[124] vReadMem32; vNor32
[125] vPushReg32 VR2; vPushReg32 VR16
[126] vPushReg32 VR16; vNor32
[127] vNor32; vPushReg32 VR2
[128] vPushReg32 VR2; vNand32
[129] vPushReg32 VR16; vNor32
[130] vNor32; vPushReg32 VR6
[131] vNand32; vPushVsp64
[132] vReadMem32; vNand32
[133] vNor32
[134] vPushVsp64; vReadMem32
[135] vNor32
[136] vPushImm32 0x80000000; vNand32
[137] vPushVsp64; vReadMem32
[138] vNand32
[139] vShr64; vAdd64 // cf
[140] vPushReg64 VR0; vPushImm64 0xfffffffffffff72a
[141] vNand64
[142] vPushVsp64
[143] vReadMem64; vNand64
[144] vAdd64; vPopReg64 VR23 // write the new rflags into VR23
[145] vPushReg64 VR23; vPushReg64 VR19
[146] vPushReg64 VR1; vPushReg64 VR11
[147] vPushReg64 VR13; vPushReg64 VR18
[148] vPushReg64 VR9; vPushReg64 VR17
[149] vPushReg64 VR22; vPushReg64 VR15
[150] vPushReg64 VR20; vPushReg64 VR8
[151] vPushReg64 VR12
[152] vJmp // will jump to loc_1 or loc_2
loc_1: // jne not taken; it's 0xdeadbeef
[153] vPopReg64 VR0; vPopReg64 VR20
[154] vPopReg64 VR8; vPopReg64 VR15
[155] vPopReg64 VR3; vPopReg64 VR17
[156] vPopReg64 VR18; vPopReg64 VR22
[157] vPopReg64 VR9; vPopReg64 VR13
[158] vPopReg64 VR19; vPopReg64 VR11
[159] vPopReg64 VR1; vPopReg64 VR4
[160] vPopReg64 VR23; vPopReg64 VR14
[161] vPopReg64 VR7; vPushImm64 0x140002534
[162] vPushReg64 VR1; vAdd64
[163] vPopReg64 VR12; vPushReg64 VR7
[164] vPushReg64 VR14; vPushReg64 VR23
[165] vPushReg64 VR4; vPushReg64 VR1
[166] vPushReg64 VR11; vPushReg64 VR19
[167] vPushReg64 VR13; vPushReg64 VR9
[168] vPushReg64 VR22; vPushReg64 VR18
[169] vPushReg64 VR17; vPushReg64 VR3
[170] vPushReg64 VR15; vPushReg64 VR8
[171] vPushReg64 VR20; vPushReg64 VR0
[172] vPushReg64 VR12; vJmp // will jump to loc_3
loc_2: // jne taken; it's not 0xdeadbeef
[173] vPopReg64 VR5; vPopReg64 VR20
[174] vPopReg64 VR8; vPopReg64 VR21
[175] vPopReg64 VR15; vPopReg64 VR17
[176] vPopReg64 VR3; vPopReg64 VR0
[177] vPopReg64 VR22; vPopReg64 VR18
[178] vPopReg64 VR13; vPopReg64 VR11
[179] vPopReg64 VR9; vPopReg64 VR19
[180] vPopReg64 VR23; vPopReg64 VR4
[181] vPopReg64 VR1; vPushImm64 0x1400025d1
[182] vPushReg64 VR9; vAdd64
[183] vPopReg64 VR14; vPushReg64 VR1
[184] vPushReg64 VR4; vPushReg64 VR23
[185] vPushReg64 VR19; vPushReg64 VR9
[186] vPushReg64 VR11; vPushReg64 VR13
[187] vPushReg64 VR18; vPushReg64 VR22
[188] vPushReg64 VR0; vPushReg64 VR3
[189] vPushReg64 VR17; vPushReg64 VR15
[190] vPushReg64 VR21; vPushReg64 VR8
[191] vPushReg64 VR20; vPushReg64 VR5
[192] vPushReg64 VR14; vJmp // will jump to loc_4
=================bytecode region[0x140002534 - 0x14000265f]=================
loc_3:
[193] vPopReg64 VR20
[194] vPopReg64 VR9; vPopReg64 VR12
[195] vPopReg64 VR18
[196] vPopReg64 VR19; vPopReg64 VR4
[197] vPopReg64 VR6; vPopReg64 VR10
[198] vPopReg64 VR15; vPopReg64 VR3
[199] vPopReg64 VR7; vPopReg64 VR17
[200] vPopReg64 VR22; vPopReg64 VR1
[201] vPopReg64 VR21; vPopReg64 VR8
[202] vPopReg64 VR11; vPushImm64 0x140001188
[203] vPushReg64 VR22
[204] vAdd64; vPopReg64 VR24
[205] vPushImm64 0xdeadbeef
[206] vPopReg64 VR0; vPushImm64 0x1615d9430
[207] vPopReg64 VR23
[208] vPushReg64 VR11
[209] vPushReg64 VR8; vPushReg64 VR21
[210] vPushReg64 VR1; vPushReg64 VR22
[211] vPushReg64 VR17; vPushReg64 VR0
[212] vPushReg64 VR3; vPushReg64 VR15
[213] vPushReg64 VR10; vPushReg64 VR6
[214] vPushReg64 VR4; vPushReg64 VR19
[215] vPushReg64 VR18; vPushReg64 VR12
[216] vPushReg64 VR9; vPushReg64 VR20
[217] vNop
loc_4:
[218] vPopReg64 VR14; vPopReg64 VR7
[219] vPopReg64 VR16; vPopReg64 VR23
[220] vPopReg64 VR20; vPopReg64 VR13
[221] vPopReg64 VR18; vPopReg64 VR19
[222] vPopReg64 VR10; vPopReg64 VR12
[223] vPopReg64 VR15; vPopReg64 VR3
[224] vPopReg64 VR4; vPopReg64 VR17
[225] vPopReg64 VR2; vPopReg64 VR8
[226] vPopReg64 VR5; vPushImm64 0x140001188
[227] vPushReg64 VR4; vAdd64
[228] vPopReg64 VR24; vPushImm64 0x1400011c0
[229] vPushReg64 VR4; vAdd64
[230] vPushReg64 VR23; vPushReg64 VR7
[231] vPushReg64 VR2; vPushReg64 VR19
[232] vPushReg64 VR14; vPushReg64 VR8
[233] vPushReg64 VR20; vPushReg64 VR13
[234] vPushReg64 VR17; vPushReg64 VR10
[235] vPushReg64 VR12; vPushReg64 VR16
[236] vPushReg64 VR5; vPushReg64 VR15
[237] vPushReg64 VR3; vPushReg64 VR18
[238] vExit
r15 = VR18
rflags = VR3
rax = VR15
rbp = VR5
r14 = VR16
rdi = VR12
rdx = VR10
r8 = VR17
r13 = VR13
r10 = VR20
rbx = VR8
r12 = VR14
rsi = VR19
r11 = VR2
rcx = VR7
r9 = VR23The reconstructed control-flow graph:
The recovered semantics (from [18] to [152]), with help from GPT-5.5 Thinking:
uint32_t A = VR2;
uint32_t B = VR16; // 0xDEADBEEF
uint32_t R = VR6; // A - B
uint64_t target;
if (R == 0)
target = 0x1400061F3;
else
target = 0x140006176;
// VPC correction / rolling-key correction
VR12 = target + VR7;
// simulate cmp/sub EFLAGS
uint64_t flags = 0;
flags |= (((A ^ B) & (A ^ R)) & 0x80000000) >> 20; // OF
flags |= (R & 0x80000000) >> 24; // SF
flags |= (R == 0 ? 1ull : 0ull) << 6; // ZF
flags |= (A ^ B ^ R) & 0x10; // AF
flags |= parity_even(R & 0xff) << 2; // PF
flags |= (((~A & B) | (~(A ^ B) & R)) & 0x80000000) >> 31; // CF
VR23 = (VR0 & 0xfffffffffffff72a) | flags;
// finally update the VM's VPC/VKEY
StoreVPC(VR12);Summary
The workflow that worked: pack a known function under VMProtect, trace it with x64dbg, split the trace into handlers, taint the VM-specific registers and bytecode reads with Triton to filter out the obfuscation noise, then reconstruct each handler’s virtual instruction sequence by hand (with LLM help for the semantics). The end result is a readable, recompilable cmp/jne — flags, branch targets, rolling key, and all.
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch