Deobfuscating VMProtect 3.9.4: Junk Code, Code Shuffling, Opaque Predicates, and More
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
The long-term goal is to identify the semantics of each VMProtect handler with symbolic execution — instead of the traditional instruction-pattern matching approach (which is also incomplete now, since modern VMProtect seems to pack multiple VM opcodes into a single handler, so pattern-matched handler semantics are no longer exhaustive). But while analyzing traces, staring at a wall of obfuscated instruction data gets exhausting. So, one step at a time: deobfuscate first. That’s what this article covers.
Tools
- Triton
- Miasm
- Keystone
Trace data is produced with the custom x64dbg plugins supertrace and supertrace-pybind, which enrich the trace with extra metadata for analysis:
Obfuscation observed in VMProtect 3.9.4
Analyzing traces of a VMProtect 3.9.4-protected sample, these are the obfuscation techniques I found:
(1) Junk code
Junk code is register/memory writes produced by earlier instructions that are overwritten by later instructions before being read. The most stable and classic technique; no further explanation needed.
mov eax, 0x66
add eax, ebx
mov eax, 0x77
(2) Code-locality scrambling (a.k.a. code shuffling)
A basic block that should be one block is forcibly split into two, connected by direct control-flow instructions (jmp imm or call imm) in order. The result is that in a debugger you watch RIP bounce back and forth constantly.
VMProtect uses this heavily, and combines it with always-true/always-false opaque predicates (below) to strengthen the effect.
code A
code B
code C
......
==>
jmp labelA
labelB:
code B
jmp labelC
labelC:
code C
jmp next
labelA:
code A
jmp labelB
next:
......

(From here on I’ll just call it “code shuffling”.)
(3) Opaque predicates (always-true / always-false)
“Opaque” means hard for the attacker to infer. The obfuscator’s author knows the predicate’s value; the attacker can’t tell from the surface. VMProtect’s opaque predicates take the always-true/always-false form — I didn’t observe any “sometimes-true-sometimes-false” predicates (though I’ve heard older VMProtect used them).
“Always-true/always-false” means the jcc in a basic block never changes its taken/not-taken fact: it either always jumps or never jumps (the condition is either always true or always false).
There are two ways to build these. One is a condition built entirely from constants, so the result is constant:
/*
520
3 + 3 == 6
256 ^ 127 != 382
......
*/
if (cond) {
// the real code block
} else {
// never executed; would crash if it ever ran
}
The other substitutes user-controllable variables into the condition, but the expression’s result is invariant regardless of the variable’s value:
/*
x * (x + 1) % 2 == 0
pow(x, 2) + pow(y, 2) >= 2*x*y // the classic AM-GM inequality
......
*/
if (cond) {
// the real code block
} else {
// never executed; would crash if it ever ran
}

(4) Memory-operand constant hiding
VMProtect is a stack-based VM, so stack access and VM-bytecode reads from the VM block naturally produce lots of memory accesses. VMProtect (this technique appears to have been introduced around 3.8) inserts complex values into otherwise-normal memory operands and represents them with registers. For example, qword ptr ss:[rsp] becomes qword ptr ss:[rsp + rdx - 0x76B2378A] (where rdx is currently 0x76B2378A).

The index or base registers VMProtect introduces into memory operands are all known constants. On closer inspection, those values are computed step-by-step from the immediates of previously executed instructions.
A constant’s computation process (backward slice on rax):
0663B | 000000014042DFC2 | mov rbx,0
rbx: 0000FFFFFFFFFFFF -> 0000000000000000
0663E | 000000014042DFD3 | mov ebx,53AE5A96
rbx: 0000000000000000 -> 0000000053AE5A96
0663F | 000000014042DFD8 | lea rdi,qword ptr ds:[rbx+rbx*8+58877E8F]
rdi: 000000000000001F -> 0000000349A8ADD5
rbx: 0000000053AE5A96
06642 | 000000014042DFEC | dec dil
rdi: 0000000349A8ADD5 -> 0000000349A8ADD4
06649 | 0000000140136F93 | mov eax,edi
rax: 00000000F958441E -> 0000000049A8ADD4
rdi: 0000000349A8ADD4
0664F | 0000000140136FA2 | rol dword ptr ss:[rsp+rax-49A8ADCF],8E
rax: 0000000049A8ADD4
rsp: 000000CCD0FCF860
(5) MBA (mixed Boolean-arithmetic) obfuscation
Transforms a simple expression into a complex arithmetic-plus-Boolean expression. A classic data-flow obfuscation; VMProtect’s “universal gate” computation uses it heavily.
examples:
x + y == (x | y) + y - (~x & y)
x ^ y == (x | y) - y + (~x & y)
x ^ y == (x & ~y) | (~x & y)
x | y == (x ^ y) + y - (~x & y)
x & y == -(x | y) + y + x
......

(6) Indirect-jump obfuscation
The indirect jump’s target is actually known and single. VMProtect breaks the conventional call contract (a call pushes the return address of the next instruction onto the stack; on return, ret pops [rsp] and returns to the call site). It reads the return address off the stack, runs a series of computations on it, derives the real target, and jumps there.
Really this is also a form of code shuffling, since call is semantically push retAddr; jmp targetAddr.
(7) Semantics-preserving instruction substitution
Replaces an instruction with another, more complex (or less common) one while preserving semantics. E.g. push rax → lea rsp, qword ptr ss:[rsp - 8]; mov qword ptr ss:[rsp], rax; or mov rax, 5 → or rax, 5 (rax guaranteed 0) or add rax, 5 (rax guaranteed 0).
The first example uses
lea rsp, qword ptr ss:[rsp - 8]rather thansub rsp, 8becausesubwould additionally introduce flag-modifying semantics. That’s the “semantics preservation” point. The second example is the same idea.
How to validate deobfuscation results
Stability check: after deobfuscating, you have to verify the new code still works. My approach: turn off VMProtect’s memory protection, overwrite the original obfuscated instructions with the deobfuscated ones, and check the program still runs.
Reliability check: manually inspect whether any obfuscated instructions remain. If you still see them, the deobfuscation is incomplete.
Setup
Compile this with VS2022 Release x64:
#include <iostream>
#include <Windows.h>
#include "VMProtectSDK.h"
void test(unsigned x) {
VMProtectBegin("secret");
if (x == 0xDEADBEEF) { // 3735928559
MessageBoxA(NULL, "DEADBEEF", "", MB_OK);
}
VMProtectEnd();
}
int main(int ac, const char* av[]) {
unsigned x = 0;
printf("x: ");
scanf("%d", &x);
test(x);
return 0;
}
Protect it with VMProtect 3.9.4:

Then record the full instruction trace with x64dbg: find the VM function, set breakpoints at its entry and exit, start tracing, and stop it when the exit breakpoint hits.

Now load the trace:
import triton
import keystone
import SupertracePybind as Supertrace
from enum import Enum
from collections import defaultdict
import graphviz
from miasm.arch.x86.arch import mn_x86
from miasm.expression.expression import ExprMem, ExprId, ExprInt, ExprOp
from supertrace_util import initTritonCtxEnv, mergeRepeatIns, checkIndirectIns
ks = keystone.Ks(keystone.KS_ARCH_X86, keystone.KS_MODE_64)
def asmDecode(CODE: bytes, addr = 0) -> bytes:
try:
encoding, count = ks.asm(CODE, addr, True)
if (count <= 0):
raise keystone.KsError
return bytes(encoding)
except keystone.KsError as e:
print("ERROR: %s" %e)
return b""
tracepath = "1.trace64" # read the trace file
trace = Supertrace.parse_x64dbg_trace(tracepath)
record = trace.getRecord()
print(f"trace instruction num: {len(record)}")
'''
trace instruction num: 110684
'''
ctx = triton.TritonContext()
ctx.setArchitecture(triton.ARCH.X86_64)
ctx.setMode(triton.MODE.ALIGNED_MEMORY, True)
ctx.setMode(triton.MODE.AST_OPTIMIZATIONS, True)
ctx.setMode(triton.MODE.CONSTANT_FOLDING, True)
def newLiftDot(expr_node) -> str:
temp_ctx = triton.TritonContext()
temp_ctx.setArchitecture(ctx.getArchitecture())
return temp_ctx.liftToDot(expr_node)
def initTritonCtxEnvSym(
ctx: triton.TritonContext,
init_refer_ins: Supertrace.InstructionRecord,
teb_addr: int, symbolizeDF: bool = False, symbolizeTF: bool = False, symbolizeIF: bool = False):
initTritonCtxEnv(ctx, init_refer_ins, teb_addr) # initialize the concrete environment
regs = ctx.registers
if (ctx.getArchitecture() == triton.ARCH.X86):
init_regdump = init_refer_ins.reg_dump32
elif ctx.getArchitecture() == triton.ARCH.X86_64:
init_regdump = init_refer_ins.reg_dump64
else:
print("unsurport arch")
# symbolize all registers
for ttreg in ctx.getAllRegisters():
ctx.symbolizeRegister(ttreg, ttreg.getName())
# flat model
ctx.setConcreteRegisterValue(ctx.registers.ss, 0)
ctx.setConcreteRegisterValue(ctx.registers.ds, 0)
ctx.setConcreteRegisterValue(ctx.registers.es, 0)
ctx.setConcreteRegisterValue(ctx.registers.cs, 0)
if ctx.getArchitecture() == triton.ARCH.X86:
ctx.setConcreteRegisterValue(ctx.registers.esp, init_regdump.regcontext.csp)
ctx.setConcreteRegisterValue(ctx.registers.eip, init_regdump.regcontext.cip)
ctx.setConcreteRegisterValue(regs.fs, teb_addr)
ctx.setConcreteRegisterValue(regs.gs, 0) # flat model
elif ctx.getArchitecture() == triton.ARCH.X86_64:
ctx.setConcreteRegisterValue(ctx.registers.rsp, init_regdump.regcontext.csp)
ctx.setConcreteRegisterValue(ctx.registers.rip, init_regdump.regcontext.cip)
ctx.setConcreteRegisterValue(regs.fs, 0) # flat model
ctx.setConcreteRegisterValue(regs.gs, teb_addr)
if (symbolizeDF):
ctx.setConcreteRegisterValue(ctx.registers.df, init_regdump.flags.d)
if (symbolizeTF):
ctx.setConcreteRegisterValue(ctx.registers.tf, init_regdump.flags.t)
if (symbolizeIF):
ctx.setConcreteRegisterValue(getattr(ctx.registers, "if"), init_regdump.flags.i)
ctx.setConcreteRegisterValue(ctx.registers.nt, 0)
ctx.setConcreteRegisterValue(ctx.registers.rf, 0)
ctx.setConcreteRegisterValue(ctx.registers.id, 0)
ctx.setConcreteRegisterValue(ctx.registers.vip, 0)
ctx.setConcreteRegisterValue(ctx.registers.vif, 0)
ctx.setConcreteRegisterValue(ctx.registers.ac, 0)
ctx.setConcreteRegisterValue(ctx.registers.vm, 0)
record = mergeRepeatIns(ctx, record, True)
print(f"after mergeing 'rep' instructions, trace instruction num: {len(record)}") # merge rep instructions in the trace
'''
after mergeing 'rep' instructions, trace instruction num: 110684
'''
threads = trace.user.meta.getThreads()
for th in threads:
if th.id == record[0].thread_id:
main_thread = th
break
print(f"main thread id: {main_thread.id} ({hex(main_thread.id)})")
print(f"teb: {hex(main_thread.teb)}")
initTritonCtxEnv(ctx, record[0], main_thread.teb)
'''
main thread id: 33076 (0x8134)
teb: 0xccd10d8000
'''
Deobfuscation
1. Extracting basic blocks
First extract basic blocks from the trace (the deobfuscation operates on basic blocks). Blocks extracted in trace order will naturally contain duplicates (repeated executions), so store them in a set to dedupe.
The basic-block class:
class ST_BasicBlock():
def __init__(self, instructions: list[Supertrace.InstructionRecord]):
self.instructions: list[Supertrace.InstructionRecord] = instructions
# for the last instruction of the block
self.isControlflow: bool = None
self.isBranch: bool = None
self.isUnconditionalJump: bool = None
self.isConditionalJump: bool = None
self.isCall: bool = None
self.isRet: bool = None
self.isDirect: bool = None
self.targetAddr: int = None
self.fallThroughAddr: int = None
# (indirectTargetAddrs is not a formal control-flow field; its data comes from the trace,
# not the instruction itself, so per convention writeBBControlIns does not touch it)
self.indirectTargetAddrs: set[int] = set()
def __hash__(self):
h: int = 0
for ins in self.instructions:
h1 = hash(ins.bytes)
h2 = hash(ins.ins_address)
h += (h1 + h2)
h += hash(len(self.instructions))
return h
def __eq__(self, other):
if not isinstance(other, ST_BasicBlock):
return NotImplemented
if (len(self.instructions) != len(other.instructions)):
return False
for i, ins in enumerate(self.instructions):
if (ins.bytes != other.instructions[i].bytes):
return False
if (ins.ins_address != other.instructions[i].ins_address):
return False
return True
def __repr__(self):
return f"<{hex(self.getCurrentAddr())}> insCount: {len(self.instructions)}"
def printbb(self):
'''print all instructions of the block'''
for ins in self.instructions:
print(f"{ins}\t{ins.ttins}")
def getFirstIns(self) -> Supertrace.InstructionRecord:
'''return the first instruction of the block'''
return self.instructions[0]
def getLastIns(self) -> Supertrace.InstructionRecord:
'''return the last instruction of the block'''
return self.instructions[-1]
def getCurrentAddr(self) -> int:
'''return the address of this block'''
return self.getFirstIns().ins_address
def isReachMe(self, jumpAddr: int) -> bool:
'''whether a jump target enters this block'''
return jumpAddr == self.getFirstIns().ins_address
def findSubIns(self, checkIns: Supertrace.InstructionRecord) -> int:
'''return the index if contained, else -1'''
for i, ins in enumerate(self.instructions):
if (ins == checkIns):
return i
return -1
def isContainsAddr(self, checkAddr: int) -> int:
'''return the index if contained, else -1'''
for i, ins in enumerate(self.instructions):
if (ins.ins_address == checkAddr):
return i
return -1
def findSubBasicBlock(self, otherBB) -> int:
'''return the start index if contained, else -1'''
otherBB: ST_BasicBlock = otherBB
if (self == otherBB): # goes through __eq__ above
return 0
if (len(self.instructions) < len(otherBB.instructions)):
return -1
startIdx = self.findSubIns(otherBB.instructions[0])
if (startIdx == -1):
return -1
i = startIdx
for otherBBIns in otherBB.instructions:
if (self.instructions[i] != otherBBIns):
return -1
i += 1
return startIdx
def writeBBControlIns(bb: ST_BasicBlock, ttins: triton.Instruction):
bb.isControlflow = False
bb.isBranch = False
bb.isUnconditionalJump = False
bb.isConditionalJump = False
bb.isCall = False
bb.isRet = False
bb.isDirect = False
bb.targetAddr = 0
bb.fallThroughAddr = 0
# ---------
bb.isControlflow = ttins.isControlFlow()
if (bb.isControlflow):
ttinsType = ttins.getType()
bb.isCall = (ttinsType == triton.OPCODE.X86.CALL)
bb.isRet = (ttinsType == triton.OPCODE.X86.RET)
bb.isBranch = ttins.isBranch()
if (bb.isBranch):
bb.isUnconditionalJump = (ttinsType == triton.OPCODE.X86.JMP)
bb.isConditionalJump = (not bb.isUnconditionalJump)
if (bb.isCall or bb.isBranch): # ret is never a direct jump
ops = ttins.getOperands()
bb.isDirect = (len(ops) == 1 and ops[0].getType() == triton.OPERAND.IMM)
if (bb.isDirect):
immop: triton.Immediate = ops[0]
if (ttins.getAddress() == 0):
print("[warning] writeBBControlIns(): ttins.getAddress() is 0")
bb.targetAddr = immop.getValue()
bb.fallThroughAddr = ttins.getNextAddress()
bbs: set[ST_BasicBlock] = set()
Filling the blocks works by reading the recorded instructions in order; on hitting a control-flow instruction, write the collected instructions into a block, call writeBBControlIns() to fill in the jump info of the last instruction, and add it to bbs. Since bbs is a set, Python dedupes automatically (at this point the blocks no longer have a notion of trace time).
bbinsList: list[Supertrace.InstructionRecord] = []
bbQuickDict = {} # just a performance aid
for i, ins in enumerate(record):
ttins = triton.Instruction()
ttins.setOpcode(ins.bytes)
ttins.setAddress(ins.ins_address)
ctx.disassembly(ttins)
ins.ttins = ttins
bbinsList.append(ins)
if (ttins.isControlFlow()):
bb = ST_BasicBlock(bbinsList.copy())
if (bb in bbs):
find = bbQuickDict[bb.getCurrentAddr()]
if (checkIndirectIns(ttins) and (i + 1 < len(record))): # fill indirectTargetAddrs separately
find.indirectTargetAddrs.add(record[i + 1].ins_address)
else:
writeBBControlIns(bb, ttins)
if (checkIndirectIns(ttins) and (i + 1 < len(record))): # fill indirectTargetAddrs separately
bb.indirectTargetAddrs.add(record[i + 1].ins_address)
bbs.add(bb)
bbQuickDict[bb.getCurrentAddr()] = bb
bbinsList.clear()
print(f"basic blocks collected from trace: {len(bbs)}")
'''
basic blocks collected from trace: 3577
'''
That yields 3577 basic blocks.
One note: extracting blocks from a trace can produce the case where a big block wraps a small block, because a jump may target the inside of another block — that would require block splitting. But by observation (at least for this sample), VMProtect doesn’t do that, so it’s omitted here.
2. Merging basic blocks
First, graph the pre-deobfuscation basic-block linkage. Define the edge class:
class BBLinkType(Enum):
INDIRECT_UNCOND = 0 # unconditional indirect jump
DIRECT_UNCOND = 1 # unconditional direct jump
DIRECT_COND_TAKEN = 2 # conditional direct jump (taken)
DIRECT_COND_NOTTAKEN = 3 # conditional direct jump (not taken)
DIRECT_FALLTHROUGH = 4 # sequential fallthrough (for blocks without control-flow instructions)
OBF_JUMP = 5 # obfuscated jump
class BBLink():
def __init__(self, linkType: BBLinkType, linkAddress):
self.type = linkType
self.addr = linkAddress
def __hash__(self):
h: int = 0
h += hash(self.type.name)
h += hash(hex(self.addr))
return h
def __eq__(self, other):
if not isinstance(other, BBLink):
return NotImplemented
if (other.type != self.type): return False
if (other.addr != self.addr): return False
return True
def __repr__(self):
return f"type: {self.type}\taddr: {hex(self.addr)}"
Then the main graph-drawing code:
def drawBBLinkGraph(bbs: set[ST_BasicBlock], createUnkNodeWhenBBInvalid: bool = False):
'''createUnkNodeWhenBBInvalid: if the target node does not exist, create an UNK node for it
based on this flag, then draw the edge; otherwise skip the edge.'''
g = graphviz.Digraph('BasicBlockLinkGraph', graph_attr={'rankdir': 'LR'})
valids = set() # fast existence check for a block
for bb in bbs:
node = bb.getCurrentAddr()
valids.add(node)
g.node(hex(node), label=f'{hex(node)}: {len(bb.instructions)}', color="")
precursors = defaultdict(set[BBLink]) # predecessors
successors = defaultdict(set[BBLink]) # successors
for bb in bbs:
current = bb.getCurrentAddr()
if (bb.isControlflow):
if (bb.isDirect):
if (bb.isConditionalJump): # conditional jump
if not (hasattr(bb, "obfOPStatus") and bb.obfOPStatus == True): # ObfOP deobfuscation
taken1 = BBLink(BBLinkType.DIRECT_COND_TAKEN, bb.targetAddr)
taken2 = BBLink(BBLinkType.DIRECT_COND_NOTTAKEN, bb.fallThroughAddr)
target = set({taken1, taken2})
else:
if (bb.obfOPCondTaken):
taken = BBLink(BBLinkType.OBF_JUMP, bb.targetAddr)
else:
taken = BBLink(BBLinkType.OBF_JUMP, bb.fallThroughAddr)
target = set({taken})
else: # unconditional jump: jmp imm or call imm
taken = BBLink(BBLinkType.DIRECT_UNCOND, bb.targetAddr)
target = set({taken})
else: # indirect jump
if not (hasattr(bb, "obfIndirectStatus") and bb.obfIndirectStatus == True): # ObfIndirect deobfuscation
target = set()
for indirectTargetAddr in bb.indirectTargetAddrs:
taken = BBLink(BBLinkType.INDIRECT_UNCOND, indirectTargetAddr)
target.add(taken)
else:
taken = BBLink(BBLinkType.OBF_JUMP, bb.obfIndirectAddr)
target = set({taken})
else:
# block with no control-flow instruction
taken = BBLink(BBLinkType.DIRECT_FALLTHROUGH, bb.fallThroughAddr)
target = set({taken})
successors[current] = successors[current].union(target)
for taken in target:
precursors[taken.addr].add(BBLink(taken.type, current))
for succSrc in successors:
succDests = successors[succSrc]
for succDestTaken in succDests:
if (succDestTaken.type == BBLinkType.INDIRECT_UNCOND):
color, style = "blue", "dashed"
elif (succDestTaken.type == BBLinkType.DIRECT_UNCOND):
color, style = "blue", "solid"
elif (succDestTaken.type == BBLinkType.DIRECT_COND_TAKEN):
color, style = "green", "solid"
elif (succDestTaken.type == BBLinkType.DIRECT_COND_NOTTAKEN):
color, style = "red", "solid"
elif (succDestTaken.type == BBLinkType.DIRECT_FALLTHROUGH):
color, style = "grey", "solid"
elif (succDestTaken.type == BBLinkType.OBF_JUMP):
color, style = "pink", "solid"
else:
raise RuntimeError()
if (succDestTaken.addr in valids): # only draw successor edges
g.edge(hex(succSrc), hex(succDestTaken.addr), color=color, style=style)
else:
if (createUnkNodeWhenBBInvalid):
g.edge(hex(succSrc), hex(succDestTaken.addr), color=color, style=style)
g.node(hex(succDestTaken.addr), label=f'UNK_{hex(succDestTaken.addr)}', color="red")
return g
Edge style legend:
- Indirect unconditional jump (
call reg,jmp reg): blue dashed - Direct unconditional jump (
jmp imm,call imm): blue solid - Taken direct conditional jump (taken
jcc): green solid - Not-taken direct conditional jump (not-taken
jcc): red solid - Fallthrough: grey solid
- Jumps proven to be obfuscated: pink solid

That’s too big, so here are a couple of local zooms:

The merge rule: a block has exactly one successor, and that successor has only this one predecessor (i.e. no branch or merge point). Additionally, blocks with no control-flow interruption (no jump) and logical continuity may also be merged.

In the figure above, 0x14043bf81: 6 has exactly one successor 0x140269a1f: 5, and 0x140269a1f: 5 has exactly one predecessor 0x14043bf81: 6, linked by a blue-solid direct unconditional jump. So they can merge into a new block 0x14043bf81: 11. Its predecessor relation is inherited from 0x14043bf81: 6, its successor relation from 0x140269a1f: 5.
Merging iterates: the new block 0x14043bf81: 11 and 0x1402819e5: 68 also satisfy the rule, so they merge into 0x14043bf81: 79.
The merge code:
def merge_basic_block(bbs: set[ST_BasicBlock]) -> tuple[set[ST_BasicBlock], bool]:
hasChange = False
bbs = set(bbs)
def mergeBasicBlock(a: ST_BasicBlock, b: ST_BasicBlock) -> ST_BasicBlock:
'''append b after a; the new block inherits b's attributes'''
newbb = ST_BasicBlock(a.instructions + b.instructions)
writeBBControlIns(newbb, b.getLastIns().ttins)
newbb.indirectTargetAddrs = b.indirectTargetAddrs
return newbb
def getPrec(precAddr: int, checkAddr: int) -> BBLink:
for prec in precursors[precAddr]:
if (prec.addr == checkAddr):
return prec
return None
precursors = defaultdict(set[BBLink]) # predecessors
successors = defaultdict(set[BBLink]) # successors
for bb in bbs:
current = bb.getCurrentAddr()
if (bb.isControlflow):
if (bb.isDirect):
if (bb.isConditionalJump): # conditional jump
if not (hasattr(bb, "obfOPStatus") and bb.obfOPStatus == True): # ObfOP deobfuscation
taken1 = BBLink(BBLinkType.DIRECT_COND_TAKEN, bb.targetAddr)
taken2 = BBLink(BBLinkType.DIRECT_COND_NOTTAKEN, bb.fallThroughAddr)
target = set({taken1, taken2})
else:
if (bb.obfOPCondTaken):
taken = BBLink(BBLinkType.DIRECT_UNCOND, bb.targetAddr)
else:
taken = BBLink(BBLinkType.DIRECT_FALLTHROUGH, bb.fallThroughAddr)
target = set({taken})
else: # unconditional jump: jmp imm or call imm
taken = BBLink(BBLinkType.DIRECT_UNCOND, bb.targetAddr)
target = set({taken})
else: # indirect jump
if not (hasattr(bb, "obfIndirectStatus") and bb.obfIndirectStatus == True): # ObfIndirect deobfuscation
target = set()
for indirectTargetAddr in bb.indirectTargetAddrs:
taken = BBLink(BBLinkType.INDIRECT_UNCOND, indirectTargetAddr)
target.add(taken)
else:
taken = BBLink(BBLinkType.DIRECT_UNCOND, bb.obfIndirectAddr)
target = set({taken})
else:
# block with no control-flow instruction
taken = BBLink(BBLinkType.DIRECT_FALLTHROUGH, bb.fallThroughAddr)
target = set({taken})
successors[current] = successors[current].union(target)
for taken in target:
precursors[taken.addr].add(BBLink(taken.type, current))
goNext = True
while(goNext):
goNext = False
for bb in bbs:
current = bb.getCurrentAddr()
for succ in successors[current]:
target = succ.addr
succCheck = (len(successors[current]) == 1) and \
(succ.type == BBLinkType.DIRECT_UNCOND or succ.type == BBLinkType.DIRECT_FALLTHROUGH)
precCheck = (target in precursors) and \
(len(precursors[target]) == 1) and \
(getPrec(target, current) != None) and \
(getPrec(target, current).type == BBLinkType.DIRECT_UNCOND or getPrec(target, current).type == BBLinkType.DIRECT_FALLTHROUGH)
if (succCheck and precCheck):
targetBB = FindBB(bbs, target)
if (targetBB == None):
print(f"error! target: {hex(target)} bb: {bb} does not exist")
continue
mergebb = mergeBasicBlock(bb, targetBB)
mergeAddr = mergebb.getCurrentAddr()
assert(mergeAddr == current)
successors[mergeAddr] = successors[target].copy()
successors[target].clear()
precursors[target].clear()
for newSucc in successors[mergeAddr]:
removePrec = getPrec(newSucc.addr, target)
assert(removePrec != None)
precursors[newSucc.addr].remove(removePrec)
precursors[newSucc.addr].add(BBLink(newSucc.type, mergeAddr))
bbs.remove(targetBB)
bbs.remove(bb)
bbs.add(mergebb)
goNext = True
hasChange = True
break
if (goNext):
break
return bbs, hasChange
Next, a code-logic model for deobfuscation. Obfuscation is generally applied iteratively, and the iteration always has a bounded number of rounds (usually decided by the protector’s parameters). Deobfuscation is the inverse: however many rounds of obfuscation were applied, undo that many rounds. Since we don’t know the exact count in practice, use “did this deobfuscation pass actually change anything” as the termination criterion: apply one pass; if it was effective, run another; until a pass is ineffective, then stop.
To coordinate the deobfuscation processors, a processor may read the obfuscation-flag attributes set by other processors, but each processor must independently complete its own task — one processor’s deobfuscation must not depend on another’s flag. (E.g. merge_basic_block below may use obfOPStatus / obfOPCondTaken set by prove_opaque_predicate, but even without those flags it must still be able to merge blocks.)
The code model:
# deobfuscation iteration loop
keepRun = True
runCount = 0 # iteration count
processors: list = [...]
while(keepRun):
runCount += 1
keepRun = False
for processor in processors:
hasChange = False
bbs, hasChange = processor(bbs)
if (hasChange):
keepRun = True
keepRun decides whether the loop continues; the local hasChange says whether the processor applied deobfuscation successfully. If any processor returns hasChange == True, the loop keeps going until no processor succeeds.
Now add merge_basic_block to the processors:
# deobfuscation iteration loop
keepRun = True
runCount = 0 # iteration count
processors: list = [merge_basic_block]
while(keepRun):
runCount += 1
keepRun = False
for processor in processors:
hasChange = False
bbs, hasChange = processor(bbs)
if (hasChange):
keepRun = True
print(f"deobfuscation loop finished! total passes: {runCount}")
print(f"current basic block count: {len(bbs)}")
'''
deobfuscation loop finished! total passes: 2
current basic block count: 1934
'''
Call drawBBLinkGraph(bbs) to see the merged linkage:

Now noticeably smaller than the original graph.

With basic-block merging, the code-shuffling obfuscation built on direct unconditional jumps is defeated.
3. Proving opaque predicates
As noted, VMProtect is sneaky and uses always-true/always-false opaque predicates to strengthen code shuffling. Block merging alone can’t resolve those, because a jcc has two successors (jump or fallthrough), but the merge rule requires exactly one successor. So we must find the jcc’s real taken/not-taken fact, then rewrite its real successor as DIRECT_UNCOND or DIRECT_FALLTHROUGH and delete the fake successor.
The two forms of always-true/false opaque predicates are constant conditions and tautology/contradiction expressions. The approach: symbolize everything (all registers and memory), symbolically execute the basic block ending in a jcc, and when the symbolic engine reaches the last instruction, check whether the current path predicate has been symbolized.
If not symbolized → the first case (constant condition), directly a candidate opaque predicate.
If symbolized → possibly the second case (tautology/contradiction); then use the SMT solver to prove whether it’s really an opaque predicate, i.e. whether one of the two branch paths is unreachable. If one branch is proven unreachable, the jcc is an opaque predicate.
The proof code:
def checkOpaquePredicate(bb: ST_BasicBlock):
insList = bb.instructions
tmpctx = triton.TritonContext()
tmpctx.setArchitecture(ctx.getArchitecture())
tmpctx.setMode(triton.MODE.ALIGNED_MEMORY, True)
tmpctx.setMode(triton.MODE.AST_OPTIMIZATIONS, True)
tmpctx.setMode(triton.MODE.CONSTANT_FOLDING, True)
tmpctx.setMode(triton.MODE.SYMBOLIZE_INDEX_ROTATION, True)
tmpctx.setMode(triton.MODE.PC_TRACKING_SYMBOLIC, False)
# symbolic memory is discouraged here; it makes solving much slower
tmpastctx = tmpctx.getAstContext()
# symbolize the register environment
initTritonCtxEnvSym(tmpctx, bb.getFirstIns(), main_thread.teb)
for i, ins in enumerate(insList):
ttins = triton.Instruction()
ttins.setAddress(ins.ins_address)
ttins.setOpcode(ins.bytes)
for memAcc in ins.mem_accs:
if (memAcc.type != Supertrace.AccessType.READ):
continue
for i in range(memAcc.acc_size):
memi = triton.MemoryAccess(memAcc.acc_address + i, triton.CPUSIZE.BYTE)
if (not tmpctx.isConcreteMemoryValueDefined(memi)):
oldby = (memAcc.old_data >> (i * 8)) & 0xFF
tmpctx.setConcreteMemoryValue(memi, oldby)
tmpctx.symbolizeMemory(memi, hex(memAcc.acc_address + i))
tmpctx.processing(ttins)
if ((i == len(insList) - 1) and ttins.isBranch() and ttins.getType() != triton.OPCODE.X86.JMP): # select JCC conditional jumps
if (hasattr(ins, "obfOPStatus") and ins.obfOPStatus == True): # already proven, skip
continue
ins.obfOPStatus = False
ins.obfOPCondTaken = False
pathPredicate = tmpctx.getPathPredicate()
satCount = 0
if (pathPredicate.isSymbolized()):
if (tmpctx.isSat(pathPredicate)):
satCount += 1
if (tmpctx.isSat(tmpastctx.lnot(pathPredicate))):
satCount += 1
else:
satCount = 1
if (satCount == 1):
# it's an opaque predicate
ins.obfOPStatus = True
ins.obfOPCondTaken = (ttins.getNextAddress() != record[ins.id + 1].ins_address)
# print(f"{hex(ins.ins_address)} is an opaque predicate! taken: {ins.obfOPCondTaken} {ttins}")
def prove_opaque_predicate(bbs: set[ST_BasicBlock]) -> tuple[set[ST_BasicBlock], bool]:
hasChange = False
bbs = set(bbs)
for bb in bbs:
if (bb.isConditionalJump):
lastIns = bb.getLastIns()
if (hasattr(lastIns, "obfOPStatus") and lastIns.obfOPStatus == True):
continue # already proven, skip
checkOpaquePredicate(bb)
if (hasattr(lastIns, "obfOPStatus") and lastIns.obfOPStatus == True):
# print(f"{hex(bb.getCurrentAddr())} has an OP: {lastIns.ttins}")
# save the opaque-predicate info onto the block
bb.obfOPStatus = lastIns.obfOPStatus
bb.obfOPCondTaken = lastIns.obfOPCondTaken
hasChange = True
return bbs, hasChange
Add prove_opaque_predicate to the processors and re-run:
# deobfuscation iteration loop
keepRun = True
runCount = 0 # iteration count
processors: list = [merge_basic_block, prove_opaque_predicate]
while(keepRun):
runCount += 1
keepRun = False
for processor in processors:
hasChange = False
bbs, hasChange = processor(bbs)
if (hasChange):
keepRun = True
print(f"deobfuscation loop finished! total passes: {runCount}")
print(f"current basic block count: {len(bbs)}")
'''
deobfuscation loop finished! total passes: 7
current basic block count: 990
'''
Call drawBBLinkGraph(bbs) again:

Even cleaner. Look at the earlier example block 0x14043bf81:

Now 0x14043bf81: 88 is back to its original form with no successors.

The always-true/false opaque predicates — and the code shuffling built on them — are now defeated too.

4. Indirect-jump obfuscation
Looking closely at the linkage graph above, some blocks whose successors are linked by indirect jumps also “look” like they satisfy the merge rule. Indeed some can be merged, because they are fake indirect jumps that are really direct jumps.
VMProtect’s indirect-jump obfuscation all seems to work by tampering with the address a call pushes onto the stack. So, similar to the opaque-predicate proof: symbolize everything (registers and memory), then check whether the indirect-jump instruction has been symbolized. If not symbolized, the indirect jump is obfuscated.
The proof code:
def checkObfIndirect(bb: ST_BasicBlock):
insList = bb.instructions
tmpctx = triton.TritonContext()
tmpctx.setArchitecture(ctx.getArchitecture())
tmpctx.setMode(triton.MODE.ALIGNED_MEMORY, True)
tmpctx.setMode(triton.MODE.AST_OPTIMIZATIONS, True)
tmpctx.setMode(triton.MODE.CONSTANT_FOLDING, True)
tmpctx.setMode(triton.MODE.SYMBOLIZE_INDEX_ROTATION, True)
# symbolize the register environment
initTritonCtxEnvSym(tmpctx, bb.getFirstIns(), main_thread.teb)
for i, ins in enumerate(insList):
ttins = triton.Instruction()
ttins.setAddress(ins.ins_address)
ttins.setOpcode(ins.bytes)
for memAcc in ins.mem_accs:
if (memAcc.type != Supertrace.AccessType.READ):
continue
for i in range(memAcc.acc_size):
memi = triton.MemoryAccess(memAcc.acc_address + i, triton.CPUSIZE.BYTE)
if (not tmpctx.isConcreteMemoryValueDefined(memi)):
oldby = (memAcc.old_data >> (i * 8)) & 0xFF
tmpctx.setConcreteMemoryValue(memi, oldby)
tmpctx.symbolizeMemory(memi, hex(memAcc.acc_address + i))
tmpctx.processing(ttins)
if ((i == len(insList) - 1) and checkIndirectIns(ttins)):
if (hasattr(ins, "obfIndirectStatus") and ins.obfIndirectStatus == True): # already proven, skip
continue
ins.obfIndirectStatus = False
ins.obfIndirectAddr = 0
if (not ttins.isSymbolized()):
ins.obfIndirectStatus = True
ins.obfIndirectAddr = record[ins.id + 1].ins_address
# print(f"{hex(ins.ins_address)} is indirect-jump obfuscation! addr: {hex(ins.obfIndirectAddr)} {ttins}")
def prove_obfuse_indirect(bbs: set[ST_BasicBlock]) -> tuple[set[ST_BasicBlock], bool]:
hasChange = False
bbs = set(bbs)
for bb in bbs:
lastIns = bb.getLastIns()
if (hasattr(lastIns, "obfIndirectStatus") and lastIns.obfIndirectStatus == True):
continue # already proven, skip
checkObfIndirect(bb)
if (hasattr(lastIns, "obfIndirectStatus") and lastIns.obfIndirectStatus == True):
bb.obfIndirectStatus = lastIns.obfIndirectStatus
bb.obfIndirectAddr = lastIns.obfIndirectAddr
hasChange = True
return bbs, hasChange
Add to the processors and run:
# deobfuscation iteration loop
keepRun = True
runCount = 0 # iteration count
processors: list = [merge_basic_block, prove_opaque_predicate, prove_obfuse_indirect]
while(keepRun):
runCount += 1
keepRun = False
for processor in processors:
hasChange = False
bbs, hasChange = processor(bbs)
if (hasChange):
keepRun = True
print(f"deobfuscation loop finished! total passes: {runCount}")
print(f"current basic block count: {len(bbs)}")
'''
deobfuscation loop finished! total passes: 8
current basic block count: 763
'''
Call drawBBLinkGraph(bbs):

Indirect-jump obfuscation — and the code shuffling built on it — is defeated too.

5. Memory-operand constant hiding
This is the newest VMProtect obfuscation. By observation, the index/base register values inserted into memory operands almost always originate from immediates of earlier instructions. So, similar to the opaque-predicate proof: symbolize everything (registers and memory), then check whether the index or base register is symbolized. If not symbolized, the register’s value is known and can be substituted directly.
def checkMemOpConstHide(bb: ST_BasicBlock) -> bool:
hasChange = False
insList = bb.instructions
tmpctx = triton.TritonContext()
tmpctx.setArchitecture(ctx.getArchitecture())
tmpctx.setMode(triton.MODE.ALIGNED_MEMORY, True)
tmpctx.setMode(triton.MODE.AST_OPTIMIZATIONS, True)
tmpctx.setMode(triton.MODE.CONSTANT_FOLDING, True)
tmpctx.setMode(triton.MODE.SYMBOLIZE_INDEX_ROTATION, True)
initTritonCtxEnvSym(tmpctx, bb.getFirstIns(), main_thread.teb) # initialize the concrete environment
tmpctx.symbolizeRegister(tmpctx.registers.rsp, tmpctx.registers.rsp.getName())
for i, ins in enumerate(insList):
ttins = triton.Instruction()
ttins.setAddress(ins.ins_address)
ttins.setOpcode(ins.bytes)
tmpctx.disassembly(ttins)
ops = ttins.getOperands()
for op in ops:
if (op.getType() != triton.OPERAND.MEM):
continue
memop: triton.MemoryAccess = op
baseReg = memop.getBaseRegister()
indexReg = memop.getIndexRegister()
ins.obfMemConstHideStatus = False
ins.obfMemConstHideDetail = set() # (TTRegID, TTRegName, RealValue)
if (tmpctx.isRegisterValid(baseReg) and tmpctx.isRegisterValid(indexReg) and baseReg == indexReg and (not tmpctx.isRegisterSymbolized(baseReg))): # same register case
if (tmpctx.isRegisterValid(baseReg) and not tmpctx.isRegisterSymbolized(baseReg)):
hasChange = True
ins.obfMemConstHideStatus = True
real = tmpctx.getConcreteRegisterValue(baseReg)
ins.obfMemConstHideDetail.add( (baseReg.getId(), baseReg.getName(), real))
# print(f"{hex(ins.dbg_id)} {ins.ttins}\t\tmemory operand's {baseReg.getName()} register is constant {hex(real)}")
else:
if (tmpctx.isRegisterValid(baseReg) and not tmpctx.isRegisterSymbolized(baseReg)):
hasChange = True
ins.obfMemConstHideStatus = True
real = tmpctx.getConcreteRegisterValue(baseReg)
ins.obfMemConstHideDetail.add( (baseReg.getId(), baseReg.getName(), real))
# print(f"{hex(ins.dbg_id)} {ins.ttins}\t\tmemory operand's {baseReg.getName()} register is constant {hex(real)}")
if (tmpctx.isRegisterValid(indexReg) and not tmpctx.isRegisterSymbolized(indexReg)):
hasChange = True
ins.obfMemConstHideStatus = True
real = tmpctx.getConcreteRegisterValue(indexReg)
ins.obfMemConstHideDetail.add( (indexReg.getId(), indexReg.getName(), real))
# print(f"{hex(ins.dbg_id)} {ins.ttins}\t\tmemory operand's {indexReg.getName()} register is constant {hex(real)}")
for memAcc in ins.mem_accs:
if (memAcc.type != Supertrace.AccessType.READ):
continue
for i in range(memAcc.acc_size):
memi = triton.MemoryAccess(memAcc.acc_address + i, triton.CPUSIZE.BYTE)
if (not tmpctx.isConcreteMemoryValueDefined(memi)):
oldby = (memAcc.old_data >> (i * 8)) & 0xFF
tmpctx.setConcreteMemoryValue(memi, oldby)
tmpctx.symbolizeMemory(memi, hex(memAcc.acc_address + i))
tmpctx.buildSemantics(ttins)
return hasChange
def prove_obfuse_memopConstHide(bbs: set[ST_BasicBlock]) -> tuple[set[ST_BasicBlock], bool]:
hasChange = False
bbs = set(bbs)
for bb in bbs:
if (checkMemOpConstHide(bb)):
hasChange = True
return bbs, hasChange
Since this processor is unrelated to control-flow obfuscation, run it standalone rather than adding it to the iteration loop:
bbs, _ = prove_obfuse_memopConstHide(bbs)
After it finishes, it adds obfuscation flags (obfMemConstHideStatus and obfMemConstHideDetail) to every instruction carrying the memory-operand constant-hiding obfuscation, which helps generate the x64dbg deobfuscation script later.

Validating the deobfuscation
So far four obfuscations are defeated: code shuffling, opaque predicates, indirect-jump obfuscation, and memory-operand constant hiding. Now write an x64dbg script to patch in the deobfuscated bytes and see whether the program still runs.
First, code shuffling and indirect-jump obfuscation can’t be reduced to a script: code shuffling preserves correct execution order, so there’s nothing to undo (unless we allocate a new region and rebuild control flow in original order, which is too much work for little gain). Indirect-jump deobfuscation would mean turning an indirect jump into a direct jump, but there aren’t enough bytes (only two):

Opaque predicates can be resolved: based on the taken/not-taken fact, rewrite as nop or jmp targetAddr:
def writeBytesX64dbg(insaddr: int, bys: bytes) -> str:
result = ""
for i, by in enumerate(bys):
result += f"memset {hex(insaddr + i)},{hex(by)},0x1"
if (i != len(bys) - 1):
result += "\n"
return result
Copy the printed commands into a file:
for ins in record:
if (hasattr(ins, "obfOPStatus") and ins.obfOPStatus == True):
print(f"memset {hex(ins.ins_address)},0x90,{hex(len(ins.bytes))}")
if (ins.obfOPCondTaken):
ops = ins.ttins.getOperands()
immop: triton.Immediate = ops[0]
writestr = f"jmp {hex(immop.getValue())}"
bys = asmDecode(writestr, ins.ins_address)
print(writeBytesX64dbg(ins.ins_address, bys))

For memory-operand constant hiding, use Miasm — its structured instruction re-assembly rewrites the memory-operand registers as values (extremely convenient):
for bb in bbs:
for ins in bb.instructions:
if (hasattr(ins, "obfMemConstHideStatus") and ins.obfMemConstHideStatus == True):
mmins = mn_x86.dis(ins.bytes, 64)
rep = {}
for detail in ins.obfMemConstHideDetail:
bit = ctx.getRegister(detail[0]).getBitSize()
rep[ExprId(detail[1].upper(), bit)] = ExprInt(detail[2], bit)
for i, arg in enumerate(mmins.args):
if (arg.is_mem()):
mmins.args[i] = arg.replace_expr(rep)
try:
asm = mn_x86.asm(mmins)
ins.obfMemConstHideRep = asm
except ValueError:
print(f"{ins} error!\t{mmins}")
continue
'''
<DbgId: 102c4> ip: 1401a6eea insSize: 8 error! LEA RBX, QWORD PTR [0xFFFFFF8D * 0x2 + 0x30AA5D90]
<DbgId: 102f0> ip: 140158a35 insSize: 8 error! LEA R10, QWORD PTR [0x46AA06EA + 0x4B380F72 * 0x8 + 0x5D0D370A]
......
'''
There’s a bug here I don’t know how to fix:
leaalso uses memory operands, but VMProtect often uses it for pure arithmetic, so after deobfuscation the memory-operand value can exceed 64 bits and Miasm can’t encode it. So just ignore those.
for bb in bbs:
for ins in bb.instructions:
if (hasattr(ins, "obfMemConstHideStatus") and ins.obfMemConstHideStatus == True and hasattr(ins, "obfMemConstHideRep")):
if (not len(ins.obfMemConstHideRep) > 0):
# print("count anomaly")
continue
rep: bytes = ins.obfMemConstHideRep[0] # the first is the shortest by default
if (len(rep) > len(ins.bytes)):
# print("not enough space")
continue
print(f"memset {hex(ins.ins_address)},0x90,{hex(len(ins.bytes))}")
print(writeBytesX64dbg(ins.ins_address, rep))

Then enter 3735928559 to make it pop the message box:

The program runs correctly. Now record a post-deobfuscation trace and compare it with the original:

Remaining obfuscations
Junk code and MBA obfuscation I plan to handle later at the IR level (didn’t implement them here). The plan: junk code via live-variable analysis; MBA via SMT solving or gray-box synthesis; semantics-preserving instruction substitution needs its own collected feature set for replacement, and you must carefully judge whether the semantics are truly preserved (unless you can prove the semantic difference has no effect on execution — e.g. the replacement introduces flag writes, but the flags are overwritten later and never read before that).
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch