Colin XiaoReverse Engineering & Security Research

Analyzing and Recovering a Major Company's VM Protection

About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

Analyzing and Recovering a Major Company’s VM Protection

Preface

Why a virtual machine, again?

Someone may ask: another VM article? The truth is, most other reverse-engineering directions have already been covered extensively, so writing more of the same adds little. VM protection, by contrast, has a higher barrier to entry and correspondingly fewer writeups — so that’s the direction I chose.

VM protection remains a hurdle many people never clear. But in current practice, the VM isn’t the only path to the goal. Often you can bypass the VM entirely — using tracing and similar techniques to directly walk backward through the app’s instruction and data flow. Even when you do need to recover the VM, the engineering cost is limited, maybe only 5–10% of the total work.

Everyone’s analysis approach, tooling habits, and even their understanding of reverse engineering differ. This article just shares my personal methods and ways of thinking — they may not be optimal, and some might even look clumsy to others. But everyone has blind spots, which is exactly why we share. Feedback is welcome.

As countermeasures have escalated over the years, the big companies now almost all deploy VM protection, which has sent analysis workloads through the roof. One person against a company’s multiple security teams is no longer a fair fight; going it alone is a thing of the past.

Reverse engineering has never been easy. A lot of the time the process is more like a mental endurance exercise. But over time you adapt. Reverse engineering spans an enormous range: experience, breadth of knowledge, comprehension, even talent. I can’t claim much advantage in any of those, and sometimes I get stuck in dead ends. But that’s the appeal — there’s always a new challenge, always room to grow.

Environment

  • App version: versionCode='1200280405' versionName='12.28.405'
  • Phone: Android 11 / Pixel 2 XL
  • PC: macOS 15.3

Analysis

Code obfuscation and out-of-order layout

Take the vm_interpreter function as an example. Its start and end boundaries are clear, and its code doesn’t mingle with other functions — meaning any obfuscation or reordering happens only within this function.

  • Function start

  • Function end

Basic-block splitting

During analysis I found that code blocks (made up of several consecutive basic blocks) are linked at their ends with unconditional jumps.

In normally compiled code, unconditional jumps appear at structured control-flow points: if-then-end, if-else-end, break, continue, switch-case, switch-end, loop entry, loop body, loop exit, and so on. In the code under analysis here, the number of unconditional jumps is clearly abnormal.

Duplicated dispatcher blocks

The app’s protection author clearly understands reverse engineering. Normally, when a handler finishes, execution returns to the fetch block. Here, the fetch/dispatch block has been duplicated and attached to the tail of each handler as its terminator, so that a breakpoint or hook at a single location can’t grab the opcode data.

That said, it’s also possible this is just function inlining producing multiple dispatch sites.

Indirect jumps

Removing indirect jumps makes the code structure easier to view and analyze in IDA. This usually means reading the constant array in the jump table, resolving the actual jump addresses, and building correct control-flow links to the corresponding code blocks.

import idautils
import idc
import idaapi
from keystone import *  # pip3 install keystone-engine
def get_insn_const(addr):
    op_val = None
    if idc.print_insn_mnem(addr) in ['MOV', 'LDR']:
        op_val = idc.get_operand_value(addr, 1)
        if op_val > 0x1000:  # possibly an indirect reference
            op_val = idc.get_wide_dword(op_val)
    else:
        raise Exception(f"error ops const: {addr}")
    return op_val
def get_patch_data(addr):
    addr_list = []
    for bl_insn_addr in idautils.XrefsTo(addr):
        bl_insn_addr = bl_insn_addr.frm
        # print(f'L1 {hex(bl_insn_addr)}:')
        for xref_addr_l2 in idautils.XrefsTo(bl_insn_addr):
            # print(f'\tL2 {hex(xref_addr_l2.frm)}:')
            index = get_insn_const(xref_addr_l2.frm - 4)
            const_table_start = bl_insn_addr + 4
            offset = idaapi.get_dword(const_table_start + index * 4)
            link_target = const_table_start + offset
            addr_list.append({"bl_insn_addr": bl_insn_addr, "patch_addr": xref_addr_l2.frm, "index": index,
                        "offset": offset, "link_target": link_target})
    return addr_list
def print_patch_data(patch_data):
    for item in patch_data:
        print(
            f"bl_insn_addr: {item["bl_insn_addr"]:#x}, patch_addr: {item["patch_addr"]:#x}, index: {item["index"]}, offset: {item["offset"]:#x}, link_target: {item["link_target"]:#x}")
def patch_insns(patch_data):
    index = 0
    for item in patch_data:
        ks = Ks(KS_ARCH_ARM64, KS_MODE_LITTLE_ENDIAN)
        asm = f'B {item["link_target"]:#x}'
        print(f'patch addr {item["patch_addr"]:#x}: {asm}')
        encoding, count = ks.asm(asm, as_bytes=True, addr=item["patch_addr"])
        print(encoding)
        for i in range(4):
            idc.patch_byte(item["patch_addr"] + i, encoding[i])
        index += 1
        # if index == 1:
        #     break
def start():
    # .text:0000000000025D00 ; __unwind {
    # .text:0000000000025D00                 STP             X0, X1, [SP,#var_10]!
    # .text:0000000000025D04                 LDR             W0, [X30,W0,UXTW#2]
    # .text:0000000000025D08                 ADD             X30, X30, W0,UXTW
    # .text:0000000000025D0C                 LDP             X0, X1, [SP+0x10+var_10],#0x10
    # .text:0000000000025D10                 RET
    modify_x30_func_address = 0x25D00
    patch_data = get_patch_data(modify_x30_func_address)
    print_patch_data(patch_data)
    patch_insns(patch_data)
start()

Call graph

Locating the VM quickly

After applying the indirect-jump removal patch and reloading the analysis file, wait for IDA Pro to finish its auto-analysis, open the Functions window, and click the Length column header to sort by function size. At this point, the second-largest function is usually where the VM interpreter lives.

  • vm_interpreter

    vm_interpreter (start 0x138518) is the core function responsible for bytecode interpretation and execution. It covers the full three-stage cycle: fetch, decode, and dispatch to the corresponding handler.

    Judging from the control-flow graph, the structure is very complex. There are many unrecognized switch statements and indirect jumps, so they don’t render correctly in the graph. The overall complexity far exceeds expectations.

  • vm_ready

    Double-click vm_interpreter (address 0x138518) to jump to its start; through cross-references you can trace upward to its caller — vm_read.

  • vm_entry

    Locate vm_read (start 0x131680), then use cross-references again to find the next caller up: vm_entry.

Function analysis

The VM entry point (vm_entry) starts at 0x1313F0. At that address there are 31 references, marking the entries of 31 functions that will be virtualized. Reference #2 points at the first VM code executed after app startup — that’s where I’ll begin.

Double-click cross-reference #2 to land on function 0x6884C. In its prototype, x0–x2 hold the first three parameters: bytecode, bc_size, and external. The remaining parameters are variadic, possibly passed through x3–x7, Q0–Q7, and the stack.

void vm_entry(void bytecode, int bc_size, void* external, …)

Description: prepares parameters and the return-value object.

  • bytecode: bytecode pointer
  • bc_size: bytecode size
  • external: external pointer array, possibly containing external function addresses and global-variable pointers.

At function start, all registers used for variadic parameters are pushed onto the stack, then memory is allocated for the return-value object. Next, the variadic parameters are converted into a pointer array pVA. Finally, after vm_ready returns, the return value is fetched from the return-value object and assigned to the real register x0.

int vm_ready(void *ret_val, void *pBytecode, int bcSize, void **external, void **pVA)

Description: builds/prepares the VMPState object corresponding to the bytecode — one VMPState per bytecode.

  • pRetVal: return value
  • regCount: virtual register count
  • pRegister: virtual registers
  • typeCount: type-table data
  • pTypeList: type table
  • insCount: VM instruction count
  • pInstructions: VM instructions
  • pBranchs: branch table

The parser is complex, so this article focuses on the simpler handler analysis. Throughout, keep the locations and meanings of these five pieces of data clear: VM PC, instruction object, context object, type-table object — and, when you hit branch instructions, the branch-table object. If you lose track of these, you’ll get lost quickly.

getVMPObject

Returns the global VM cache object — a std::list that caches every VMPState object.

getVMPState

Given pBytecode, tries to find the corresponding VMPState in the cache. If it isn’t cached, it jumps to start_build to parse the bytecode and produce a new VMPState (see the bytecode-first-decode section). Otherwise (cache hit), it starts creating the context object.

Creating a context object

Initializing registers

All registers are zeroed.

Copying register data

During the first decode, some registers are assigned initial values. These values come from unencrypted constants in the original assembly code; because they’re immutable, a register copy is created.

Copying input parameters into VM registers

If no input parameters are detected, run vm_interpreter; otherwise, load the variadic parameters into the virtual register sequence in order (starting from V0).

The VMPState->entryFunction member holds the pre-encryption function info, including argument count, argument types, and return type.

Preparing execution

Before entering the interpreter, the system passes the VMPState data members as arguments to prepare for executing the VM interpreter.

int vm_interpreter()

  • void *pRetVal: return value
  • int regCount: virtual register count
  • void *pRegister: VM registers
  • int typeCount: type-table count
  • void **pTypeList: type table
  • int insCount: VM instruction count
  • int16_t **pInstructions: VM instructions
  • int16_t **pBranchs: branch table

First read of the primary opcode

After entering the interpreter, the dispatch logic here executes only once. Because the dispatch block was duplicated into the tails of 13 handlers, each handler owns an independent primary-opcode dispatcher. When a handler completes, it reads the next VM instruction’s primary opcode and jumps to the matching next handler. Details are in the duplicated-dispatch-block section above.

Objects to watch during dispatch:

  • VM PC: 0x0
  • Instruction object: X24

When reading the primary operand, two key objects matter: the instruction-object pointer and the PC. At the very start, PC is 0, so you only need to watch the instruction object held in x24. Note that in some primary-opcode dispatchers the instruction object may live in a different general-purpose register rather than x24.

13A5C0 MOV W19, #1 After reading the primary opcode at relative offset 0, that byte is usually no longer used. So the program points at the next byte of the current instruction, i.e., offset 1. Most primary-opcode dispatchers do this, so after entering a handler, bytecode parsing typically starts from offset 1. Hint: in most dispatchers, W19 points at the offset-1 position.

Primary-opcode switch dispatch table

The switch table contains 13 real handlers. The unnamed loc_-prefixed dark-blue entries are decoy addresses used as padding — they don’t correspond to any actual handler implementation. They exist to interfere with analysis and add confusion during disassembly.

The diagram below shows the full instruction set the VM supports, plus whether each instruction has a second opcode. If an instruction has a second opcode, it usually means the instruction triggers an extra sub-dispatch switch to further resolve its specific behavior.

Instruction-set analysis

This section only walks through a few basic instructions, to give a feel for the VM instruction structure and execution logic. The more complex instructions are left out to keep the article coherent.

MOV

MOV format (6): [opcode, op2, dtype, stype, sreg, dreg]

  • opcode: primary opcode
  • op2: second opcode
  • dtype: destination operand type
  • stype: source operand type
  • sreg: source register
  • dreg: destination register

Assembly syntax:

MOV dreg, sreg

Sometimes MOV isn’t just a copy — it may implicitly zero-extend, sign-extend, or truncate. That diversity of behavior is exactly why op2 exists as a second opcode to further specify the data handling.

Note: in the IDA Pro comments, op and pBytecode refer to the same pointer variable — they’re equivalent; I was too lazy to go back and rename every comment.

When analyzing the VM execution flow, the key objects rank roughly in this order of importance: VM PC, instruction object, context object, type-table object — and the branch-table object (only relevant when hitting branch instructions).

  • VM PC (current instruction offset op/pInsn): the current MOV came from one of the 13 handlers; its tail primary-opcode dispatcher already fixed up the PC pointer. So original offset 0 is unused and PC now points at offset 1; W19 was updated in the main dispatcher to the correct current-instruction offset, so from here on you only need to watch W19.
  • Instruction object (pReg): pointed at by real register x24, it’s the current instruction’s storage.
  • Predecessor node: w25=0.

x24 points at the start of pInstructions, and w19 points at offset 1 in the current instruction bytecode.

Address 0x139400: extracts the second opcode op2 into w8 and validates it.

ADD W9, W19, #1 ADD W10, W19, #2 ADD W11, W19, #3 ADD W12, W19, #4 Compute the current instruction’s byte offsets, then read the bytes at offsets 2, 3, 4, 5 in order.

  • w19 = current instruction offset op
  • x24 = instruction object: a 16-bit array, indexed by the offset computed from w19
  • x21 = type object (pType): an array of Type pointers, accessed by index
  • x28 = register object (pReg): an array of Register[]; different VMPStates hold different element counts. Each element is 0x18 bytes (the Register size), accessed as base + index * 0x18.

When execution reaches 0x13C49C, op2 dispatch begins; the data state is:

  • destination operand type: x2 = pType[op[2]] — read the index at offset 2, then use it to fetch the type pointer
  • source operand type: x8 = pType[op[3]] — read the index at offset 3
  • source register: x26 = pReg[op[4]] — read the index at offset 4
  • destination register: x20 = pReg[op[5]] — read the index at offset 5

0x13C48C ADD W22, W19, #5 next PC: at this point W19 points at offset 1 of the current instruction; adding 5 makes W22 point at the next instruction’s start (offset 0). Hence the current instruction is 6 bytes long.

When executing, the source register’s value is fetched and stored into the destination register.

0x13C4BC: the entry to the primary-opcode dispatch logic appended to the MOV handler’s tail.

MOV handler’s primary-opcode dispatcher:

x24 points at the instruction object; index w22 (new PC) fetches the next instruction’s primary opcode.

At this point W19 points at the previous instruction’s offset 1 and that instruction is 6 bytes long; after ADD W19, W19, #6, W19 points exactly at the next instruction’s offset 1. Note the +6 is because this dispatcher is MOV-specific — only after a MOV completes does execution reach this point.

CMP

CMP format (6): [opcode, type, sreg1, sreg2, creg, op2]

  • opcode: primary opcode
  • type: operand type of reg1/reg2
  • sreg1: source register 1
  • sreg2: source register 2
  • creg: destination condition-code register
  • op2: second opcode
  • Note: CMP usually appears paired with jcc, or with csel.

Assembly syntax:

CMP.EQ creg, sreg1, sreg2

Before starting, restate the five key objects (ranked by importance):

  • VM PC

  • instruction object

  • context object

  • type-table object

  • branch-table object (only for branch instructions)

  • VM PC: the current CMP came from one of the 13 handlers’ tail dispatch logic. The instruction is now dispatched to the current position, so offset 0 is unused. In the previous dispatcher, W19 was updated to offset 1, so only W19’s changes matter from here on.

  • Instruction object: real register x24 points at it.

W19 points at offset 1 of the current instruction; adding 4 makes W8 point at offset 5. Per this instruction’s format, offset 5 is operand op2.

ADD W8, W19, #4
LDR W8, [X24, W8, UXTW#2]

To preserve the predecessor basic block pointer, W25 is backed up to W26. W25 points at the predecessor basic block; later, when executing a PHI instruction, this is used to access the matching predecessor-path data.

MOV W26, W25

Check whether op2 is valid:

CMP W8, #0x28

Next, dispatch the second operand, which specifies the comparison condition type (EQ, NE, GT, GE, LT, LE, etc.).

  • Context object: X28
  • Type-table object: X21

Fetch the operand type type:

LDR W23, [X24, W19, UXTW#2]
LDR X25, [X21, X23, LSL#3]

Fetch the first source register sreg1’s pointer: pRegsBase + index * 0x18.

ADD W9, W19, #1
MOV W12, #0x18
LDR W9, [X24, W9, UXTW#2]
MADD X22, X9, X12, X28

Fetch the second source register sreg2’s pointer:

ADD W10, W19, #2
LDR W10, [X24, W10, UXTW#2]
MADD X9, X11, X12, X28

Fetch the destination condition-code register creg’s pointer:

ADD W11, W19, #3
LDR W11, [X24, W11, UXTW#2]
MADD X9, X11, X12, X28

When execution reaches 0x13BB70, all operands are extracted, and the type object and instruction object no longer participate. Only type, sreg1, sreg2, and creg matter now.

  • type: X25
  • sreg1: X22
  • sreg2: X27
  • creg: X9

The second operand specifies the CMP condition-code type, supporting several comparison modes including floating-point. This analysis uses CMP.EQ.

First, determine the comparison operand size per the instruction, then load the operand value from the sreg1 register pointer (4-byte operand in this example).

Fetch sreg1’s value — using LDRSW, which loads 4 bytes from memory:

  • sreg1_val: X20

Fetch sreg2’s operand size and value:

  • sreg2_val: X8

Compare sreg1 and sreg2 to get the EQ condition code.

Update the next instruction’s PC and check it against maxPC. W19 points at offset 1; after ADD W8, W19, #5, W8 points at the next instruction’s offset 0, confirming the current instruction (CMP) is 6 bytes long.

CMP tail primary-opcode dispatcher: read the primary opcode, update W19 to offset 1, then execute BR X8 to jump to the next handler.

LDR

LDR format (4): [opcode, type, mreg, dreg]

  • opcode: primary opcode
  • type: operand type
  • mreg: source register memory (no immediate offset)
  • dreg: destination register
  • Note: LDR dreg, [mreg] — memory to register.

Assembly syntax:

LDR dreg, [mreg]

Key objects: VM PC, instruction object, context object, type-table object, branch-table object (when hitting branch instructions).

  • VM PC: W19
  • Instruction object: X24
  • Context object: X28
  • Type-table object: X21

Fetch the operand type type:

0x13A1AC LDR W10, [X24, W19, UXTW#2]
0x13A1C0 LDR X1, [X21, X10, LSL#3]

Fetch the destination register dreg:

0x13A1B0 ADD W8, W19, #1
0x13A1B8 LDR W8, [X24, W8, UXTW#2]
0x13A1C4 MOV W10, #0x18
0x13A1C8 MADD X2, X8, X10, X28

LDR_With_Type analysis:

Prototype: void LDR_With_Type(void *dreg, void *type, void *mReg);

Args: x0=dreg, x1=type, x2=mreg. Checks whether the operand type is a char* pointer type — if so, copies the pointer; otherwise, loads the value from the mreg address into dreg.

Get the type length:

0x13FF60 BLR X8

Type length must be ≤ 8 bytes:

0x13FF68 CMP W8, #7

Get the pointer in mreg:

LDR X8, [X20]

Load the value according to the type length (1, 2, 4, 8 bytes):

0x13FF88 LDRSB W8, [X8]
0x13FFA4 LDRSH W8, [X8]
0x13FFB0 LDR W8, [X8]
0x13FF94 LDR X8, [X8]

Finally, assign to dreg:

0x13FF98 STR X8, [X19]

or

0x13FFB4 STR W8, [X19]

Bytecode first decode

The purpose of decoding is to deserialize the bytecode into a VMPState object.

VBR encoding

The bytecode uses Variable Bitrate (VBR) encoding, a format widely used in audio/video and other domains. The VM bytecode uses a 6-bit encoding — similar to protobuf’s Varint, but protobuf uses 8-bit units while this VM bytecode uses 6-bit units.

The VM’s 6-bit VBR encoding: the 6-bit binary encoding of the number 5 is read least-significant-bit first, with the high bit as the continuation flag (0 = no more bytes). Decoding yields 5:

000101

In 6-bit encoding, a high bit of 1 means more data follows:

100101 001111

When decoding combined data, first take the low 5 bits of the first byte (00101); bit 5 is 1, so more follows. Then take the next byte’s 6 bits (001111); its bit 5 is 0, so no more. The second group’s low 5 bits are 01111. The final combined value is 0x1E5 (i.e., 0b111100101). Note the later 6-bit group goes in the higher position:

01111 00101 ---> 111100101

If the current byte’s high bit is 1, keep taking the next 6-bit group; repeat until a byte with high bit 0 is found. Here’s a decode script (default 6 bits):

class BytecodeDecoder():
    def __init__(self, bytecode: bytes, extern_address: list, filename):
        self.extern_address = extern_address
        self.bytecode_bits = bitarray(endian='little')
        self.bytecode_bits.frombytes(bytecode)
        self.bytecode = bytecode
        self.bit_index = 0
        self.filename = filename

    def decode(self, nBit=6):
        num = 0
        index = 0
        bit_num = 0
        exit = False
        while index + nBit <= len(self.bytecode_bits):
            # read nBit bits
            chunk = self.bytecode_bits[self.bit_index:self.bit_index + nBit]
            high_bit = chunk[-1]  # the high (continuation) bit
            low_bits = ba2int(chunk[:-1])  # low 5 bits as integer
            if high_bit == 1:
                num = num | (low_bits << bit_num)  # shift left and merge
            else:
                num = num | (low_bits << bit_num)
                exit = True
            self.bit_index += nBit  # advance the index
            if exit:
                return num
            bit_num += 5
        raise Exception("bytecode decode error")

Bytecode decode flow

Decode-flow bookmarks

In IDA Pro, press Ctrl+M to open bookmarks (or View → Open subviews → Bookmarks). [step<n>] marks the bytecode decode flow, helping you jump to the relevant decode locations.

Reading the first bytecode

Bytecode decoding processes data in 64-bit units, while VBR encoding processes data in 6-byte units. If the current 64-bit block has fewer than 6 bytes left, the decoder reads the next 64-bit block and extracts bytes from the low end until it has 6 bytes. Meanwhile, the stack maintains the current VBR decode state: the bytecode pointer pBytecode, the remaining 64-bit data remain_bytecode, and the remaining bit count remain_bit.

Read 64 bits from bytecode:

  1. Compute the register count

    At the start of the bytecode the register count is stored. Decoding reads this count to determine how many registers to create, then builds the corresponding context object.

    Initialize and fill data:

    Decode script:

    def decode_register_count(self):
  regs_count = self.decode()
  return regs_count

  2. Decode the registers to initialize

    • Count of registers to initialize — a heap block is allocated to store these register indices.

    • Decode the register table to initialize. After the register count, the VBR stream stores the register table for setting up the registers.

      Decode script:

    def decode_register_initial_value(self):
  regs_initial_count = self.decode()
  regs_initial = []
  for i in range(0, regs_initial_count):
    reg_num = self.decode()
    # init_regs_num.append((insn, f'{insn:#x}'))
    regs_initial.append(reg_num)
    return regs_initial

  3. Set the external-address list into registers

    Note that the external-address register index data here is independent of the initial register data from step 2 — they don’t share data.

    Decode script:

    def set_registers_extern_address(self,
                                 registers: list[Register],
                                 extern_address: list[int]):
  extern: ExternInstructions = ExternInstructions()
  addr_list_count = self.decode()
  for i in range(0, addr_list_count):
    reg_idx = self.decode()  # the register index to initialize
    addr_list_idx = self.decode()  # the address-list index
    addr = self.read_extern_address(extern_address, addr_list_idx)
    registers[reg_idx].value = addr
    extern.targetRegs.append(reg_idx)
    extern.externalAddress.append(addr)
    # print(
    #     f"extern register[{reg_idx}] = {addr:#x}")
    return extern

  4. Decode the type-object table

    All type data used during VM interpretation comes from this table.

    Decode script:

    def decode_types(self):
        # 4. decode the type table
        types_count = self.decode()
        types = [None] * types_count
        for i in range(0, types_count):
            type = self.decode()
            # print(f'type[{i}]: {type:#x}')
            match(type):
                case 0x3 | 0x10 | 0x12:
                    raise Exception("error")
                case 0x5 | 0xc | 0x13:
                    struct_0xC = 0
                    struct_0xD = self.decode()
                    struct_0xE = 1
                    member_count = self.decode()
                    # 4.1 set the struct member types
                    members = []
                    for j in range(0, member_count):
                        member_type_index = self.decode()
                        t_membetr = types[member_type_index]
                        if t_membetr is None:
                            types[member_type_index] = t_membetr = StructType(0, [], "")
                        members.append(t_membetr)
                    # 4.2 get the struct type name
                    type_name = []
                    type_name_size = self.decode()
                    for j in range(0, type_name_size):
                        c = self.decode()
                        type_name.append(c)

                    name = "".join(chr(c) for c in type_name)
                    struct_type = StructType(member_count, members, name)
                    struct_type.init()
                    types[i] = struct_type
                case 0x1:
                    types[i] = VMPType()
                case 0x6:
                    nbit = self.decode()
                    types[i] = IntegerType(nbit)
                case 0x9:
                    element_count = self.decode()
                    element_type_idx = self.decode()
                    element_type = types[element_type_idx]
                    if element_type is None:
                        # array element must be a struct type...
                        raise Exception("error")
                    types[i] = ArrayType(element_count, element_type)
                case 0x7:
                    ptr_type_index = self.decode()
                    ptr_type = types[ptr_type_index]
                    if ptr_type is None:
                        ptr_type = StructType(0, [], "")
                        types[ptr_type_index] = ptr_type
                    types[i] = PointerType(ptr_type)
                case 0xb:
                    types[i] = FloatType()
                case 0x14:
                    flag = self.decode()
                    return_value_type = self.decode()
                    argument_count = self.decode()

                    return_value = types[return_value_type]
                    arguments = []
                    if return_value is None:
                        return_value = None
                        raise Exception("return value type error")
                    for j in range(0, argument_count):
                        arg_type_idx = self.decode()
                        arg_type = types[arg_type_idx]
                        if arg_type is not None:
                            arguments.append(arg_type)
                        else:
                            # build the struct type
                            raise Exception("error")

                    types[i] = FunctionType(
                        return_value, argument_count, arguments, flag)
                case 0x15:
                    types[i] = DoubleType()
                case _:
                    input(f"unknown type: {type:#x}\n")
        return types

  5. Set initial register values

    The register list here comes from step 2. Besides setting initial values, this also handles other instruction operations. Note the target registers are static/read-only, so the interpreter doesn’t modify their values during execution.

    Decode script:

        def set_registers_inial_value(self, registers: list[Register], types: list[int],
                                regs_initial: list[int]):
        init_instructions: InialInstructions = InialInstructions()
        count = self.decode()
        for i in range(0, count):
            init_type = self.decode()
            reg_idx = regs_initial[i]
            # print(f'init reg type: {init_type:#x}')
            # record the init instruction
            init_instructions.opcodes.append(init_type)
            init_instructions.dRegs.append(reg_idx)
            match(init_type):   # ADD.MO
                case 0 | 0xb | 0x17:
                    type_idx = self.decode()
                    deep = self.decode()
                    breg = self.decode()
                    member_offset_table = []
                    for j in range(1, deep):
                        member_offset_table.append(self.decode())
                    # record the init instruction
                    init_instructions.imm.append(member_offset_table)
                    init_instructions.type.append(type_idx)
                    init_instructions.sRegs.append(breg)
                case 7: # MOV REG, IMM
                    imm = self.decode()
                    registers[reg_idx].value = imm
                    # record the init instruction
                    init_instructions.imm.append(imm)
                    init_instructions.type.append(None)
                    init_instructions.sRegs.append(None)
                    # print(f'init register[{reg_idx}] = {imm:#x}')
                case 8:# MOV REG, ???
                    type_index = self.decode()
                    t = types[type_index]
                    if t.tag != 0xe:
                        registers[reg_idx].value = 0
                    else:
                        raise Exception("error")
                    # record the init instruction
                    init_instructions.type.append(type_index)
                    init_instructions.imm.append(None)
                    init_instructions.sRegs.append(None)
                case 0x15:  # MOV REG.T, REG@FuncPtr
                    while self.decode() & 0x20:
                        pass
                    type_idx = self.decode()
                    t = types[type_idx]
                    sreg = self.decode()
                    registers[reg_idx].value = registers[sreg].value
                    registers[reg_idx].type = t
                    # record the init instruction
                    init_instructions.type.append(type_idx)
                    init_instructions.imm.append(None)
                    init_instructions.sRegs.append(sreg)
                case _:
                    input(f"unknown init register type: {init_type:#x}\n")
        return init_instructions

  6. Get the entry function object

    Fetch the entry function’s type info from step 4’s type-object table. The entry function is the native (non-virtualized) function info.

    Decode script:

    def get_function_type(self, types: list[int]):
  func_type_idx = self.decode()
  return types[func_type_idx].type

  7. Decode VM instructions

    The instructions executed by the VM interpreter control the VM’s operation and bytecode processing — arithmetic, jumps, data transfer, etc.

    Decode script:

    def decode_bytecode(self):
  vm_insn_count = self.decode()
  vm_instructions = []
  for i in range(0, vm_insn_count):
    insn = self.decode()
    vm_instructions.append(insn)
    return vm_instructions

  8. Create the branch table

    The branch table stores jump target addresses. Instructions like BSEL.PHI, J, JCC, and SWITCH fetch target addresses from this table.

    Decode script:

    def decode_branches(self):
  branches_count = self.decode()
  branches = []
  for i in range(0, branches_count):
    offset = self.decode()
    branches.append(offset)
    return branches

Creating the VMPState object

After decoding the bytecode, we have the entry function type, virtual register count, type-table count, instruction count, context object, type-table object, instruction-table object, and branch-table data — enough to build the VMPState object.

After decoding, all decoded data is stored in the VMPState object, which is then added to the global cache so the next execution of the same bytecode avoids rebuilding.

VM data structures

These structures matter a great deal during reverse analysis and VM execution. Understanding them keeps you from getting lost — especially during interpretation, where you constantly access the instruction, type, register, and branch structures.

VM state object

struct VMPState {
  0x0:  Type* entryFunction;
  0x8:  int regCount;
  0xC:  int typeCount;
  0x10: int insCount;
  0x18  Context* context;
  0x20: Type** pTypeList;
  0x28: int16_t** pInstructions;
  0x30: Branches* pBranches;
}

Type objects

Type objects describe type data; they don’t carry the type’s value.

  • Type tag

    enum TypeTag {
  VoidType = 0,
  FloatType = 2,
  DoubleType = 3,
  InteterType = 0xb,
  FunctionType = 0xc,
  StructType = 0xd,
  ArrayType = 0xe,
  PointerType = 0xf,
  VertorType = 0x10
}

  • Base class of all types — memory size 0x10:

    struct Type {
+0x0:    void* vtable;
+0x8:    TypeTag tag;
+0xc:    union {
              int nbit;    // used by IntegerType
                              bool field1;
          }
+0xd:    bool isInit;  // used by StructType: whether the struct memory size was computed
+0xe:    bool field3;
}

  • Void type — size 0x10:

    struct VoidType : public Type {
}

  • Floating-point types — size 0x10; the tag distinguishes float (32-bit) from double (64-bit):

    struct FloatType : public Type {
}
    struct DoubleType : public Type {
}

  • Integer — size 0x10; nbit describes 1/8/16/32/64 bits:

    struct IntegerType : public Type {
}

  • Function type — size 0x28:

    struct FunctionType : public Type {
+0x10:  Type*   returnValueType;
+0x18:  int.       argumentCount;
+0x20: type**   arguments;
}

  • Struct type — size 0x38:

    struct StructType : public Type {
  0x10: int32   memberCount;
  0x18: type**  members;
  0x20: int32   memorySize;
  0x28: int32*  memberOffsetTable;
  0x30: char*  typeName;
}

  • Array type — size 0x18:

    struct ArrayType{
  0xc: int32   count;
  0x10: Type*  element;
}

  • Pointer type — size 0x18:

    struct PointerType{
  0x10: Type*  pointee;
}

Context object

Register — size 0x18:

    struct Register {
          0x0: int64 value;
          0x8: Type* t;
      0x10: bool isBuffer; // whether value was allocated by a memory function: cmalloc/malloc
  }

Context — size depends on count: size = count * sizeof(Register):

struct Context {
      0x0: int count;
    0x8: Register regs[];
}

Branch table

Size varies; the length is stored during first decode (see decode-flow step 8):

struct Branchs {
0x0: int16_t br[];
}

Instruction object

Size varies; VMPState->insCount holds the instruction count (see decode-flow step 1):

struct Instructions {
0x0: int16_t ins[];
}

Instruction decode format

Note: on first decode, the system builds the core VMPState structure that stores bytecode basic info. The second decode happens during VM execution, for instruction decode and execution.

The outermost [] is like a Python list; inner [] means optional; () usually means paired data.

Instruction opcode length format note
ARITH 0x1 6 [opcode, op2, type, sreg1, sreg2, dreg]
MOV 0x2 6 [opcode, op2, dtype, stype, sreg, dreg]
CALLOC 0x6 5 [opcode, a2_type, a1_type, a1_sreg, dreg]
STR 0xa 4 [opcode, stype, sreg, mreg]
CMP 0xc 6 [opcode, type, sreg1, sreg2, creg, op2]
PHI 0xe var (3+) [opcode, dreg, count, [(reg, branch), ...]]
CALL 0xf var (5+) [opcode, ftype, [count], tag, [retreg], ereg, none, [areg, ...]]
RET 0x13 var (2+) [opcode, type, [retreg]]
SWITCH 0x16 var (5+) [opcode, treg, type, defbranch, count, [(casereg, branch), ...]]
J/JCC 0x1d 3 or 6 [opcode, brtype, tbranch, [creg, ctype, fbranch]]
CSEL 0x1e 7 [opcode, ntype, creg, type, treg, freg, dreg]
GEP 0x28 var (5+) [opcode, dreg, none, type, count, breg, [index, ...]]
LDR 0x2a 4 [opcode, type, mreg, dreg]

Recovery

VM recovery splits into two parts. Recovering to assembly is usually enough to understand the logic; recovering to native code is recovery taken to the extreme. Recovering assembly directly is closest to the VM instructions — easiest to understand, least error-prone, and the true mapping of the VM instructions.

Recovering to assembly

Decoded bytecode is still a pile of data. To understand it, convert it to assembly. If the VM instructions resemble a native architecture, you can borrow that architecture’s assembly. For a VM with its own instruction set, you generally define an assembly language matching the VM’s semantics. The key is converting bytecode into semantically close assembly. The assembly here borrows syntax from MIPS, RISC-V, Smali, and Binary Ninja’s intermediate language.

  • Operand widths — general and floating register counts vary:

    Register 1-bit 8-bit 16-bit 32-bit 64-bit
    General V0.b V1.b V2.h V3.d V4
    Float H0 (half) S0 (single) D0 (double)

  • Operand identifier notes:

    Identifier Meaning Note
    op primary opcode
    op2 second opcode
    dreg destination register
    sreg source register naming when there’s a single source
    sreg1 first source register
    sreg2 second source register
    type operand type operand width and data type
    stype source operand type
    dtype destination operand type
    a1_type first argument type CALLOC only
    a2_type second argument type CALLOC only
    a1_sreg first argument register CALLOC only
    mreg memory operand register STR/LDR
    creg condition-code register CMP
    count element count PHI [val, lab] count; GEP [index] count; SWITCH case count
    branch branch the basic-block start label
    ftype function type CALL only; describes return type, arg count, arg types
    retreg return-value register CALL only
    ereg call-target register CALL only; holds the target address, e.g. call 0x1234, call reg
    areg argument register CALL only
    treg comparison target register SWITCH only
    defbranch default branch block label SWITCH only
    casereg SWITCH case constant SWITCH only; holds the case immediate
    brtype branch instruction type 0 = no jump, 1 = conditional jump
    tbranch true branch J/JCC
    fbranch false branch JCC only
    freg false register CSEL only
    breg base register GEP only
    index element index register GEP only; holds the member access value

  • Instruction set

  1. Arithmetic:

    Mnemonic Syntax Operand format Note
    XOR XOR V0, V1, V2 XOR dreg, sreg1, sreg2
    SUB SUB V0.b, V1.b, V2.b SUB dreg, sreg1, sreg2
    UDIV UDIV V0.h, V1.h, V2.h UDIV dreg, sreg1, sreg2
    ADD ADD V0.d, V1.d, V2.d ADD dreg, sreg1, sreg2
    OR OR V0, V1, V2 OR dreg, sreg1, sreg2
    SMOD SMOD V0.b, V1.b, V2.b SMOD dreg, sreg1, sreg2
    SDIV SDIV V0.h, V1.h, V2.h SDIV dreg, sreg1, sreg2
    UMOD UMOD V0.d, V1.d, V2.d UMOD dreg, sreg1, sreg2
    ASR ASR V0, V1, V2 ASR dreg, sreg1, sreg2
    LSL LSL V0.b, V1.b, V2.b LSL dreg, sreg1, sreg2

  2. MOV:

    Mnemonic Syntax Operand format Note
    MOV MOV V0, V1 MOV dreg, sreg data move
    MOV.T MOV.T V0, V1.b MOV.T dreg, sreg truncate V1 to 8 bits, move to V0
    MOV.Z MOV.Z V0, V1.d MOV.Z dreg, sreg zero-extend 32-bit V1.d to 64-bit V0
    MOV.S MOV.S V0, V1.d MOV.S dreg, sreg sign-extend 32-bit V1.d to 64-bit V0

  3. MOV auxiliary ops:

    Operator Description
    T truncate
    Z zero-extend
    S sign-extend

  4. CALLOC — heap allocation, like libc calloc:

    Mnemonic Syntax Operand format Note
    CALLOC CALLOC (0x1, 0x20), V21 CALLOC (num, size), RetVal same as libc calloc

  5. Memory access:

    Mnemonic Syntax Operand format Note
    STR STR V0.b, [V3] STR dreg, [mreg] same as ARM
    LDR LDR V5, [V80] LDR dreg, [mreg] same as ARM

  6. Comparison (FCMP is rarely used, omitted):

    Mnemonic Syntax Operand format Note
    CMP.EQ CMP.EQ V3.b, V0, V3 CMP.EQ creg, sreg1, sreg2 store EQ result into creg
    CMP.NE CMP.NE V3.b, V0, V3 CMP.NE creg, sreg1, sreg2 store NE result into creg
    CMP.CC CMP.CC V3.b, V0, V3 CMP.CC creg, sreg1, sreg2 store CC result into creg
    CMP.LT CMP.LT V3.b, V0, V3 CMP.LT creg, sreg1, sreg2 store LT result into creg

  7. PHI — checks which predecessor branch the current instruction came from (via BranchIndex), then assigns the matching branch’s register value to dreg. Often represents the loop start (for-loop init / increment block). See LLVM IR’s phi.

    Mnemonic Syntax Operand format Note
    BSEL.PHI BSEL.PHI V8, [(V0, #0x3), ...] BSEL.PHI dreg, [(reg, branch), ...] () is paired, usually two pairs

  8. Subroutine call:

    Mnemonic Syntax Operand format Note
    CALL CALL V3, [V0, V8, V21,...], V33 CALL dreg, [areg, areg,...], retreg args and return are optional

  9. Branch (basic-block terminators):

    Mnemonic Syntax Operand format Note
    RET RET V8 RET [retreg] return; retreg optional
    SWITCH SWITCH V3, #0x1234, [(V2, #0x5678)] SWITCH treg, #defbranch, [(reg, #branch), ...] C-style switch
    J J 0x1234 J address unconditional jump
    JCC JCC<V8.b> 0x1234, 0x5678 JCC<creg> tbranch, fbranch conditional; true when creg == 1

  10. Conditional select:

    Mnemonic Syntax Operand format Note
    CSEL CSEL<V3.b> V6, V7, V9 CSEL dreg, sreg1, sreg2 like ARM64 CSEL

  11. Getelementptr (GEP):

    Mnemonic Syntax Operand format Note
    GEP GEP V2, V3, [V8, V9, V20] GEP dreg, breg, [index, ...]

Assembly

Having designed the custom assembly and understood the decode format, we now parse instruction data and print the assembly. The code shows this VM differs from native assembly: it has no stack-pointer register — because it’s a register-based VM, not stack-based.

Recovering to native

Two approaches were tried. The first converts VM instructions to LLVM IR and compiles the .ll with Clang to produce native code — so far the most effective. The second builds a decompiler IL and tries to recover pseudo-C — only an early attempt, unfinished.

llvmlite

llvmlite implements most of LLVM IR’s functionality, which is enough for this recovery. If you dislike llvmlite, the official LLVM or other libraries work too.

Install:

pip install llvmlite

Docs: llvmlite documentation

Building LLVM IR

When initializing external pointers and register constant assignments, register type info is lost, which forces lots of type and width conversion later. In practice, from a disassembly perspective you only need to recognize 1/2/4/8-byte data and pointer types — the lack of type info actually simplifies writing.

IR basics

The recovered assembly has many registers, and different native functions use different register counts after virtualization. These registers can be treated as variables. In high-level languages, we don’t think about register allocation — the compiler does it. Likewise in IR, register allocation is automatic.

Defining a variable — C/C++ declares a typed local without an initial value:

int num;

IR declares a variable:

t_int32 = ir.IntType(32)
ptr_num = builder.alloca(t, name="num")

In high-level languages int num; allocates on the stack by default; in IR, builder.alloca also allocates on the stack. The difference is that IR returns a pointer (e.g., int* ptr_num); to use the value, dereference it with builder.load — like *ptr_num.

Addition — C/C++:

int a = 111;
int b = 222;
int c = a + b;

IR:

t = ir.IntType(32)
# int a = 111;
ptr_a = builder.alloca(t, name="a") # define a, return a stack pointer
builder.store(ir.Constant(ir.IntType(32), 111), ptr_a)
# int b = 222;
ptr_b = builder.alloca(t, name="b") # define b, return a stack pointer
builder.store(ir.Constant(ir.IntType(32), 111), ptr_b)
# int c = a + b;
a_value = builder.load(ptr_a) # load a's value from the stack
b_value = builder.load(ptr_b) # load b's value from the stack
result = builder.add(a_value, b_value) # add; result is a value, not a stack pointer
ptr_c = builder.alloca(t, name="c") # define c, return a stack pointer
builder.store(result, ptr_c) # c = a + b;

Three lines of high-level code expand into quite a few IR lines — a bit clumsy.

Value extension — in IR, lvalue and rvalue types/widths must match before use; convert first. C/C++:

signed char c = 0xf8;
signed int n = c;

IR:

t_i8 = ir.IntType(32)
t_i32 = ir.IntType(32)
# signed char c = 0xf8;
ptr_c = builder.alloca(t_i8, name="c")
builder.store(ir.Constant(ir.IntType(32), 0xf8), ptr_c)
# signed int n = c;
ptr_n = builder.alloca(t_i32, name="n")
builder.store(ir.Constant(ir.IntType(32), 0xf8), ptr_n)
sext_type = ir.IntType(32) # the target extension type
c_value = builder.load(ptr_c) # load c's value
cast_value = builder.sext(c_value, sext_type) # sign-extend; result is a value
builder.store(cast_value, ptr_n) # store into n

Defining a function — first declare the function type (params and return), then define the function object. The function object contains basic blocks, which contain instructions. You need at least one basic block before adding instructions; for multiple blocks, use an “instruction pointer” to target a block.

Move the IR pointer to a block’s end:

builder.position_at_end(curr_ir_basic_block)

Move it to a block’s start:

builder.position_at_start(curr_ir_basic_block)

Common IR instructions — add/sub/mul/div:

builder.add(lhs, rhs)
builder.sub(lhs, rhs)
builder.mul(lhs, rhs)
builder.sdiv(lhs, rhs)
builder.udiv(lhs, rhs)

Signed modulo:

temp = builder.sdiv(lhs, rhs)
result = builder.sub(lhs, builder.mul(temp, rhs))

Pointer-to-pointer casts:

builder.bitcast(ptr, target_type)

Integer to pointer:

builder.inttoptr(val, target_ptr_type)

Pointer to integer:

builder.ptrtoint(val, target_int_type)

Comparison:

builder.icmp_signed("==", lhs, rhs) # lvalue/rvalue; don't use stack pointers for int compares
builder.icmp_signed("!=", lhs, rhs)
builder.icmp_signed(">", lhs, rhs)
builder.icmp_signed(">=", lhs, rhs)
icmp_unsigned("==", lhs, rhs)
icmp_unsigned("!=", lhs, rhs)
icmp_unsigned("<", lhs, rhs)
icmp_unsigned("<=", lhs, rhs))

Unconditional branch:

builder.branch(target_basic_block)

Conditional branch:

builder.cbranch(builder.load(ptr_cond), true_br, false_br)

Return:

builder.ret_void()
ret_val = builder.load(ptr_ret_val)
builder.ret(ret_val)

Recovery flow

My recovery method isn’t necessarily great — much of it was improvised. If you know IR well, implement your own way.

  1. Decode VM instruction bytes

    Instruction lengths vary, so the decoded instructions are kept in a list, making it easy to add them to basic blocks while finding block boundaries.

    vmState: VMState = BytecodeDecoder.build_vmp_state(
  filename, offset, size, extern_list)
instructions = vmState.deocde_instruction()

  2. Create basic blocks

    These are the recovered VM basic blocks. Terminator instructions are ret, switch, j, jcc. Scan from the first instruction, record the true/false and multi-way jump targets (which are block start addresses), and when a terminator is hit, end the block and save it.

    def get_jump_targets(insn):
    jump_targets = []
    match insn[0]:
        case 0x16:  # switch
            opcode, treg, type, defbranch, element_count, switch_branches = insn
            # [deftar, casetar, casetar, ...]
            jump_targets.append(defbranch)
            jump_targets.extend(switch_branches)
        case 0x1d:  # jcc
            truebr = insn[2]
            jump_targets.append(truebr)
    return jump_targets
def is_successors_insn(insn):
    match insn[0]: # instruction opcode
        case 0x13 | 0x16 | 0x1d: # ret: 0x13 | switch: 0x16 |  jmp/jcc: 0x1d
            return True
        case _:
            return False
def find_basic_blocks(instructions):
    """identify basic blocks"""
    jump_targets = set()
    basic_blocks = []
    current_block = []
    # first find all jump targets
    for insn in instructions:
        targets = get_jump_targets(insn)
        if targets:
            jump_targets.update(targets)
    pc = start = end = 0
    for insn in instructions:
        insn_len = get_insn_length(insn)
        print(f"{pc:#x}, {insn_len}")
        # if the current instruction is a jump target, start a new block
        if pc in jump_targets and current_block:
            basic_block_info = {"start_hex": f"{start:#x}", "end_hex": f"{pc + insn_len:#x}",
                                "start": start, "end": pc + insn_len,
                                "basic_block": current_block}
            basic_blocks.append(basic_block_info)
            current_block = []
            start = pc + insn_len
            print(f"{'-' * 50}")
        current_block.append(insn)
        # check if this is a terminator
        if is_successors_insn(insn):
            basic_block_info = {"start_hex": f"{start:#x}", "end_hex": f"{pc + insn_len:#x}",
                                "start": start, "end": pc + insn_len,
                                "basic_block": current_block}
            basic_blocks.append(basic_block_info)
            current_block = []
            start = pc + insn_len
            print(f"{'-' * 50}")
        end += insn_len
        pc += insn_len
    # add the last block if any
    if current_block:
        basic_block_info = {"start_hex": f"{start:#x}", "end_hex": f"{pc + len(insn):#x}",
                            "start": start, "end": pc + len(insn),
                             "basic_block": current_block}
        basic_blocks.append(basic_block_info)
    return basic_blocks

  3. Initialize the module

    Create the module and set the target architecture/platform:

     module = ir.Module(f"{filename}_{size:#x}_{extern_list:#x}")
 module.triple = "aarch64-unknown-linux-gnu" # target: linux aarch64

  4. Declare external functions

    The function calls calloc to allocate memory. From the assembly dump, calloc sizes are usually under 0x100; small allocations could be moved to the stack for speed, but I didn’t implement that optimization.

    size = 0xa8
t_array = ir.ArrayType(ir.IntType(8), size)
buffer = builder.alloca(t_array, name="buffer")

    Declare a function type with params/return, then create the external symbol calloc:

    def declare_external_calloc(module: ir.Module):
    declare_calloc = ir.FunctionType(ir.PointerType(ir.IntType(8)), [ir.IntType(32), ir.IntType(32)])
    fn_calloc = ir.Function(module, declare_calloc, "calloc")
    fn_calloc.args[0].name = "num"
    fn_calloc.args[1].name = "size"
    return fn_calloc

  5. Define an entry function

    All VM instructions will be written into the function via the IR interface.

    def ini_entry_function(vmState: VMState, module: ir.Module, alloca_regs: dict):
    entry_func_define = create_type(vmState.entry_function)
    entry_func = ir.Function(module, entry_func_define, "entry_func")
    for i in range(vmState.entry_function.argumentCount):
        name = "A" + str(i)
        entry_func.args[i].name = name
        alloca_regs[name] = entry_func.args[i]
    # entry_func.append_basic_block("entry")
    return entry_func

  6. Create all basic blocks for the entry function

    Use func.append_basic_block to add blocks (still empty, no instructions yet).

    def create_all_basic_blocks(func: ir.Function, basic_blocks):
    for bb in basic_blocks:
        func.append_basic_block(f"bb_{bb["start"]:x}")

    After create_all_basic_blocks returns, type print(entry_func) in the debug console to dump the function’s IR so far:

    define void @"entry_func"(i8* %"A0", i32 %"A1", i8* %"A2")
{
bb_0:
bb_32d:
bb_340:
bb_342:
}

  7. Create an IR builder for the module

    When a block is added via func.append_basic_block(), the block’s parent points to its function, and the function’s parent to its module. So creating the IR builder with a basic block auto-associates it with the module through the hierarchy.

    builder = ir.IRBuilder(entry_func.blocks[0])

  8. Initialize external pointers

    In the recovered assembly, external pointers are loaded into virtual registers, so those register variables need initial values. As noted, external-address references and register initialization carry no type info, so for now set them as 32-bit integer constants.

    def set_extern_regs(vmState: VMState, builder: ir.IRBuilder, alloca_regs):
    for i in range(0, len(vmState.extern.targetRegs)):
        v = get_register_ptr(alloca_regs, builder, vmState.extern.targetRegs[i], ir.IntType(32))
        builder.store(ir.Constant(ir.IntType(32), vmState.extern.externalAddress[i]), v)

  9. Initialize registers

    Register initial values are constants that stay unchanged through execution. They come from immediates in the original assembly, e.g. MOV X3, #0x88’s #0x88; virtualized, that becomes MOV V20, #0x88, where V20 is X3’s virtual register. Other native instructions can also initialize registers. For now only the encountered instructions are handled:

    def set_inial_regs(vmState: VMState, builder: ir.IRBuilder, alloca_regs):
    inial = vmState.inial
    types = vmState.types
    for i in range(0, len(inial.opcodes)):
        match inial.opcodes[i]:
            case 0 | 0xb | 0x17:
                ptr_breg = get_register_ptr(alloca_regs, builder, inial.sRegs[i])
                if isinstance(ptr_breg, ir.Argument):
                    val_breg = ptr_breg
                else:
                    t = types[inial.type[i]]
                    if t.tag != 0xf:
                        ir_type = ir.PointerType(create_type(t))
                    else:
                        ir_type = create_type(t)
                    val_breg = builder.bitcast(ptr_breg, ir_type)
                index_table = []
                for idx in inial.imm[i]:
                    index_table.append(ir.Constant(ir.IntType(32), idx))
                ele_ptr = builder.gep(val_breg, index_table)
                builder.store(ele_ptr, get_register_ptr(alloca_regs, builder, inial.dRegs[i], ele_ptr.type))
            case 7:
                v = get_register_ptr(alloca_regs, builder, inial.dRegs[i], ir.IntType(32))
                builder.store(ir.Constant(ir.IntType(32), inial.imm[i]), v)
            case 8:
                t = types[inial.type[i]]
                if t.tag != 0xe:
                    v = get_register_ptr(alloca_regs, builder, inial.dRegs[i], ir.IntType(32))
                    builder.store(ir.Constant(ir.IntType(32), 0), v)
                else:
                    raise Exception("error")
            case 0x15:
                print(
                    f"{'-' * 6:<}    {'-' * 5}registers const{'-' * 6}    MOV\tV{inial.dRegs[i]}.T@{Decoder.get_type_annotation(types[inial.type[i]])}, V{inial.sRegs[i]}")
            case _:
                raise Exception("error")

  10. Pre-allocation

    This makes the generated code nicer: allocate all stack variables at the start of the IR so the stack frame size is computed up front (e.g., sub sp, sp, #0x240). Without pre-allocation, the compiler inserts many mid-function stack adjustments, hurting assembly readability.

    First, walk the VM instructions and allocate their destination registers:

    def get_register_ptr(alloca_regs, builder: ir.IRBuilder, reg, t: ir.Type = ir.IntType(64)):
    name = f"V{reg}"
    arg_name = f"A{reg}"
    if alloca_regs.get(arg_name, None) is None:
        if alloca_regs.get(name, None) is None:
            alloca_regs[name] = builder.alloca(t, name=name)    # allocate a stack variable
        return alloca_regs[name]
    else:
        return alloca_regs[arg_name]
def preallocation_registers(basic_blocks, types: list[Type], alloca_regs, builder, vmState: VMState):
    pc = 0
    for bb in basic_blocks:
        for insn in bb["basic_block"]:
            print(f"prealloc regs addr: {pc:#x}")
            opcode = insn[0]
            match (opcode):
                case 0x1:   # ARITH
                    t = types[insn[2]]
                    sreg1 = insn[3]
                    sreg2 = insn[4]
                    dreg = insn[5]
                    get_register_ptr(alloca_regs, builder, dreg, create_type(t))
                case 0x2:   # MOV
                    op2 = insn[1]
                    dtype = types[insn[2]]
                    stype = types[insn[3]]
                    sreg = insn[4]
                    dreg = insn[5]
                    match op2:
                        case 0 | 5 | 0xA | 0xC:
                            get_register_ptr(alloca_regs, builder, dreg, create_type(stype))
                        case _:
                            get_register_ptr(alloca_regs, builder, dreg, create_type(dtype))
                case 0x6:   # CALLOC
                    a2_type = types[insn[1]]
                    a1_type = types[insn[2]]
                    sreg = insn[3]
                    dreg = insn[4]

                    t2_type = create_type(a2_type)
                    get_register_ptr(alloca_regs, builder, dreg, ir.PointerType(t2_type))
                    # get_register_ptr(alloca_regs, builder, dreg, t2_type)
                case 0xa:   # STR
                    type = types[insn[1]]
                    sreg = insn[2]
                    mreg = insn[3]
                    # get_register_ptr(alloca_regs, builder, sreg, create_type(type))
                    get_register_ptr(alloca_regs, builder, mreg, create_type(PointerType(type)))
                case 0xc:  # CMP
                    dreg = insn[4]
                    get_register_ptr(alloca_regs, builder, dreg, ir.IntType(1))
                # case 0xe:  # BSEL.PHI
                case 0xf:  # CALL
                    func_define: FunctionType = types[insn[1]]
                    if func_define.returnValueType.tag != 0:
                        ret_reg = insn[3]
                        t_ret_val = create_type(func_define.returnValueType)
                        get_register_ptr(alloca_regs, builder, ret_reg, t_ret_val)
                case 0x13:  # RET
                    ret_type = types[insn[1]]
                    if ret_type.tag != 0:
                        ret_reg = insn[2]
                        get_register_ptr(alloca_regs, builder, ret_reg, create_type(ret_type))
                case 0x16:  # SWITCH
                    raise Exception("not implemented")
                case 0x1d:  # JCC
                    cond = insn[1]
                    jmpTrue = insn[2]
                    if cond != 0:
                        flagReg = insn[3]
                        t = types[insn[4]]
                        get_register_ptr(alloca_regs, builder, flagReg, create_type(t))
                case 0x1e:  # CSEL
                    dreg = insn[6]
                    type = types[insn[3]]
                    get_register_ptr(alloca_regs, builder, dreg, create_type(type))
                case 0x28:  # GEP
                    dreg = insn[1]
                    t = types[insn[3]]
                    # for idx in insn[6]:
                    #     if not vmState.is_static_reg(idx):
                    #         ptr_reg = get_register_ptr(alloca_regs, builder, idx)
                    #         gep_idx = builder.load(ptr_reg)
                    # get_register_ptr(alloca_regs, builder, dreg, create_type(t))
                case 0x2a:  # LDR
                    t = types[insn[1]]
                    dreg = insn[3]
                    get_register_ptr(alloca_regs, builder, dreg, create_type(t))
            pc += get_insn_length(insn)

  11. Collect all PHI instructions and their params

    Walk every basic block and inspect its PHI instructions. For each PHI: reserve stack space for the result operand, parse the operand pairs to get each predecessor block and its incoming value, then insert writes to the reserved slot at the exit of each predecessor.

    def get_phi_nodes(state: VMState, alloca_regs, builder: ir.IRBuilder, func: ir.Function, basic_blocks, ):
    phi_nodes = []
    for bb in basic_blocks:
        bb_name = f"bb_{bb["start"]:x}"
        ir_basic_block = get_ir_basic_block_by_name(func, bb_name)
        for insn in bb["basic_block"]:
            match insn[0]:
                case 0xe:
                    # init the phi result (destination) register
                    phi_store_reg = insn[1]
                    builder.store(ir.Constant(ir.IntType(64), 0), get_register_ptr(alloca_regs, builder, phi_store_reg, ir.IntType(64)))
                    # get phi info
                    phi_node_info = []
                    for node in insn[3]:
                        val = node[0]
                        if not state.is_static_reg(val):
                            # allocate a stack var for a variable register and set 0
                            ptr_val = get_register_ptr(alloca_regs, builder, val, ir.IntType(64))
                            builder.store(ir.Constant(ir.IntType(64), 0), ptr_val)
                        ir_bb = get_basic_block(func, node[1])
                        phi_node_info.append([val, ir_bb])
                    phi_nodes.append({"phi_basic_block": ir_basic_block, "phi_store_reg": phi_store_reg, "phi_node_info": phi_node_info})
    return phi_nodes

  12. Emit IR for all instructions

    After all preparation, insert IR instructions. Points to note: instruction insertion must target a specific position in a block via the IRBuilder; set the target block at the start of each block iteration; because register init already placed some instructions, position the builder at the block end.

    for bb in basic_blocks:
    bb_name = f"bb_{bb["start"]:x}"
    curr_ir_basic_block = get_ir_basic_block_by_name(entry_func, bb_name)   # current instruction's block
    builder.position_at_end(curr_ir_basic_block)

    Then walk the VM instructions and emit IR (framework only):

    for bb in basic_blocks:
    bb_name = f"bb_{bb["start"]:x}"
    curr_ir_basic_block = get_ir_basic_block_by_name(entry_func, bb_name)   # current instruction's block
    builder.position_at_end(curr_ir_basic_block)    # move builder to the new block end
    for insn in bb["basic_block"]:
        print(f"addr: {pc:#x}")
        opcode = insn[0]
        match (opcode):
            case 0x1:   # ARITH
                op2 = insn[1]
                t = types[insn[2]]
                sreg1 = insn[3]
                sreg2 = insn[4]
                dreg = insn[5]
                match op2:
                    case 0: # XOR
                      # TODO IR
                    case 0x1: # SUB
                      # TODO IR
                    case 0x2: # LSR
                      # TODO IR
                    case 0x3: # UDIV
                      # TODO IR
                    case 0x4: # ADD
                      # TODO IR
                    case 0x5:    # OR
                      # TODO IR
                    case 0x6: # SMOD
                      # TODO IR
                    case 0x7: # SIDV
                      # TODO IR
                    case 0x8: # UMOD
                    case 0xA: # ASR
                      # TODO IR
                    case 0xC: # LSL
                      # TODO IR
                    case _:
                        # TODO Handler Not Impl
            case 0x2:   # MOV
                op2 = insn[1]
                dtype = types[insn[2]]
                stype = types[insn[3]]
                sreg = insn[4]
                dreg = insn[5]
                match op2:
                  case 0 | 5 | 0xA | 0xC: # keep operand types consistent
                    # TODO IR MOV dreg, sreg
                  case 0x1:   # extend
                    # TODO IR  MOV dreg, sreg.8
                  case 0x7:   # truncate
                    # TODO IR  MOV dreg.d, sreg.8
                  case 0x9:
                    # TODO IR
                  case _:
                      # TODO Handler Not Impl
            case 0x6:   # CALLOC
                a2_type = types[insn[1]]
                a1_type = types[insn[2]]
                sreg = insn[3]
                dreg = insn[4]
                # TODO call calloc(num, size)
            case 0xa:   # STR
                type = types[insn[1]]
                sreg = insn[2]
                mreg = insn[3]
                # TODO STR sreg, [mreg]
            case 0xc:  # CMP
                t = types[insn[1]]
                reg1 = insn[2]
                reg2 = insn[3]
                dreg = insn[4]
                op2 = insn[5]
                match op2:
                  case 0x20:  # CMP_EQ
                        # TODO
                  case 0x21:  # CMQ_NE
                      raise Exception(f"CMP.NE not impl")
                  case 0x24:  # CMP_CC
                      raise Exception(f"CMP.CC not impl")
                  case 0x28:  # CMP_LT
                      raise Exception(f"CMP.LT not impl")
                  case _:
                      input(f"unrecognized CMP op2={op2:#x}")
            case 0xe:  # BSEL.PHI
                phi_basic_blocks.append(curr_ir_basic_block)    # record the phi's block
            case 0xf:  # CALL
                # TODO
            case 0x13:  # RET
                ret_type = types[insn[1]]
                if ret_type.tag == 0:
                    # TODO no return value
                else:
                    # TODO with return value
            case 0x16:  # SWITCH
                raise Exception("not implemented")
            case 0x1d:  # JCC
                  # ------ check if the current node is a phi predecessor ------
                  # at a terminator, before writing it, check whether a phi node's
                  # predecessor label is in the current block
                # 1. save the current block info
                # 2. fetch the phi parameter node variable

                # ------ then handle the jump instruction ------
                cond = insn[1]
                jmpTrue = insn[2]
                if cond == 0:  # jmp
                    # TODO IR link to successor block
                else:  # j.cond
                    flagReg = insn[3]
                    t = types[insn[4]]
                    jmpFalse = insn[5]
                    # TODO IR link true and false blocks
            case 0x1e:  # CSEL
                ntype = insn[pc + 1]
                if ntype.tag == 0x10:
                    raise Exception("error")
                creg = insn[pc + 2]
                type = insn[pc + 3]
                treg = insn[pc + 4]
                freg = insn[pc + 5]
                dreg = insn[pc + 6]
                # TODO IR
            case 0x28:  # GEP
                dreg = insn[pc + 1]
                none = insn[pc + 2]
                type = insn[pc + 3]
                count = insn[pc + 4]
                breg = insn[pc + 5]
                # TODO IR
            case 0x2a:  # LDR
                type = insn[pc + 1]
                mreg = insn[pc + 2]
                dreg = insn[pc + 3]
                # TODO IR

    After emitting all instructions, handle PHI: PHI must be the first instruction of its block, so move the IR pointer to the block start, add the PHI combination args (variable, block<label>), then save the PHI result variable.

     # finally handle phi instructions
  if phi_basic_blocks:
      for bb in phi_basic_blocks:
          for phi_node in phi_nodes:
              if phi_node["phi_basic_block"] == bb:
                  builder.position_at_start(bb)   # insert phi at the block start
                  phi = builder.phi(int64_type, f"phi_var_{bb.name}")
                  for node in phi_node["phi_node_info"]:
                      val = node[0]
                      phi.add_incoming(val, node[1])
                  dreg_ptr = get_register_ptr(alloca_regs, builder, phi_node["phi_store_reg"], int64_type)
                  builder.store(phi, dreg_ptr)

    Finally, print and save the IR:

    print(module)

Compiling the IR

To prevent compiler optimization from deleting the generated native code, use -O0, and use -shared to build a library from the main-less code.

clang -O0 -shared devmp_0x168B60.ll -o devmp_0x168B60_O0.o

Verify the logic against disassembly

Load devmp_0x168B60_O0.o in IDA Pro and check that the recovered native code matches the VM assembly logic.

Pseudo-C without optimization

Optimized compile — with -O1:

clang -O1 -shared devmp_0x168B60.ll -o devmp_0x168B60_O1.o

Comparing -O0 vs -O1, the optimized build has noticeably fewer instructions.

Decompiling the optimized binary yields cleaner pseudo-C.

Building Binary Ninja IL

This is another viable recovery route. I tried building Binary Ninja IL for a few instructions but didn’t finish. From the few recovered instructions, it works — though not as well as LLVM IR, whose optimizer can collapse many instructions into surprisingly little, producing compact, readable code. The attached DecodeBNIL.py is a reference if you’re interested.

The VM and LLVM bitcode

One IR instruction, getelementptr, maps closely to a VM instruction — previously named ADD.MO, now GEP (“MO” = member offset). Looking at the LLVM source, the VM is very closely related to LLVM bitcode: the bitcode instructions, bytecode parsing, and interpreter logic are all strikingly similar to this VM. I’ll leave it at that — the article is already very long.

Summary

  • Identifying a VM — it’s hard to tell from a CFG alone; I have no quick method. VM protection’s goal is to hide real execution. To determine whether something is a VM, obfuscation, or a VM inside obfuscation, first understand obfuscation’s principles and signatures to rule out pure obfuscation. A VM has three steps — fetch, decode, execute-handler — often with switch dispatch tables between them (unless deliberately hidden). A complete VM handler set simulates a full instruction set, so there are many handlers: MOV-style moves, arithmetic, logic, subroutine/external calls, memory access, etc. After a handler completes it returns to the fetch point. Judge by these traits; usually you must analyze part of the logic to confirm. It takes accumulated experience — simple cases 1–3 days, complex ones 1–3 weeks.

  • Analysis time — many people ask how long this took; it’s a real cost consideration. Simple VMs take 1–2 weeks of working days; domestic big-company ones 2–4 weeks; international big-company ones are extremely strong and take 2–4 months. At that cost, they’ve achieved “practically unbreakable”.

    The earlier short-video VM analysis and recovery script took about 3.5 working weeks (~18 days), plus 1.5 weeks writing the article — about 5 weeks total. That SO was an unobfuscated VM with unencrypted bytecode. Elsewhere in the same SO the VM is the same system, just with stronger obfuscation and encrypted bytecode — harder, but not drastically.

    This app’s VM analysis and assembly-recovery script took about 4 working weeks total; IR writing about 1.5 weeks; roughly 6 weeks in all.

  • Recovery theory — for any VM whose instructions are close to native, you can borrow native assembly to recover to assembly. For VMs with custom instruction sets, in theory you can always recover to an intermediate language first, then to native assembly.

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch