Colin XiaoReverse Engineering & Security Research

Emulating an aarch64 Shared Library with Qiling and Reversing Its VM-Protected RC4

About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

I had to reverse a libxx.so encryption routine recently, and it turned out to be an aarch64 binary. Getting it to run on a Windows host with the Qiling emulation framework was an exercise in debugging Qiling itself. This post is a walkthrough of what broke, why it broke, and how I got from “cannot even run a hello-world ELF” to fully recovering the algorithm.

The three things I want to cover:

  • How to emulate an aarch64 .so with Qiling
  • The Windows-specific problems I hit, and their fixes
  • Emulating a VM-protected algorithm and recovering it

I’m writing this from a beginner’s angle, so the steps should be reproducible.

The target and why emulation

The target is a libxx.so whose encryption algorithm I needed to understand. It’s aarch64, so there are three obvious approaches:

  • Static analysis in IDA
  • Debugging with a debugger to trace the algorithm
  • Emulation with unidbg or unicorn

Static analysis is too expensive

Looking at the encryption function in IDA, it resembles nested control-flow flattening — but it’s actually a virtual machine, which makes static analysis considerably harder.

The obfuscated encryption function in IDA

Debugging

My first idea was to compile an ARM64 executable, load libxx.so, and call the encryption function under a debugger. The string info in IDA shows the .so was built with NDK + LLVM for Android, so I considered spinning up an ARM64 Ubuntu VM — but Ubuntu only ships a server image for ARM64 guests. Other distros might have worked, but emulation gives better tracing facilities than a debugger, so I went with the emulator.

In hindsight, the most stable route is probably an Android Studio app debugged with lldb. If you have a better approach, I’d love to hear it.

Emulation

unidbg builds on unicorn and can emulate Android .so files well. But unidbg has known emulation gaps (see the post I reference below), so I chose Qiling instead. Qiling ships its own qdb debugger, but it doesn’t support multithreading; the udbserver project adds that and pairs with pwndbg:

Multithreaded debugging with udbserver + pwndbg

Qiling also has an IDA plugin that lets you emulate a .so directly inside IDA. Since algorithm analysis means constantly cross-referencing IDA views, I picked the Qiling IDA plugin for “visual debugging” of the .so.

Running an ARM64 demo

Under qiling/examples/rootfs/arm64_linux/bin/ there are many arm64 binaries to emulate, and lib/ holds the matching dynamic linker — for ARM64 that’s usually ld-linux-aarch64.so.1. The dynamic linker is critical: it’s responsible for loading the binary’s dependencies and performing relocations.

Let’s use arm64_hello (which prints “Hello, World!”) to demonstrate the IDA plugin workflow.

Setting up Qiling

Load arm64_hello in IDA, then run File → Script file... and load qiling/extensions/idaplugin/qilingida.py.

On my machine the symlink approach didn’t work — Qiling was likely only tested on Linux — so I load qilingida.py directly from IDA.

Then right-click in the disassembly view → Qiling Emulator → Setup, using Qiling’s default helper script (custom_script.py). Automation logic — adding syscalls or hooking addresses — goes into this helper script.

Loading the Qiling IDA plugin

The demo failed on first run:

  File "C:\Program Files\Python39\lib\site-packages\qiling\os\linux\linux.py", line 30, in __init__
    super(QlOsLinux, self).__init__(ql)
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\posix\posix.py", line 190, in __init__
    super().__init__(ql)
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\os.py", line 63, in __init__
    sys.stdin.fileno()
AttributeError: 'NoneType' object has no attribute 'fileno'

Looking at the relevant os.py source:

try:
            # Qiling may be used on interactive shells (ex: IDLE) or embedded python
            # interpreters (ex: IDA Python). such environments use their own version
            # for the standard streams which usually do not support certain operations,
            # such as fileno(). here we use this to determine how we are going to use
            # the environment standard streams
            sys.stdin.fileno()
        except UnsupportedOperation:
            # Qiling is used on an interactive shell or embedded python interpreter.
            # if the internal stream buffer is accessible, we should use it
            self._stdin  = getattr(sys.stdin,  'buffer', sys.stdin)
            self._stdout = getattr(sys.stdout, 'buffer', sys.stdout)
            self._stderr = getattr(sys.stderr, 'buffer', sys.stderr)

The error says sys.stdin is None, because IDA replaces sys.stdin with its own I/O stream. This is a known issue in Qiling’s GitHub issues and pull requests. I just caught AttributeError directly:

except (UnsupportedOperation, AttributeError):

After that fix, restart IDA and reload Qiling (the Python-installed Qiling), then Qiling → Setup again, and a new error appears:

The os.py file I fixed here is the Qiling under the Python library: C:\Program Files\Python39\lib\site-packages\qiling\os\os.py.

  File "E:\gitRepo\qiling\examples\extensions\idaplugin\custom_script.py", line 1, in <module>
    from future import __annotations__
ImportError: cannot import name '__annotations__' from 'future' (C:\Program Files\Python39\lib\site-packages\future\__init__.py)
[INFO][(unknown file):0] Custom user script not found.

I’m on Python 3.9.13. Python has __future__, not future — not sure if this is version-related. Changing it to the correct form fixed it:

from __future__ import annotations

Emulating

arm64_hello is now loaded into memory. The dynamic linker must fix up relocations and set the entry point, then execution starts.

Reading qiling/os/linux.py’s QlOsLinux::run: this is a single-threaded environment with no explicit start address, so the start address is the dynamic linker’s entry point. After the linker initializes, it jumps to the program’s real entry point:

  try:
  	# 如果是一段二进制代码
      if self.ql.code:
          self.ql.emu_start(self.entry_point, (self.entry_point + len(self.ql.code)), self.ql.timeout, self.ql.count)
      else:
          # 如果是多线程环境
          if self.ql.multithread:
              # start multithreading
              thread_management = thread.QlLinuxThreadManagement(self.ql)
              self.ql.os.thread_management = thread_management
              thread_management.run()

          else:
              # 不是多线程环境
              # 程序入口点是否有显式指定
              if self.ql.entry_point is not None:
                  self.ql.loader.elf_entry = self.ql.entry_point

              # do we have an interp?
              elif self.ql.loader.elf_entry != self.ql.loader.entry_point:
                  entry_address = self.ql.loader.elf_entry

                  if self.ql.arch.type == QL_ARCH.ARM:
                      entry_address &= ~1

                  # start running interp, but stop when elf entry point is reached
                  self.ql.emu_start(self.ql.loader.entry_point, entry_address, self.ql.timeout)
                  self.ql.do_lib_patch()
                  self.ql.run_function_after_load()
                  self.ql.loader.skip_exit_check = False
                  self.ql.write_exit_trap()

              self.ql.emu_start(self.ql.loader.elf_entry, self.exit_point, self.ql.timeout, self.ql.count)

Right-click in the disassembly → Qiling Emulator → Continue to start. It errors again:

[+] 	brk: increasing program break from 0x555555568000 to 0x555555589000
[+] 	0x00007fffb7e9e93c: brk(inp = 0x555555589000) = 0x555555589000
[+] 	Received interrupt: 0x2
[+] 	write() CONTENT: b'Hello, World!\n'
[x] 	Syscall ERROR: ql_syscall_write DEBUG: A string expected
Traceback (most recent call last):
...
File "C:\Program Files\Python39\lib\site-packages\qiling\os\posix\syscall\unistd.py", line 410, in ql_syscall_write
    f.write(data)
  File "D:\Tools\IDA 7.5\python\3\init.py", line 63, in write
    ida_kernwin.msg(text)
  File "D:\Tools\IDA 7.5\python\3\ida_kernwin.py", line 236, in msg
    return _ida_kernwin.msg(*args)
TypeError: A string expected

The puts call ran and printed “Hello, World”, but passing that string to Python’s print raised an exception — Qiling passed a byte string to ida_kernwin.msg, which expects a str. Two ways to fix it:

  • Patch D:\Tools\IDA 7.5\python\3\ida_kernwin.py to accept byte strings, or
  • Hook write and replace Qiling’s own ql_syscall_write.

The first is global and affects other binaries, and hooking syscalls is a first-class Qiling feature, so I went with the hook. Updating custom_script.py:

def my_syscall_write(ql: Qiling, fd: int, buf: int, count: int):
    ql.log.info('my_syscall_write called')
    try:
        # read data from emulated memory
        data = ql.mem.read(buf, count)

        # select the emulated file object that corresponds to the requested
        # file descriptor
        fobj = ql.os.fd[fd]
        if fobj == None:
            ql.log.ingo('none file descriptor')
        # write the data into the file object, if it supports write operations
        elif hasattr(fobj, 'write'):
            fobj.write(data.decode('utf-8'))
    except:
        ret = -1
    else:
        ret = count

    ql.log.info(f'my_syscall_write({fd}, {buf:#x}, {count}) = {ret}')

    return ret
    
class QILING_IDA:

    def _show_context(self, ql: Qiling):
        registers = tuple(ql.arch.regs.register_mapping.keys())
        grouping = 4

        for idx in range(0, len(registers), grouping):
            ql.log.info('\t'.join(f'{r:5s}: {ql.arch.regs.read(r):016x}' for r in registers[idx:idx + grouping]))

    def custom_prepare(self, ql: Qiling) -> None:
        ql.log.info('Context before starting emulation:')
        ql.os.set_syscall('write', my_syscall_write)
        ql.log.info('my_syscall_write registered')
        self._show_context(ql)

I implemented my_syscall_write, then in QILING_IDA.custom_prepare called ql.os.set_syscall to register it. After editing, Qiling Emulator → Reload User Scripts reloads the script, then Restart re-runs — this time it prints “Hello, World!” correctly:

[+] 	Received interrupt: 0x2
[=] 	my_syscall_write called
Hello, World!
[=] 	my_syscall_write(1, 0x555555568260, 14) = 14
[+] 	0x00007fffb7e98afc: my_syscall_write(fd = 0x1, buf = 0x555555568260, count = 0xe) = 0xe
[+] 	Received interrupt: 0x2
[+] 	0x00007fffb7e78a2c: exit_group(code = 0x0) = ?

Setting the program counter explicitly

With Qiling → Continue working, I tried the “set run address” feature. First Qiling → Restart, set a breakpoint on __libc_start_main in IDA, then Qiling → Continue — it breaks there. Qiling → View Register shows the PC is 0x5E0 + program base.

PC register after breaking on __libc_start_main

Qiling → Step a few times shows different colors in IDA, each representing an execution operation: blue is single-step, green is the path covered by direct execution.

Single-step coloring in IDA

sub_724 is the real print function. Move the cursor to its first instruction and Qiling → Set PC:

[INFO][(unknown file):0] QIling PC set to 0x724

Then Qiling → Continue fails:

[x] 	CPU Context:
[x] 	x0	: 0x555555554724
[x] 	x1	: 0x1
[x] 	x2	: 0x80000000de18
...
[x] 	PC = 0x0000000000000724 (unreachable)
...
  File "C:\Program Files\Python39\lib\site-packages\qiling\core.py", line 771, in emu_start
    self.uc.emu_start(begin, end, timeout, count)
  File "C:\Program Files\Python39\lib\site-packages\unicorn\unicorn.py", line 547, in emu_start
    raise UcError(status)
unicorn.unicorn.UcError: Invalid memory fetch (UC_ERR_FETCH_UNMAPPED)

The error says PC 0x724 is unreachable. Looking at the registers:

Register view showing the bad PC

The PC here is 0x724. Recalling that Qiling registers are IDA address + base, the cause is clear: Qiling → Set PC used the raw IDA address without adding the module base. Patching qilingida.py:

    def ql_set_pc(self):
        if self.qlinit:
            # ea = IDA.get_current_address()
            ea = self.qlemu.ql_addr_from_ida(IDA.get_current_address())
            self.qlemu.ql.arch.regs.arch_pc = ea
            logging.info(f"QIling PC set to {hex(ea)}")
            self.qlemu.status = self.qlemu.ql.save()
            self.ql_update_views(self.qlemu.ql.arch.regs.arch_pc, self.qlemu.ql)
        else:
            logging.error('Qiling should be setup firstly.')

After the patch, unload and reload the plugin (Qiling → Unload Plugin, then File → Script file), and redo the steps — now the PC includes the base:

[INFO][(unknown file):0] QIling PC set to 0x555555554724

Finally, move the cursor to the last instruction of sub_724 and Qiling → Execute Till. It executes successfully this time; orange marks the instructions covered by Execute Till:

Execute Till through sub_724

The next instruction is 0x748:

Next instruction at 0x748

Takeaways so far

Qiling’s IDA plugin is still rough — even on the official dev branch (the recommended one), emulating a demo hits multiple bugs, so it’s not beginner-friendly. That’s exactly why I’m writing these fixes down. There are more annoying bugs ahead.

Emulating libxx.so (and fixing what breaks)

The demo runs, so now the real target. The algorithm’s core steps are init, encrypt, and decrypt, so I start with the init function.

After loading libxx.so and initializing Qiling, you can’t just point the PC at the init function — the runtime environment isn’t ready yet. tpidr_el0 (roughly analogous to the fs segment register / TEB on Windows) hasn’t been set up, and the .so hasn’t been relocated yet, so global references like a memset call won’t resolve.

Load libxx.so in IDA, initialize Qiling (Qiling → Setup), set a breakpoint on the .so entry (the start function), then Qiling → Continue:

[x] 	Syscall ERROR: ql_syscall_futex DEBUG: 'NoneType' object has no attribute 'cur_thread'
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\posix\posix.py", line 374, in load_syscall
    retval = syscall_hook(self.ql, *params)
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\posix\syscall\futex.py", line 43, in ql_syscall_futex
    regreturn = ql.os.futexm.futex_wake(ql, uaddr,ql.os.thread_management.cur_thread, val)
AttributeError: 'NoneType' object has no attribute 'cur_thread'

The relevant source:

elif op & (FUTEX_PRIVATE_FLAG - 1) == FUTEX_WAKE:
        regreturn = ql.os.futexm.futex_wake(ql, uaddr,ql.os.thread_management.cur_thread, val)

futex is a Linux lock; thread_management is only initialized when Qiling runs in multithreaded mode. So enable multithread when the Qiling instance is created in qilingida.py:

class QlEmuQiling:
    def __init__(self):
        self.path = None
        self.rootfs = None
        self.ql: Qiling = None
        self.status = None
        self.exit_addr = None
        self.baseaddr = None
        self.env = {}

    def start(self, *args, **kwargs):
        self.ql = Qiling(argv=self.path, rootfs=self.rootfs, verbose=QL_VERBOSE.DEFAULT, multithread=True, env=self.env, log_plain=True, *args, **kwargs)
        # ...

I also dropped the log level from QL_VERBOSE.DEBUG to QL_VERBOSE.DEFAULT to reduce noise. After reloading qilingida.py and re-running:

[x] [Thread 2000]	Syscall ERROR: ql_syscall_writev DEBUG: A string expected
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\posix\posix.py", line 374, in load_syscall
    retval = syscall_hook(self.ql, *params)
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\posix\syscall\uio.py", line 23, in ql_syscall_writev
    ql.os.fd[fd].write(buf)
  File "D:\Tools\IDA 7.5\python\3\init.py", line 63, in write
    ida_kernwin.msg(text)
  File "D:\Tools\IDA 7.5\python\3\ida_kernwin.py", line 236, in msg
    return _ida_kernwin.msg(*args)
TypeError: A string expected

Same bug as before, but this time the syscall is writev, not write. Add a writev hook (my_syscall_writev) to custom_script.py, reload the user script, restart Qiling — this time it reaches the start function:

Reaching the start function

But tpidr_el0 is 0, meaning the thread runtime environment isn’t ready. Continuing anyway hits another error:

[x] [Thread 2000]	PC = 0x000000000009aec0 (unreachable)
[x] [Thread 2000]	Memory map:
[x] [Thread 2000]	Start            End              Perm    Label        Image
[x] [Thread 2000]	00555555554000 - 00555555847000   r-x     libxx.so     E:\gitRepo\qiling\examples\rootfs\arm64_linux\bin\libxx.so
[x] [Thread 2000]	00555555856000 - 0055555588f000   rw-     libxx.so     E:\gitRepo\qiling\examples\rootfs\arm64_linux\bin\libxx.so
[x] [Thread 2000]	0055555588f000 - 00555555891000   rwx     [hook_mem]   
[x] [Thread 2000]	007ffffffde000 - 0080000000e000   rwx     [stack]      
Traceback (most recent call last):
...
  File "C:\Program Files\Python39\lib\site-packages\unicorn\unicorn.py", line 547, in emu_start
    raise UcError(status)
unicorn.unicorn.UcError: Invalid memory fetch (UC_ERR_FETCH_UNMAPPED)

The PC is about to execute 0x9aec0, which isn’t mapped. The bigger clue: no dynamic linker is mapped. Checking with readelf:

$ readelf -S libxx.so | grep interp
---
$ readelf -S arm64_hello | grep interp
  [ 1] .interp           PROGBITS         0000000000000200  00000200

libxx.so has no .interp section while arm64_hello does, so Qiling didn’t load a dynamic linker for it. I need to point it at one explicitly. To find where, turn on debug logging:

    def start(self, *args, **kwargs):
        self.ql = Qiling(argv=self.path, rootfs=self.rootfs, verbose=QL_VERBOSE.DEBUG, multithread=True, env=self.env, log_plain=True, *args, **kwargs)

Reload and initialize — no relevant output. But opening arm64_hello shows:

[INFO][qilingida:1034] Custom env: {}
[+] 	Profile: default
[+] 	Mapped 0x555555554000-0x555555555000
[+] 	Mapped 0x555555564000-0x555555566000
[+] 	mem_start : 0x555555554000
[+] 	mem_end   : 0x555555566000
[+] 	Interpreter path: /lib/ld-linux-aarch64.so.1
[+] 	Interpreter addr: 0x7ffff7dd5000
[+] 	Mapped 0x7ffff7dd5000-0x7ffff7df2000
[+] 	Mapped 0x7ffff7e01000-0x7ffff7e04000
[+] 	mmap_address is : 0x7fffb7dd6000

So arm64_hello’s linker is /lib/ld-linux-aarch64.so.1. Tracing that log string into Qiling’s source:

def load_with_ld()
	def load_elf_segments()
    	# ...
        # determine interpreter path
        interp_seg = next(elffile.iter_segments(type='PT_INTERP'), None)
        interp_path = str(interp_seg.get_interp_name()) if interp_seg else ''

interp_path is the dynamic linker path. Specify it explicitly when empty:

        if len(interp_path) == 0:
            interp_path = "/lib/ld-linux-aarch64.so.1"

Restart Qiling and the linker loads:

[INFO][qilingida:1034] Custom env: {}
[+] 	Profile: default
[+] 	Mapped 0x555555554000-0x555555847000
[+] 	Mapped 0x555555856000-0x55555588f000
[+] 	mem_start : 0x555555554000
[+] 	mem_end   : 0x55555588f000
[+] 	Interpreter path: /lib/ld-linux-aarch64.so.1
[+] 	Interpreter addr: 0x7ffff7dd5000
[+] 	Mapped 0x7ffff7dd5000-0x7ffff7df2000
[+] 	Mapped 0x7ffff7e01000-0x7ffff7e04000

Continue emulation, and yet another error:

[+] [Thread 2000]	b'Inconsistency detected by ld.so: '
Inconsistency detected by ld.so: [+] [Thread 2000]	b'rtld.c'
rtld.c[+] [Thread 2000]	b': '
: [+] [Thread 2000]	b'1266'
1266[+] [Thread 2000]	b': '
: [+] [Thread 2000]	b'dl_main'
dl_main[+] [Thread 2000]	b': '
: [+] [Thread 2000]	b'Assertion `'
Assertion `[+] [Thread 2000]	b'GL(dl_rtld_map).l_libname'
GL(dl_rtld_map).l_libname[+] [Thread 2000]	b"' failed!\n"
' failed!
...
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\linux\linux.py", line 167, in run
    thread_management.run()
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\linux\thread.py", line 613, in run
    previous_thread = self._prepare_lib_patch()
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\linux\thread.py", line 593, in _prepare_lib_patch
    raise QlErrorExecutionStop('Dynamic library .init() failed!')
qiling.exception.QlErrorExecutionStop: Dynamic library .init() failed!

The first line says the linker is incompatible with this .so. This .so was NDK-built for Android, so the right linker is Android’s linker64.

The udbserver project forks Qiling and includes linker64 plus the other .so files Android needs (libz, libm, liblog). readelf shows libxx.so depends on these, so prepare them all:

$ readelf -d libxx.so

Dynamic section at offset 0x311f88 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [liblog.so]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000000e (SONAME)             Library soname: [[redacted].so]
 0x0000000000000019 (INIT_ARRAY)         0x302dc0
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x302dc8
 ...
E:\gitRepo\qiling\examples\rootfs\arm64_linux\system\lib64：保存libm.so等库
E:\gitRepo\qiling\examples\rootfs\arm64_linux\lib：保存linker64

After swapping in linker64, restart IDA, initialize Qiling, and Qiling → Continue:

[x] [Thread 2000]	Disassembly:
[=] [Thread 2000]	00007ffff7ddfbd0 [linker64             + 0x00abd0]  9c 20 40 79          ldrh                 w28, [x4, #0x10]
[=] [Thread 2000]	00007ffff7ddfbd4 [linker64             + 0x00abd4]  9f 0f 00 71          cmp                  w28, #3
[=] [Thread 2000]	00007ffff7ddfbd8 [linker64             + 0x00abd8]  81 34 00 54          b.ne                 #0x7ffff7de0268
...
  File "C:\Program Files\Python39\lib\site-packages\qiling\core.py", line 771, in emu_start
    self.uc.emu_start(begin, end, timeout, count)
  File "C:\Program Files\Python39\lib\site-packages\unicorn\unicorn.py", line 547, in emu_start
    raise UcError(status)
unicorn.unicorn.UcError: Invalid memory read (UC_ERR_READ_UNMAPPED)

X4 is 0, causing the memory access fault. Opening linker64 in IDA at the crash point: the function is __dl___linker_init, which initializes the shared library. Tracing why X4 is 0 takes time, and this linker64 is from an old Android 6.0 and may be incompatible. I happened to have a Pixel on hand, so I copied its linker64 out and retried:

[+] [Thread 2001]	b'libc'
libc[+] [Thread 2001]	b': '
: [+] [Thread 2001]	b'unable to stat "/proc/self/exe": Operation not permitted'
unable to stat "/proc/self/exe": Operation not permitted[+] [Thread 2001]	b'\n'

It can’t read /proc/self/exe — a symlink linker64 uses to get its own absolute path. The linker pseudocode:

linker64 /proc/self/exe logic

Hook address 0x27DA8 and return the absolute path and length, updating custom_script.py:

def custom_continue(self, ql: Qiling) -> List[HookRet]:
    # ...
    def addr_27DA8_hook(ql: Qiling) -> None:
        ql.arch.regs.W0 = 0
        ql.arch.regs.PC += 4
    # ...
    return [ql.hook_address(addr_27DA8_hook, 0x27DA8+linker_baseaddr)]

Run again, and another one:

[+] [Thread 2000]	b'libc'
libc[+] [Thread 2000]	b': '
: [+] [Thread 2000]	b'Could not find a PHDR: broken executable?'
Could not find a PHDR: broken executable?[+] [Thread 2000]	b'\n'

No PHDR found. Checking with readelf:

$ readelf -l libxx.so

Elf file type is DYN (Shared object file)
Entry point 0xa5370
There are 8 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x00000000002f25b4 0x00000000002f25b4  R E    0x10000
  LOAD           0x00000000002f2dc0 0x0000000000302dc0 0x0000000000302dc0
                 0x0000000000034478 0x0000000000037d58  RW     0x10000
  DYNAMIC        0x0000000000311f88 0x0000000000321f88 0x0000000000321f88
                 0x0000000000000210 0x0000000000000210  RW     0x8
  NOTE           0x0000000000000200 0x0000000000000200 0x0000000000000200
                 0x0000000000000024 0x0000000000000024  R      0x4
  NOTE           0x00000000002f251c 0x00000000002f251c 0x00000000002f251c
                 0x0000000000000098 0x0000000000000098  R      0x4
  GNU_EH_FRAME   0x00000000002da5dc 0x00000000002da5dc 0x00000000002da5dc
                 0x000000000000355c 0x000000000000355c  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x00000000002f2dc0 0x0000000000302dc0 0x0000000000302dc0
                 0x0000000000025240 0x0000000000025240  R      0x1

PHDR is normally the first element of the program table, but it’s genuinely missing here. In IDA:

PHDR check in IDA

There’s a +0x10 check here too. Comparing the linker64 context from the udbserver fork with the Pixel one, it’s the same problem — I have to reverse it further.

The image annotation is the result of that analysis.

Two options: full trace with Qiling’s hook_code (printing the context before each instruction), or break at the relevant code and single-step. I chose single-step. To break at a specific address, understand this part of qilingida.py:

def ql_continue(self):
    logging.info("before continue...")
    if self.qlinit:
        userhook = None
        # 调用hook_code，每条指令执行前调用ql_path_hook
        pathhook = self.qlemu.ql.hook_code(self.ql_path_hook)

def ql_path_hook(self, ql, addr, size):
    addr = addr - self.qlemu.baseaddr + get_imagebase()
    set_color(addr, CIC_ITEM, 0x007FFFAA)
    # 获取断点数量
    bp_count = get_bpt_qty()
    bp_list = []
    if bp_count > 0:
        for num in range(0, bp_count):
            bp_list.append(get_bpt_ea(num))

        # 如果当前准备执行的指令是断点处的指令，调用ql.save和ql.os.stop()保存当前模拟执行环境
        if addr in bp_list and (addr != self.lastaddr or self.is_change_addr>1):
            self.qlemu.status = ql.save()
            ql.os.stop()
            self.lastaddr = addr
            self.is_change_addr = -1
            jumpto(addr)

        self.is_change_addr += 1

When emulation hits a breakpoint, this saves the state and stops. So to break at the code I want to analyze, add a breakpoint in ql_path_hook:

def ql_path_hook(self, ql, addr, size):
    addr = addr - self.qlemu.baseaddr + get_imagebase()
    set_color(addr, CIC_ITEM, 0x007FFFAA)
    bp_count = get_bpt_qty() + 1
    bp_list = []
    bp_list.append(0x27f2c+0x007ffff7dd5000-0x555555554000)
    if bp_count > 0:
        for num in range(0, bp_count):
            bp_list.append(get_bpt_ea(num))

Add one element to bp_list: 0x27f2c is the RVA, followed by the relocated base adjustment. That breaks there and allows single-stepping.

  • The second line subtracts the libxx.so module base during relocation, so the element added to bp_list must subtract libxx.so’s base — not 0x27f2c+0x007ffff7dd5000-0x007ffff7dd5000=0x27f2c.

  • After the interrupt, the PC points into linker64, so this error appears — but it doesn’t affect further single-stepping:

    [x] [Thread 2000]    [Thread 2000] Expect 0x5555555f9370 but get 0x7ffff7e9e920 when running loader.
Traceback (most recent call last):
...
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\linux\linux.py", line 167, in run
    thread_management.run()
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\linux\thread.py", line 613, in run
    previous_thread = self._prepare_lib_patch()
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\linux\thread.py", line 593, in _prepare_lib_patch
    raise QlErrorExecutionStop('Dynamic library .init() failed!')
qiling.exception.QlErrorExecutionStop: Dynamic library .init() failed!

After single-stepping, the pseudocode is:

i = 0;
while (program_table_element[i].p_type!=PT_PHDR) {
	i++;
    if (i >= elf_header.e_phnum)
        break;
}

if (i == elf_header.e_phnum) {
    print("no phdr");
}
else {
    // 获取文件基址在内存的地址
    a = phdr_addr - program_table_element[i].p_paddr;
    // 获取内存基址
    b = phdr_addr - program_table_elemet[i].p_offset;
}

I can simulate the program header table to get variables a and b, updating custom_script.py:

def custom_continue(self, ql: Qiling) -> List[HookRet]:
	#...
	def addr_27FB4_hook(ql: Qiling) -> None:
        	# 如果是检测第一个程序表元素，且不是程序头表
            if ql.arch.regs.X11 == 0 and ql.arch.regs.W12 == 1:
                phdr_vir_addr = ql.arch.regs.X8
                poffset_addr = ql.arch.regs.x10
                write_base = ql.arch.regs.X19
                ql.log.info(f'phdr_vir_addr: {phdr_vir_addr:#x}')
                ql.log.info(f'poffset_addr: {poffset_addr:#x}')
                sub1 = to_bstring(phdr_vir_addr-0x40)
                # *(QWORD*)((__int64)v50+0x100) = image_vir_addr
                ql.mem.write(write_base+0x100, sub1)
                ql.log.info(b'sub1: ' + sub1)

                image_vir_addr = phdr_vir_addr-0x40
                # *(QWORD*)((__int64)v50+0x10) = image_vir_addr
                ql.mem.write(write_base+0x10, to_bstring(0x40))
                ql.log.info(f'sub2: {image_vir_addr:#x}')

                ql.arch.regs.X8 = image_vir_addr
                ql.arch.regs.PC = 0x28070 + 0x7ffff7dd5000
                
	ql.hook_address(addr_27FB4_hook, 0x27FB4+linker_baseaddr)

Remember to remove the breakpoint added earlier in qilingida.py.

Restart Qiling and it can break at libxx.so’s start. Unfortunately, we still can’t emulate the target function smoothly, because reaching start takes 3–4 minutes — restarting Qiling after every tweak is far too slow.

While writing this post I re-ran the flow for a demo, and this time it hung: IDA burned CPU all night without finishing. The cause: Qiling → Continue calls the callback (qilingida.py → ql_path_hook) on every instruction. Disabling that callback registration gets to start in about 2–3 seconds.

Restarting Qiling after that change hit this error:

[x] [Thread 2000]	v28	: 0x0
[x] [Thread 2000]	v29	: 0x0
[x] [Thread 2000]	v30	: 0x0
[x] [Thread 2000]	v31	: 0x0
[x] [Thread 2000]	PC = 0x0000000000000000 (unreachable)

[x] [Thread 2000]	Memory map:
[x] [Thread 2000]	Start            End              Perm    Label              Image
[x] [Thread 2000]	00555555554000 - 00555555847000   r-x     libxx.so           E:\gitRepo\qiling\examples\rootfs\arm64_linux\bin\libxx.so
[x] [Thread 2000]	00555555856000 - 0055555587c000   r--     libxx.so           E:\gitRepo\qiling\examples\rootfs\arm64_linux\bin\libxx.so
[x] [Thread 2000]	0055555587c000 - 0055555588f000   rw-     libxx.so           E:\gitRepo\qiling\examples\rootfs\arm64_linux\bin\libxx.so
[x] [Thread 2000]	0055555588f000 - 00555555891000   rwx     [hook_mem]         
[x] [Thread 2000]	007fffb7dd6000 - 007fffb7dd7000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb7dd7000 - 007fffb7dda000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7dda000 - 007fffb7ddb000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb7ddb000 - 007fffb7ddc000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb7ddc000 - 007fffb7de0000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7de0000 - 007fffb7de1000   r--     [mmap anonymous]   
[x] [Thread 2000]	007fffb7de1000 - 007fffb7de2000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7de2000 - 007fffb7de3000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7de3000 - 007fffb7de4000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7de5000 - 007fffb7de6000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7de6000 - 007fffb7de7000   r--     [mmap anonymous]   
[x] [Thread 2000]	007fffb7de7000 - 007fffb7de8000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb7de8000 - 007fffb7de9000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7de9000 - 007fffb7dea000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb7dea000 - 007fffb7deb000   r--     [mmap anonymous]   
[x] [Thread 2000]	007fffb7deb000 - 007fffb7dec000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7dec000 - 007fffb7ded000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7ded000 - 007fffb7dee000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7dee000 - 007fffb7def000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7def000 - 007fffb7df0000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7df0000 - 007fffb7df1000   r--     [mmap anonymous]   
[x] [Thread 2000]	007fffb7df1000 - 007fffb7df2000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7e31000 - 007fffb7e32000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7e32000 - 007fffb7e33000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7e4b000 - 007fffb7f0e000   r-x     [mmap] libc.so     
[x] [Thread 2000]	007fffb7f0e000 - 007fffb7f1d000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb7f1d000 - 007fffb7f23000   r--     [mmap] libc.so     
[x] [Thread 2000]	007fffb7f23000 - 007fffb7f26000   rw-     [mmap] libc.so     
[x] [Thread 2000]	007fffb7f26000 - 007fffb7f34000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb7f5d000 - 007fffb7f5e000   r-x     [mmap] libdl.so    
[x] [Thread 2000]	007fffb7f5e000 - 007fffb7f6d000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb7f6d000 - 007fffb7f6e000   r--     [mmap] libdl.so    
[x] [Thread 2000]	007fffb7f6e000 - 007fffb7f6f000   rw-     [mmap] libdl.so    
[x] [Thread 2000]	007fffb7f81000 - 007fffb8058000   r-x     [mmap] libc++.so   
[x] [Thread 2000]	007fffb8058000 - 007fffb8067000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb8067000 - 007fffb806e000   r--     [mmap] libc++.so   
[x] [Thread 2000]	007fffb806e000 - 007fffb806f000   rw-     [mmap] libc++.so   
[x] [Thread 2000]	007fffb806f000 - 007fffb8072000   rw-     [mmap anonymous]   
[x] [Thread 2000]	007fffb80b5000 - 007fffb80ed000   r-x     [mmap] libm.so     
[x] [Thread 2000]	007fffb80ed000 - 007fffb80fd000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb80fd000 - 007fffb80fe000   r--     [mmap] libm.so     
[x] [Thread 2000]	007fffb80fe000 - 007fffb80ff000   rw-     [mmap] libm.so     
[x] [Thread 2000]	007fffb8106000 - 007fffb8122000   r-x     [mmap] libz.so     
[x] [Thread 2000]	007fffb8122000 - 007fffb8131000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb8131000 - 007fffb8132000   r--     [mmap] libz.so     
[x] [Thread 2000]	007fffb8132000 - 007fffb8133000   rw-     [mmap] libz.so     
[x] [Thread 2000]	007fffb8160000 - 007fffb8165000   r-x     [mmap] liblog.so   
[x] [Thread 2000]	007fffb8165000 - 007fffb8174000   ---     [mmap anonymous]   
[x] [Thread 2000]	007fffb8174000 - 007fffb8175000   r--     [mmap] liblog.so   
[x] [Thread 2000]	007fffb8175000 - 007fffb8176000   rw-     [mmap] liblog.so   
...
  File "C:\Program Files\Python39\lib\site-packages\qiling\core.py", line 771, in emu_start
    self.uc.emu_start(begin, end, timeout, count)
  File "C:\Program Files\Python39\lib\site-packages\unicorn\unicorn.py", line 547, in emu_start
    raise UcError(status)
unicorn.unicorn.UcError: Invalid memory fetch (UC_ERR_FETCH_UNMAPPED)

The PC jumped to 0x0, but the memory map shows all of libxx.so’s dependencies loaded cleanly — so the linker finished loading it. Since I removed the ql_path_hook callback, breakpoints no longer fire, so I need another way to break at start.

Qiling has a “run to here” feature (Qiling → Execute Till). Looking at its IDA plugin implementation:

def ql_run_to_here(self):
    if self.qlinit:
        curr_addr = get_screen_ea()
        # 每执行一条命令前，调用ql_until_hook
        untillhook = self.qlemu.ql.hook_code(self.ql_untill_hook)
        if self.qlemu.status is not None:
            self.qlemu.ql.restore(self.qlemu.status)
            show_wait_box("Qiling is processing ...")
            try:
                self.qlemu.run(begin=self.qlemu.ql.arch.regs.arch_pc, end=curr_addr+self.qlemu.baseaddr-get_imagebase())
            finally:
                hide_wait_box()
        else:
            show_wait_box("Qiling is processing ...")
            try:
                self.qlemu.run(end=curr_addr+self.qlemu.baseaddr-get_imagebase())
            finally:
                hide_wait_box()

Qiling’s run takes end — where emulation should stop. Setting end to the start address is effectively a breakpoint at start. There’s still a per-instruction callback (line 5), so check ql_until_hook:

def ql_untill_hook(self, ql, addr, size):
    addr = addr - self.qlemu.baseaddr + get_imagebase()
    set_color(addr, CIC_ITEM, 0x00B3CBFF)

It only highlights executed code, so I can comment out this callback registration (remove the hook_code and hook_del calls for ql_until_hook in ql_run_to_here). Also, the callbacks set in custom_continue must be copied to custom_run_to_here; since custom_script.py has no custom_run_to_here, create one:

qilingida.py:

def ql_run_to_here(self):
    if self.qlinit:
        curr_addr = get_screen_ea()
        userhook = None
        # 注册新创建的custom_run_to_here方法
        if self.userobj is not None:
            userhook = self.userobj.custom_run_to_here(self.qlemu.ql)
        # 去掉ql_until_hook的回调
        # untillhook = self.qlemu.ql.hook_code(self.ql_untill_hook)
        if self.qlemu.status is not None:
            self.qlemu.ql.restore(self.qlemu.status)
            show_wait_box("Qiling is processing ...")
            try:
                self.qlemu.run(begin=self.qlemu.ql.arch.regs.arch_pc, end=curr_addr+self.qlemu.baseaddr-get_imagebase())
            finally:
                hide_wait_box()
        else:
            show_wait_box("Qiling is processing ...")
            try:
                self.qlemu.run(end=curr_addr+self.qlemu.baseaddr-get_imagebase())
            finally:
                hide_wait_box()

        set_color(curr_addr, CIC_ITEM, 0x00B3CBFF)
        # 回调删除部分也注视掉
        # self.qlemu.ql.hook_del(untillhook)
        if userhook and userhook is not None:
            for hook in userhook:
                self.qlemu.ql.hook_del(hook)
        self.qlemu.status = self.qlemu.ql.save()
        self.ql_update_views(self.qlemu.ql.arch.regs.arch_pc, self.qlemu.ql)
    else:
        logging.error('Qiling should be setup firstly.')

custom_script.py:

def custom_run_to_here(self, ql: Qiling) -> List[HookRet]:

    linker_baseaddr = 0x007ffff7dd5000

    def addr_27DA8_hook(ql: Qiling) -> None:
		# content from custom_continue

    def addr_27FB4_hook(ql: Qiling) -> None:
        # content from custom_continue

    return [ql.hook_address(addr_27DA8_hook, 0x27DA8+linker_baseaddr),
            ql.hook_address(addr_27FB4_hook, 0x27FB4+linker_baseaddr)]

No more errors, and the registers show we’ve reached start:

Reached the start function

After the fix, restart Qiling, move the cursor to start, and Qiling → Execute Till.

Next is single-stepping, which hits this:

[=] [Thread 2000]	custom_step hook
[=] [Thread 2001]	Executing: 0x7ffff7e0373c
[x] [Thread 2001]	[Thread 2001] Expect 0x5555555f9370 but get 0x7ffff7e03740 when running loader.
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\linux\thread.py", line 611, in run
    previous_thread = self._prepare_lib_patch()
  File "C:\Program Files\Python39\lib\site-packages\qiling\os\linux\thread.py", line 590, in _prepare_lib_patch
    raise QlErrorExecutionStop('Dynamic library .init() failed!')
qiling.exception.QlErrorExecutionStop: Dynamic library .init() failed!

The single-step executed 0x7ffff7e0373c instead of 0x5555555f9370. To locate this, read the Qiling source flow:

# qilingida.py
# 单步执行的入口是qlemu.run
def ql_step(self):
    self.qlemu.run(begin=self.qlemu.ql.arch.regs.arch_pc, end=self.qlemu.exit_addr)
    
# qiling/os/linux/linux.py
class QlOsLinux(QlOsPosix):
    def run(self):
        if self.ql.multithread:
            # start multithreading
            # 多线程环境下调用thread_management.run()
            thread_management = thread.QlLinuxThreadManagement(self.ql)
            self.ql.os.thread_management = thread_management
            thread_management.run()
        else:
            # 单线程环境
            if self.ql.entry_point is not None:
                self.ql.loader.elf_entry = self.ql.entry_point

            # do we have an interp?
            # 如果有动态链接器，就用动态链接器初始化目标so
            elif self.ql.loader.elf_entry != self.ql.loader.entry_point:
                entry_address = self.ql.loader.elf_entry

                if self.ql.arch.type == QL_ARCH.ARM:
                    entry_address &= ~1

                # start running interp, but stop when elf entry point is reached
                self.ql.emu_start(self.ql.loader.entry_point, entry_address, self.ql.timeout)
            
            # 开始模拟执行目标so
            self.ql.emu_start(self.ql.loader.elf_entry, self.exit_point, self.ql.timeout, self.ql.count)
        
# qiling/os/linux/thread.py
class QlLinuxThreadManagement:
    def run(self):
        # 调用_prepare_lib_patch，让动态链接器加载目标libxx.so
        previous_thread = self._prepare_lib_patch()
        
    def _prepare_lib_patch(self):
		# 如果有动态链接器，就用动态链接器初始化目标libxx.so
		if self.ql.loader.elf_entry != self.ql.loader.entry_point:
            entry_address = self.ql.loader.elf_entry

            if self.ql.arch.type == QL_ARCH.ARM:
                entry_address &= ~1

            self.main_thread = self.ql.os.thread_class.spawn(self.ql, self.ql.loader.entry_point, entry_address)
            self.cur_thread = self.main_thread
            self._clear_queued_msg()
            gevent.joinall([self.main_thread], raise_error=True)
            if self.ql.arch.regs.arch_pc != entry_address:
                self.ql.log.error(f"{self.cur_thread} Expect {hex(self.ql.loader.elf_entry)} but get {hex(self.ql.arch.regs.arch_pc)} when running loader.")
                raise QlErrorExecutionStop('Dynamic library .init() failed!')
            self.ql.do_lib_patch()
            self.ql.os.run_function_after_load()
            self.ql.loader.skip_exit_check = False
            self.ql.write_exit_trap()
            return self.main_thread
        return None

In single-threaded mode, if ql.entry_point is set (via ql.run’s begin argument), the dynamic loader is skipped and emulation starts directly. That’s fine — on the first run we don’t set begin, so the linker initializes the .so; but we register callbacks so we can save a snapshot and resume, and resuming sets ql.run’s begin. In multithreaded mode, Qiling clearly doesn’t account for this: every ql.run calls _prepare_lib_patch and re-initializes via the linker. In our case, libxx.so is already initialized and we want to single-step it, but Qiling re-initializes it, so the next instruction after single-step is linker entry + 4, not libxx.so entry + 4.

I mirror the single-threaded behavior:

# qiling/os/linux/thread.py

def _prepare_lib_patch(self):
    if self.ql.entry_point is not None:
        self.ql.loader.elf_entry = self.ql.entry_point
        return None
    elif self.ql.loader.elf_entry != self.ql.loader.entry_point:
        entry_address = self.ql.loader.elf_entry

Add a check on ql.entry_point: if ql.run set begin, skip relocation and emulate directly from begin.

Restart Qiling and Qiling → Execute Till again:

Single-step now breaks correctly

Single-stepping works now — it breaks at libxx.so’s next instruction.

Continue stepping into .__cxa_finalize. This function reads the real address of __cxa_finalize from memory and executes it; that address needs relocation since the function lives in another module. Checking its memory:

__cxa_finalize relocation

__cxa_finalize_ptr holds the function address; IDA View-B shows its RVA is 0x3266D0, and register X16 holds the page containing it. Reading memory at offset 0x6D0 gives 0x7FFFB80F942C, which is inside libc.so — so the linker correctly relocated libxx.so.

[x] [Thread 2000]	007fffb8085000 - 007fffb8148000   r-x     [mmap] libc.so

Analyzing the VM framework

libxx.so has three important interfaces:

  • [redacted]_init: initializes and collects environment info, then sends it to the server
  • [redacted]_encrypt, [redacted]_decrypt: data encryption/decryption

[redacted]_init is unrelated to the algorithm, so I skip it. The encrypt/decrypt pseudocode:

Encrypt function pseudocode

Decrypt function pseudocode

The two functions share the same structure. sub_F736C is the real encrypt/decrypt function; encryption vs. decryption only differs in the 1st, 3rd, and 4th arguments.

A quick tour of the VM

Looking at sub_F736C’s overall structure:

sub_F736C structure

It looks like several nested layers of control-flow flattening, but single-stepping reveals it’s actually a simplified VM. The flow:

  • Initialize some factors, then enter a while(1) loop. v66 reads a 4-byte unsigned int from the a1 argument, ANDs it with 0x3F, and passes it to switch.

    IDA’s decompiler got this slightly wrong — what’s passed to switch isn’t v39.

    The switch dispatch

  • A big switch:

    The switch cases

  • If v39 is 0x9, jump to the matching case and execute [vmstack-off1] = [vmstack-off2] + number, then jump to LABEL_331.

    A sample case

  • LABEL_331 is the tail of every case — like a switch where no case breaks and all fall through to the default branch.

    LABEL_331

    LABEL_331 does two things:

    • Changes control flow — either:

      • Advance the a1 array to the next element:

        a1 = (unsigned int *)(*(_QWORD *)beg + 4LL);
*(_QWORD *)beg = a1;

      • Or jump directly to some element of the a1 array:

        //func_data - 0x18地址处保存的是上一个switch-case写入的值
a1 = *(unsigned int **)(func_data - 0x18);
*(_QWORD *)(func_data - 0x20) = 0LL;
*(_QWORD *)beg = a1;

    • Calls a callback:

      real_data = ((__int64 (__fastcall *)(_QWORD, _QWORD))func)(*v13, *v14);

  • Finally, back to while(1) and the switch.

VM structure

  • The switch-case argument source: sub_F736C’s first argument is a 4-byte array; part of each 4-byte value is the switch-case argument, i.e. an operation.

    Handler argument source

    So each 4-byte value is a handler. Its internal layout — the handler index and subhandler index are the switch-case arguments:

    Handler layout

  • Handler execution examples:

    handler index:0x2B
subhandler index:0xC2B
handler：[vm_stack-off3] = [vm_stack-off1] + [vm_stack-off2]
example:
[=] [Thread 2008]    index: 0xc2b -> [func_data-0x110] = [func_data-0x80] + [func_data-0x128]; 0xffffffffff971488, 0x555555cd9eb0, rs: 0x55555564b338
------------------------------------------------
handler index:0x2B
subhandler index:0xA6B
handler:[vm_stack-off3] = [vm_stack-off1] >> operand2
example:
[=] [Thread 2008]    index: 0xa6b -> [func_data-0x128] = d[func_data-0x88] >> 0x18; 0x555f9370, rs: 0x55
------------------------------------------------
handler index:0x9
no subhandler index
handler:[vm_stack-off1] = [vm_stack-off2] + operand1
example:
[=] [Thread 2008]    index: 0x9 -> [func_data-0x120] = [func_data-0x130] + 0x4; 0x0, rs: 0x4

    In the first example, index 0x2B further checks the subindex and executes subindex 0xC2B. There are three variables — off1, off2, off3 — extracted from the handler via bit operations.

  • Control-flow changes: encryption/decryption is just iterating over the handler array, one handler at a time. For loops, the VM must change control flow. Example:

    [=] [Thread 2008]    index: 0x34 -> [func_data-0x130]; 0x0
        [func_data-0x128]; 0x1
        [func_data-0x18] = index_table_pointer+4+((int)(0xfffc << 0x10) >> 0xE); 0x5555557e5bd0
        [func_data-0x20] = 2
[=] [Thread 2008]    index: 0x9 -> [func_data-0x120] = [func_data-0x120] + 0x1; func_data-0x234, rs: func_data-0x233
        index_table_pointer = [func_data-0x18]; 0x5555557e5bc4
        [func_data-0x20] = 0
        [beg] = index_table_pointer

    When handler 0x34 runs and *(DWORD*)(func_data-0x130) != *(DWORD*)(func_data-0x128), the handler array pointer decreases by 0xC, and func_data-0x20 is set to 2 — meaning the next handler execution will jump:

    index_table_pointer+4+((int)(0xfffc << 0x10) >> 0xE) = index_table_pointer-0xC = 0x5555557e5bd0-0xC= 0x5555557e5bc4

    When the next handler 0x9 runs, since func_data-0x20 is 2, it jumps to 0x5555557e5bc4. If 0x34 saw equality, the next handler continues sequentially.

Tracing the VM

Each handler execution is one pass through the switch. Since we’ve decoded the handler’s 4-byte layout, we can record the semantics of each handler. Printing each handler’s semantics on every switch entry gives the full encrypt/decrypt flow.

So I use ql.hook_code to print the handler semantics whenever execution reaches 0xF7E20. Updating custom_script.py:

def custom_run_to_here(self, ql: Qiling) -> List[HookRet]:
	def addr_tps_F7E20_hook(ql: Qiling) -> None:
        # ql.log.info(f'index_base: {ql.arch.regs.X19:#x}')
        index = ql.arch.regs.X12 & 0x3F
        # ql.log.info(f'index_value: {index:#x}')
        # ql.log.info(f'off1: {ql.arch.regs.W8:#x}')
        # ql.log.info(f'off2: {ql.arch.regs.W9:#x}')

        indent = f'\n\t\t\t\t\t\t'
        off = calc_off(ql)
        W11 = calc_W11(ql)
        rs = f'index: {index:#x} -> '
        base_W8 = f'[func_data-{0x130-ql.arch.regs.W8*8:#x}]'
        base_W9 = f'[func_data-{0x130-ql.arch.regs.W9*8:#x}]'
        base_W10 = f'[func_data-{0x130-ql.arch.regs.W10*8:#x}]'

        global update_index_table_flag
        if update_index_table_flag == 2:
            update_index_table_flag += 1

        if index == 0x9 or index == 0x1C:
            base_W9_value = get_base_W9(ql)
            rs += f'{base_W8} = {base_W9} + {off:#x}; {get_base(ql, base_W9_value)}, rs: {get_base(ql, base_W9_value+off)}'
        elif index in [0, 0x34]:
            flag_20 = ql.arch.regs.X0
            if flag_20 != 0:
                rs += "ignore!!"
            v78 = f'(int)({off:#x} << 0x10) >> 0xE'
            v77 = get_base_W8(ql)
            rs += f'{base_W8}; {v77:#x}'
            v79 = get_base_W9(ql)
            rs += f'{indent}{base_W9}; {v79:#x}'

            str1 = f'{indent}[func_data-0x18] = index_table_pointer+8(2 elements); {ql.arch.regs.X19:#x}, rs: {ql.arch.regs.X19+8:#x}'
            str2 = f'{indent}[func_data-0x18] = index_table_pointer+4+({v78}); {ql.arch.regs.X19:#x}'
            if index == 0:
                if v77 != v79:
                    rs += str1
                else:
                    rs += str2

            if index == 0x34:
                if v77 == v79:
                    rs += str1
                else:
                    rs += str2
		# elif...

        if log_flag:
            ql.log.info(rs)

    return [ql.hook_address(addr_27DA8_hook, 0x7ffff7dfcda8),
            ql.hook_address(addr_27FB4_hook, 0x7ffff7dfcfb4),
            ql.hook_address(addr_tps_ADAB8_hook, 0x555555601ab8),
            ql.hook_address(addr_tps_F7E20_hook, 0x55555564be20)]

  • The complete custom_script.py is attached to the original post.

  • Some switch-case paths never execute, so the handler set isn’t the full set. When writing handlers, add one elif at a time — when a new handler appears, add it. No need to write out every path upfront.

  • Adding a handler means restarting Qiling, which is slow. When an unrecorded handler appears, call ql.os.stop() manually so ql_run_to_here saves the current state and we can still single-step:

    # qilingida.py
def ql_run_to_here(self):
    # ...
    userhook = None
    if self.userobj is not None:
        userhook = self.userobj.custom_run_to_here(self.qlemu.ql)
    # 当qlemu.run方法在模拟执行时，如果custom_run_to_here调用了ql.os.stop(),
    # 那么qlemu.run就会返回。
    self.qlemu.run(end=curr_addr+self.qlemu.baseaddr-get_imagebase())
    # qlemu.run返回后保存当前模拟执行环境。
    self.qlemu.status = self.qlemu.ql.save()

# custom_script.py
def custom_run_to_here(self, ql: Qiling) -> List[HookRet]:
    def next_pc(ql: Qiling, index = ""):
        next_pc = ql.arch.regs.X1 - 0x555555554000
        index = ql.arch.regs.X12 & 0x3F
        rs = f'next unknown index: {index:#x} -> '
        ql.log.info(rs + f'next pc: {next_pc:#x}')
        ql.os.stop()
        global log_flag
        log_flag = False

Sample emulation output:

[=] [Thread 2008]	index: 0x9 -> [func_data-0x48] = [func_data-0x48] + 0xfea0; func_data-0x150, rs: func_data--0xfd50
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x158] = [func_data-0x38]; func_data-0x158, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x150] = [func_data-0x80]; func_data-0x160, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x148] = [func_data-0x88]; func_data-0x168, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x140] = [func_data-0x90]; func_data-0x170, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x138] = [func_data-0x98]; func_data-0x178, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x130] = [func_data-0xa0]; func_data-0x180, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x128] = [func_data-0xa8]; func_data-0x188, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x120] = [func_data-0xb0]; func_data-0x190, 0x0
[=] [Thread 2008]	index: 0x3c -> [func_data-0xb0] = [[func_data-0x110] + 0x10]; func_data-0x450 -> func_data-0x440, 0x20
[=] [Thread 2008]	index: 0x3c -> [func_data-0xa8] = [[func_data-0x110] + 0x8]; func_data-0x450 -> func_data-0x448, 0x100000
[=] [Thread 2008]	index: 0x3c -> [func_data-0x90] = [[func_data-0x108] + 0x0]; 0x555555857110 -> 0x555555857110, 0x555555f0abb0
[=] [Thread 2008]	index: 0x35 -> [func_data-0x88] = (signed )d[[func_data-0x110] + 0x0]; func_data-0x450, 0x555f9370
[=] [Thread 2008]	index: 0x3c -> [func_data-0x128] = [[func_data-0x100] + 0x0]; 0x555555857118 -> 0x555555857118, 0x555555cd9eb0
[=] [Thread 2008]	index: 0x2d -> [func_data-0x98] = d[func_data-0x130] + (int64)(int)0x0; 0x0, rs: 0x0
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x48] + 0x110] = b[func_data-0x130]; func_data-0x1a0, 0x0
[=] [Thread 2008]	index: 0x9 -> [func_data-0xa0] = [func_data-0x48] + 0x4; func_data-0x2b0, rs: func_data-0x2ac
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x108] = [func_data-0xa0]; func_data-0x1a8, func_data-0x2ac
[=] [Thread 2008]	index: 0x9 -> [func_data-0x120] = [func_data-0x130] + 0x4; 0x0, rs: 0x4
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x118] = [func_data-0x120]; func_data-0x198, 0x4
[=] [Thread 2008]	index: 0x1f -> [func_data-0x120] = int(0xff97 << 0x10)
[=] [Thread 2008]	index: 0x1d -> [func_data-0x80] = [func_data-0x120] | 0x1488; 0xffffffffff970000, rs:0xffffffffff971488
[=] [Thread 2008]	index: 0xc2b -> [func_data-0x110] = [func_data-0x80] + [func_data-0x128]; 0xffffffffff971488, 0x555555cd9eb0, rs: 0x55555564b338
[=] [Thread 2008]	index: 0x92b -> [func_data-0x68] = [func_data-0x130] | [func_data-0xf8]; 0x0, 0x55555564b34c, rs: 0x55555564b34c
[=] [Thread 2008]	index: 0xdeb -> [func_data-0x18] = [func_data-0x68]; 0x55555564b34c
						[func_data-0x20] = 2
						[func_data-0x38] = index_table_pointer+8; 0x5555557e5b90, rs: 0x5555557e5b98
						[func_data-0x10] = index_table_pointer+8; 0x5555557e5b90, rs: 0x5555557e5b98
[=] [Thread 2008]	index: 0x9 -> [func_data-0x108] = [func_data-0x48] + 0x108; func_data-0x2b0, rs: func_data-0x1a8
						index_table_pointer = [func_data-0x18]; 0x55555564b34c
						[func_data-0x20] = 0
						[beg] = index_table_pointer
						call func:(0x55555564b34c)([func_data-0x110], [func_data-0x108]); memset
						index_table_pointer = [func_data-0x10]; 0x5555557e5b98
						[beg] = index_table_pointer
[=] [Thread 2008]	index: 0xa6b -> [func_data-0x128] = d[func_data-0x88] >> 0x18; 0x555f9370, rs: 0x55
[=] [Thread 2008]	index: 0xa6b -> [func_data-0x120] = d[func_data-0x88] >> 0x10; 0x555f9370, rs: 0x555f
[=] [Thread 2008]	index: 0xa6b -> [func_data-0x118] = d[func_data-0x88] >> 0x8; 0x555f9370, rs: 0x555f93
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x48] + 0x5] = b[func_data-0x118]; func_data-0x2ab, 0x93
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x48] + 0x4] = b[func_data-0x88]; func_data-0x2ac, 0x70
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x48] + 0x6] = b[func_data-0x120]; func_data-0x2aa, 0x5f
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x48] + 0x7] = b[func_data-0x128]; func_data-0x2a9, 0x55
[=] [Thread 2008]	index: 0xc2b -> [func_data-0x118] = [func_data-0x80] + [func_data-0x90]; 0xffffffffff971488, 0x555555f0abb0, rs: 0x55555587c038
[=] [Thread 2008]	index: 0x38 -> [func_data-0x128] = [func_data-0x98] < 0x100; 0x0, rs: 0x1

Recovering the algorithm

Reviewing all handler execution records before sub_F736C returns, they fall into four parts:

The full handler execution process.txt is attached to the original post.

Environment initialization

The VM stack is initialized and sub_F736C’s arguments are saved to it:

[=] [Thread 2008]	index: 0x9 -> [func_data-0x48] = [func_data-0x48] + 0xfea0; func_data-0x150, rs: func_data--0xfd50
// here the comment after the semicolon is auto-generated, showing the memory values left-to-right. e.g. the next line: [func_data-0x48] is func_data-0x158; [func_data-0x38] is 0. since this is a write, [[func_data-0x48]+0x158]'s value isn't printed.
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x158] = [func_data-0x38]; func_data-0x158, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x150] = [func_data-0x80]; func_data-0x160, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x148] = [func_data-0x88]; func_data-0x168, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x140] = [func_data-0x90]; func_data-0x170, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x138] = [func_data-0x98]; func_data-0x178, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x130] = [func_data-0xa0]; func_data-0x180, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x128] = [func_data-0xa8]; func_data-0x188, 0x0
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x120] = [func_data-0xb0]; func_data-0x190, 0x0
[=] [Thread 2008]	index: 0x3c -> [func_data-0xb0] = [[func_data-0x110] + 0x10]; func_data-0x450 -> func_data-0x440, 0x20
[=] [Thread 2008]	index: 0x3c -> [func_data-0xa8] = [[func_data-0x110] + 0x8]; func_data-0x450 -> func_data-0x448, 0x100000
[=] [Thread 2008]	index: 0x3c -> [func_data-0x90] = [[func_data-0x108] + 0x0]; 0x555555857110 -> 0x555555857110, 0x555555f0abb0
[=] [Thread 2008]	index: 0x35 -> [func_data-0x88] = (signed )d[[func_data-0x110] + 0x0]; func_data-0x450, 0x555f9370
[=] [Thread 2008]	index: 0x3c -> [func_data-0x128] = [[func_data-0x100] + 0x0]; 0x555555857118 -> 0x555555857118, 0x555555cd9eb0
[=] [Thread 2008]	index: 0x2d -> [func_data-0x98] = d[func_data-0x130] + (int64)(int)0x0; 0x0, rs: 0x0
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x48] + 0x110] = b[func_data-0x130]; func_data-0x1a0, 0x0
[=] [Thread 2008]	index: 0x9 -> [func_data-0xa0] = [func_data-0x48] + 0x4; func_data-0x2b0, rs: func_data-0x2ac
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x108] = [func_data-0xa0]; func_data-0x1a8, func_data-0x2ac
[=] [Thread 2008]	index: 0x9 -> [func_data-0x120] = [func_data-0x130] + 0x4; 0x0, rs: 0x4
[=] [Thread 2008]	index: 0x29 -> [[func_data-0x48]+0x118] = [func_data-0x120]; func_data-0x198, 0x4
[=] [Thread 2008]	index: 0x1f -> [func_data-0x120] = int(0xff97 << 0x10)
[=] [Thread 2008]	index: 0x1d -> [func_data-0x80] = [func_data-0x120] | 0x1488; 0xffffffffff970000, rs:0xffffffffff971488
[=] [Thread 2008]	index: 0xc2b -> [func_data-0x110] = [func_data-0x80] + [func_data-0x128]; 0xffffffffff971488, 0x555555cd9eb0, rs: 0x55555564b338
[=] [Thread 2008]	index: 0x92b -> [func_data-0x68] = [func_data-0x130] | [func_data-0xf8]; 0x0, 0x55555564b34c, rs: 0x55555564b34c
[=] [Thread 2008]	index: 0xdeb -> [func_data-0x18] = [func_data-0x68]; 0x55555564b34c
						[func_data-0x20] = 2
						[func_data-0x38] = index_table_pointer+8; 0x5555557e5b90, rs: 0x5555557e5b98
						[func_data-0x10] = index_table_pointer+8; 0x5555557e5b90, rs: 0x5555557e5b98
[=] [Thread 2008]	index: 0x9 -> [func_data-0x108] = [func_data-0x48] + 0x108; func_data-0x2b0, rs: func_data-0x1a8
						index_table_pointer = [func_data-0x18]; 0x55555564b34c
						[func_data-0x20] = 0
						[beg] = index_table_pointer
						call func:(0x55555564b34c)([func_data-0x110], [func_data-0x108]); memset
						index_table_pointer = [func_data-0x10]; 0x5555557e5b98
						[beg] = index_table_pointer

Preparing a 256-byte S-box

Read a byte from [func_data-0x98], write it to func_data-0x2a8, and increment both [func_data-0x98] and [func_data-0x120] until [func_data-0x98] == 0x100:

[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x120] + 0x0] = b[func_data-0x98]; func_data-0x2a8, 0x0
[=] [Thread 2008]	index: 0x2d -> [func_data-0x98] = d[func_data-0x98] + (int64)(int)0x1; 0x0, rs: 0x1
[=] [Thread 2008]	index: 0x38 -> [func_data-0x128] = [func_data-0x98] < 0x100; 0x1, rs: 0x1
[=] [Thread 2008]	index: 0x34 -> [func_data-0x130]; 0x0
						[func_data-0x128]; 0x1
						[func_data-0x18] = index_table_pointer+4+((int)(0xfffc << 0x10) >> 0xE); 0x5555557e5bd0
						[func_data-0x20] = 2
[=] [Thread 2008]	index: 0x9 -> [func_data-0x120] = [func_data-0x120] + 0x1; func_data-0x2a8, rs: func_data-0x2a7
						index_table_pointer = [func_data-0x18]; 0x5555557e5bc4
						[func_data-0x20] = 0
						[beg] = index_table_pointer
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x120] + 0x0] = b[func_data-0x98]; func_data-0x2a7, 0x1
[=] [Thread 2008]	index: 0x2d -> [func_data-0x98] = d[func_data-0x98] + (int64)(int)0x1; 0x1, rs: 0x2
[=] [Thread 2008]	index: 0x38 -> [func_data-0x128] = [func_data-0x98] < 0x100; 0x2, rs: 0x1
[=] [Thread 2008]	index: 0x34 -> [func_data-0x130]; 0x0
						[func_data-0x128]; 0x1
						[func_data-0x18] = index_table_pointer+4+((int)(0xfffc << 0x10) >> 0xE); 0x5555557e5bd0
						[func_data-0x20] = 2
[=] [Thread 2008]	index: 0x9 -> [func_data-0x120] = [func_data-0x120] + 0x1; func_data-0x2a7, rs: func_data-0x2a6
						index_table_pointer = [func_data-0x18]; 0x5555557e5bc4
						[func_data-0x20] = 0
						[beg] = index_table_pointer
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x120] + 0x0] = b[func_data-0x98]; func_data-0x2a6, 0x2
[=] [Thread 2008]	index: 0x2d -> [func_data-0x98] = d[func_data-0x98] + (int64)(int)0x1; 0x2, rs: 0x3
[=] [Thread 2008]	index: 0x38 -> [func_data-0x128] = [func_data-0x98] < 0x100; 0x3, rs: 0x1
[=] [Thread 2008]	index: 0x34 -> [func_data-0x130]; 0x0
						[func_data-0x128]; 0x1
						[func_data-0x18] = index_table_pointer+4+((int)(0xfffc << 0x10) >> 0xE); 0x5555557e5bd0
						[func_data-0x20] = 2
[=] [Thread 2008]	index: 0x9 -> [func_data-0x120] = [func_data-0x120] + 0x1; func_data-0x2a6, rs: func_data-0x2a5
						index_table_pointer = [func_data-0x18]; 0x5555557e5bc4
						[func_data-0x20] = 0
						[beg] = index_table_pointer

In C++:

#define SHUFFLE_TABLE_LEN 256

unsigned char shuffle_table[SHUFFLE_TABLE_LEN] = { 0 };
for (int i = 0; i < SHUFFLE_TABLE_LEN; ++i)
    shuffle_table[i] = i;

S-box permutation

Iterate the S-box, i from 0 to 255:

// read the i-th element of the S-box
[=] [Thread 2008]	index: 0x72b -> [func_data-0x128] = (int)d[func_data-0x120] >> 0x1f; 0x0
[=] [Thread 2008]	index: 0xa6b -> [func_data-0xe0] = d[func_data-0x128] >> 0x1b; 0x0, rs: 0x0
[=] [Thread 2008]	index: 0xcab -> [func_data-0xe0] = d[func_data-0xe0] + d[func_data-0x120]; 0x0, 0x0, rs: 0x0
[=] [Thread 2008]	index: 0x9eb -> [func_data-0xe0] = [func_data-0x108] & [func_data-0xe0]; 0xffffffffffffffe0, 0x0, rs: 0x0
[=] [Thread 2008]	index: 0xb2b -> [func_data-0xe0] = d[func_data-0x120] - d[func_data-0xe0]; 0x0, 0x0, rs: 0x0
[=] [Thread 2008]	index: 0x4ab -> [func_data-0xe0] = d[func_data-0xe0] << 0x0; 0x0
[=] [Thread 2008]	index: 0x9 -> [func_data-0xd8] = [func_data-0xf0] + 0x1; func_data-0x2a8, rs: func_data-0x2a7
[=] [Thread 2008]	index: 0x30 -> [func_data-0xd0] = b[[func_data-0xf0] + 0x0]; func_data-0x2a8, 0x0

// the i-th S-box element plus the previous result: a+last_result
// last_result starts at 0
[=] [Thread 2008]	index: 0xcab -> [func_data-0xe8] = d[func_data-0xd0] + d[func_data-0xe8]; 0x0, 0x0, rs: 0x0

// [func_data-0x118] is sub_F736C's 3rd argument: a 0x20-byte array of 1-byte elements, named shuffle_factor_table here.
// take (i%0x20) from shuffle_factor_table, then add to the previous step:
// a+last_result+shuffle_factor_table[i%0x20]
[=] [Thread 2008]	index: 0x2d -> [func_data-0xc8] = d[func_data-0x120] + (int64)(int)0x1; 0x0, rs: 0x1
[=] [Thread 2008]	index: 0xa6b -> [func_data-0x128] = d[func_data-0x128] >> 0x1e; 0x0, rs: 0x0
[=] [Thread 2008]	index: 0xc2b -> [func_data-0xe0] = [func_data-0xe0] + [func_data-0x118]; 0x0, 0x55555587c038, rs: 0x55555587c038
[=] [Thread 2008]	index: 0x30 -> [func_data-0xe0] = b[[func_data-0xe0] + 0x0]; 0x55555587c038, 0x48
[=] [Thread 2008]	index: 0xcab -> [func_data-0xe8] = d[func_data-0xe0] + d[func_data-0xe8]; 0x48, 0x0, rs: 0x48

// the first argument of [redacted]_encrypt and [redacted]_decrypt is 'serino', stored little-endian, then take one byte and add to the previous step:
// a+last_result+shuffle_factor_table[i%0x20]+serino[i%4]
[=] [Thread 2008]	index: 0xcab -> [func_data-0x128] = d[func_data-0x128] + d[func_data-0x120]; 0x0, 0x0, rs: 0x0
[=] [Thread 2008]	index: 0x9eb -> [func_data-0x128] = [func_data-0xf8] & [func_data-0x128]; 0xfffffffffffffffc, 0x0, rs: 0x0
[=] [Thread 2008]	index: 0xb2b -> [func_data-0x128] = d[func_data-0x120] - d[func_data-0x128]; 0x0, 0x0, rs: 0x0
[=] [Thread 2008]	index: 0x4ab -> [func_data-0x128] = d[func_data-0x128] << 0x0; 0x0
[=] [Thread 2008]	index: 0xc2b -> [func_data-0x128] = [func_data-0x128] + [func_data-0xa0]; 0x0, func_data-0x2ac, rs: func_data-0x2ac
[=] [Thread 2008]	index: 0x30 -> [func_data-0x128] = b[[func_data-0x128] + 0x0]; func_data-0x2ac, 0x70
[=] [Thread 2008]	index: 0xcab -> [func_data-0x128] = d[func_data-0x128] + d[func_data-0xe8]; 0x70, 0x48, rs: 0xb8

// modulo 0xFF on the previous result
// last_result = (a+last_result+shuffle_factor_table[i%0x20]+serino[i%4]) & 0xFF
[=] [Thread 2008]	index: 0x72b -> [func_data-0x120] = (int)d[func_data-0x128] >> 0x1f; 0xb8
[=] [Thread 2008]	index: 0xa6b -> [func_data-0x120] = d[func_data-0x120] >> 0x18; 0x0, rs: 0x0
[=] [Thread 2008]	index: 0xcab -> [func_data-0x120] = d[func_data-0x120] + d[func_data-0x128]; 0x0, 0xb8, rs: 0xb8
[=] [Thread 2008]	index: 0x9eb -> [func_data-0x120] = [func_data-0x100] & [func_data-0x120]; 0xffffffffffffff00, 0xb8, rs: 0x0
[=] [Thread 2008]	index: 0xb2b -> [func_data-0xe8] = d[func_data-0x128] - d[func_data-0x120]; 0xb8, 0x0, rs: 0xb8
[=] [Thread 2008]	index: 0x4ab -> [func_data-0x128] = d[func_data-0xe8] << 0x0; 0xb8

// S-box permutation
// tmp = S-box[i]
// S-box[i] = S-box[last_result]
// S-box[last_result] = tmp
[=] [Thread 2008]	index: 0xc2b -> [func_data-0x128] = [func_data-0x128] + [func_data-0x110]; 0xb8, func_data-0x2a8, rs: func_data-0x1f0
[=] [Thread 2008]	index: 0x30 -> [func_data-0x120] = b[[func_data-0x128] + 0x0]; func_data-0x1f0, 0xb8
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0xf0] + 0x0] = b[func_data-0x120]; func_data-0x2a8, 0xb8
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x128] + 0x0] = b[func_data-0xd0]; func_data-0x1f0, 0x0
[=] [Thread 2008]	index: 0x92b -> [func_data-0x120] = [func_data-0x130] | [func_data-0xc8]; 0x0, 0x1, rs: 0x1

// not yet iterated to 0x256, jump back to the first S-box permutation instruction and continue
[=] [Thread 2008]	index: 0x38 -> [func_data-0x128] = [func_data-0x120] < 0x100; 0x1, rs: 0x1
[=] [Thread 2008]	index: 0x34 -> [func_data-0x130]; 0x0
						[func_data-0x128]; 0x1
						[func_data-0x18] = index_table_pointer+4+((int)(0xffde << 0x10) >> 0xE); 0x5555557e5c80
						[func_data-0x20] = 2
[=] [Thread 2008]	index: 0x92b -> [func_data-0xf0] = [func_data-0x130] | [func_data-0xd8]; 0x0, 0x80000000db39, rs: func_data-0x2a7
						index_table_pointer = [func_data-0x18]; 0x5555557e5bfc
						[func_data-0x20] = 0
						[beg] = index_table_pointer

In C++:

void Fisher_Yates(unsigned char shuffle_table[SHUFFLE_TABLE_LEN], unsigned int serino) {

    unsigned char shuffle_factor_table[SHUFFLE_FACTOR_TABLE_LEN] = { 
        0x48, 0xA9, 0xC8, 0x12, 0xCA, 0xFC, 0xD3, 0x5E, 0xB7, 0x61, 0x50, 0x17, 0x68, 0xBA, 0x7E, 0x2E, 
        0xB9, 0xA2, 0x38, 0x85, 0x35, 0x48, 0x55, 0x6C, 0x2C, 0x38, 0x43, 0x1F, 0x51, 0xD6, 0x30, 0x30 };

    unsigned char last_result = 0;
    unsigned char serino_table[] = { serino & 0xFF, (serino >> 8) & 0xFF, (serino >> 16) & 0xFF, (serino >> 24) & 0xFF };

    for (int i = 0; i < SHUFFLE_TABLE_LEN; ++i) {
        last_result = (shuffle_factor_table[i % SHUFFLE_FACTOR_TABLE_LEN] + shuffle_table[i] + last_result + serino_table[i % 4]) & 0xFF;
        unsigned char tmp = shuffle_table[i];
        shuffle_table[i] = shuffle_table[last_result];
        shuffle_table[last_result] = tmp;
        //printf("exchange: %x -> %x, factor: %x\n", i, last_result, shuffle_factor_table[i%SHUFFLE_FACTOR_TABLE_LEN]);
    }
}

Data encrypt/decrypt

Iterate over the data, i from 0 to len(data)-1:

// a = (i+1) & 0xFF
[=] [Thread 2008]	index: 0x2d -> [func_data-0x128] = d[func_data-0x120] + (int64)(int)0x1; 0x100, rs: 0x101
[=] [Thread 2008]	index: 0x72b -> [func_data-0x120] = (int)d[func_data-0x128] >> 0x1f; 0x101
[=] [Thread 2008]	index: 0xa6b -> [func_data-0x120] = d[func_data-0x120] >> 0x18; 0x0, rs: 0x0
[=] [Thread 2008]	index: 0xcab -> [func_data-0x120] = d[func_data-0x120] + d[func_data-0x128]; 0x0, 0x101, rs: 0x101
[=] [Thread 2008]	index: 0x9eb -> [func_data-0x120] = [func_data-0x118] & [func_data-0x120]; 0xffffffffffffff00, 0x101, rs: 0x100
[=] [Thread 2008]	index: 0xb2b -> [func_data-0x120] = d[func_data-0x128] - d[func_data-0x120]; 0x101, 0x100, rs: 0x1
[=] [Thread 2008]	index: 0x4ab -> [func_data-0x128] = d[func_data-0x120] << 0x0; 0x1

// func_data-0xa8 is the data to encrypt/decrypt, named x here
// save the current byte x[i] to [func_data-0x100]
[=] [Thread 2008]	index: 0x9 -> [func_data-0xf8] = [func_data-0x100] + 0x1; 0x0, rs: 0x1
[=] [Thread 2008]	index: 0xc2b -> [func_data-0x100] = [func_data-0x100] + [func_data-0xa8]; 0x0, 0x100000, rs: 0x100000

// func_data-0x110 holds the S-box address
// take one byte from the permuted S-box at index a, add the previous result
// b = S-box[a]+last_result
[=] [Thread 2008]	index: 0xc2b -> [func_data-0x128] = [func_data-0x128] + [func_data-0x110]; 0x1, func_data-0x2a8, rs: func_data-0x2a7
[=] [Thread 2008]	index: 0x30 -> [func_data-0xf0] = b[[func_data-0x128] + 0x0]; func_data-0x2a7, 0xa9
[=] [Thread 2008]	index: 0xcab -> [func_data-0x108] = d[func_data-0xf0] + d[func_data-0x108]; 0xa9, 0x0, rs: 0xa9
[=] [Thread 2008]	index: 0x72b -> [func_data-0xe8] = (int)d[func_data-0x108] >> 0x1f; 0xa9
[=] [Thread 2008]	index: 0xa6b -> [func_data-0xe8] = d[func_data-0xe8] >> 0x18; 0x0, rs: 0x0
[=] [Thread 2008]	index: 0xcab -> [func_data-0xe8] = d[func_data-0xe8] + d[func_data-0x108]; 0x0, 0xa9, rs: 0xa9
[=] [Thread 2008]	index: 0x9eb -> [func_data-0xe8] = [func_data-0x118] & [func_data-0xe8]; 0xffffffffffffff00, 0xa9, rs: 0x0
[=] [Thread 2008]	index: 0xb2b -> [func_data-0x108] = d[func_data-0x108] - d[func_data-0xe8]; 0xa9, 0x0, rs: 0xa9
[=] [Thread 2008]	index: 0x4ab -> [func_data-0xe8] = d[func_data-0x108] << 0x0; 0xa9

// S-box permutation
// tmp = S-box[a]
// S-box[a] = S-box[b]
// S-box[b] = S-box[a]
[=] [Thread 2008]	index: 0xc2b -> [func_data-0xe8] = [func_data-0xe8] + [func_data-0x110]; 0xa9, func_data-0x2a8, rs: func_data-0x1ff
[=] [Thread 2008]	index: 0x30 -> [func_data-0xe0] = b[[func_data-0xe8] + 0x0]; func_data-0x1ff, 0xc1
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x128] + 0x0] = b[func_data-0xe0]; func_data-0x2a7, 0xc1
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0xe8] + 0x0] = b[func_data-0xf0]; func_data-0x1ff, 0xa9

// x[i] = x[i] ^ ((S-box[a] + tmp) & 0xFF)
[=] [Thread 2008]	index: 0x30 -> [func_data-0x128] = b[[func_data-0x128] + 0x0]; func_data-0x2a7, 0xc1
[=] [Thread 2008]	index: 0xcab -> [func_data-0x128] = d[func_data-0xf0] + d[func_data-0x128]; 0xa9, 0xc1, rs: 0x16a
[=] [Thread 2008]	index: 0x30 -> [func_data-0xf0] = b[[func_data-0x100] + 0x0]; 0x100000, 0x12
[=] [Thread 2008]	index: 0x37 -> [func_data-0x128] = [func_data-0x128] & 0xff; 0x16a, rs:0x6a
[=] [Thread 2008]	index: 0x7 -> [func_data-0x128] = ((1(LL) << (W10+1)) - 1) & ([func_data-0x128] >> W11); 0x1f, 0x6a, 0x0, rs: 0x6a
[=] [Thread 2008]	index: 0xc2b -> [func_data-0x128] = [func_data-0x128] + [func_data-0x110]; 0x6a, func_data-0x2a8, rs: func_data-0x23e
[=] [Thread 2008]	index: 0x30 -> [func_data-0x128] = b[[func_data-0x128] + 0x0]; func_data-0x23e, 0xef
[=] [Thread 2008]	index: 0x36b -> [func_data-0x128] = [func_data-0x128] ^ [func_data-0xf0]; 0xef, 0x12, rs: 0xfd
[=] [Thread 2008]	index: 0x2a -> b[[func_data-0x100] + 0x0] = b[func_data-0x128]; 0x100000, 0xfd

// not yet reached x's length, continue iterating
[=] [Thread 2008]	index: 0x92b -> [func_data-0x100] = [func_data-0x130] | [func_data-0xf8]; 0x0, 0x1, rs: 0x1
[=] [Thread 2008]	index: 0x1eb -> base_W9_value < base_W8_value: [func_data-0x128] = 1; w8=[func_data-0xb0]=32, w9=[func_data-0x100]=1
[=] [Thread 2008]	index: 0x34 -> [func_data-0x130]; 0x0
						[func_data-0x128]; 0x1
						[func_data-0x18] = index_table_pointer+4+((int)(0xffde << 0x10) >> 0xE); 0x5555557e5d24
						[func_data-0x20] = 2
[=] [Thread 2008]	index: 0x4ab -> [func_data-0x130] = d[func_data-0x130] << 0x0; 0x0
						index_table_pointer = [func_data-0x18]; 0x5555557e5ca0
						[func_data-0x20] = 0
						[beg] = index_table_pointer

In C++:

void encrypt_and_decrypt_data(unsigned char shuffle_table[SHUFFLE_TABLE_LEN], unsigned char* x, int len) {

    unsigned char last_result = 0;

    for (int i = 0; i < len; ++i) {

        int s_i = (i + 1) & 0xFF;
        unsigned char cur = shuffle_table[s_i];

        // last_result = (last_result + cur) & 0xFF;
        last_result += cur; // no need to 'and 0xFF', because the type of last_result is unsigned char

        shuffle_table[s_i] = shuffle_table[last_result];
        shuffle_table[last_result] = cur;

        unsigned char tmp = (cur + shuffle_table[s_i]) & 0xFF;
        x[i] = x[i] ^ shuffle_table[tmp];
    }
}

Conclusion

This was my first time using Qiling for emulation, and I hit plenty of pitfalls. With those filled in, future emulation will be much smoother.

libxx.so’s algorithm is simple once you see it: it’s RC4, with a small twist — the key is split into two parts. The main protection is the VM, with a bit of junk instructions and dynamic string decryption, all of which are simple in libxx.so, so I didn’t go into them.

I hope this helps anyone who wants to use Qiling for emulation skip past some of its rough edges.

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch