About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes.

Bypassing a Mobile Game’s mmap-Modular Anti-Hook Protection

Preface

I analyzed this sample roughly four times in total, and the first three attempts all ended in failure. Maybe for ordinary people, failure really is the main theme of life — a feeling that grows stronger the more I work in reverse engineering.

The main target of this article is Frida/hook detection.

The crash

After hooking with Frida, the app crashes immediately. Hooking dlopen shows that the crash happens while loading lib__6dba__.so — specifically inside that library’s .init_array. And the .init_array contains only a single start function.

After hooking once with Frida, even if you don’t hook the next time, the app still crashes on a normal launch. It most likely also fingerprints Frida via the maps file.

Analyzing start

At the beginning, it calls get_custom_scetion to fetch the encrypted data inside lib__6dba__.so.

Fig. 1

The concrete implementation:

First it opens and reads lib__6dba__.so using the openat, lseek, and read syscalls, then walks the program headers to find the end address of the last loadable segment, which we’ll call last_loadseg_end.

Fig. 2

Looking at the data pointed to by last_loadseg_end in 010 Editor, it’s clearly high-entropy data. We’ll call it enc_data.

Fig. 3

Continuing down, it walks the section header table to locate a custom section.

Fig. 4

In 010 Editor, that section also points to the same area around last_loadseg_end.

Fig. 5

It’s not obvious why it locates enc_data via both the program headers and the section headers, but in short, get_custom_scetion just returns enc_data.

Back in start, after getting enc_data, it calls decrypt1 and decrypt2 to decrypt it.

Fig. 6

The decrypted data is actually executable logic. Because it’s run by mmap-ing it and granting execute permission with mprotect, I call these “mmap modules”, numbered in creation order as mmap1, mmap2, and so on. The Frida detection logic is obviously among them.

Fig. 7

Note: this protection makes heavy use of raw syscalls (the mmap and mprotect above are syscalls, not libc wrappers). Basic functions like strcpy, strlen, and memset are all self-implemented.

Hooking & dumping the mmap modules

I first tried dynamic debugging on mmap1, and found that mmap1 creates and invokes mmap2, mmap3, and mmap4. Likewise mmap2–mmap4 each create and invoke further mmap modules, which makes dynamic debugging painful (mainly because comments, renamed variables, and renamed functions inside the mmap modules can’t be persisted).

But dynamic debugging wasn’t fruitless — it revealed a few things:

1. Every mmap module has a very similar structure (you’ll understand what I mean after debugging one).
2. Most functions across mmap modules are implemented identically, such as the string-decryption function.
3. Every mmap module wraps its syscalls, which makes them very convenient to hook.
4. Every mmap module creates and invokes the next mmap module the same way — through the mmap + mprotect syscalls.

Since dynamic debugging was hard, I switched to a pure hooking approach. But first I had to dump all the mmap modules. There are 13 mmap modules created before the game crashes.

You can dump and trace all the mmap modules with Frida or QBDI. Dumps are named mmap_<base>_<size>_<idx>.bin, and the trace goes to log.txt (mainly recording the function call graph, which QBDI makes easy).

Then, matching byte signatures identifies mmap1–mmap13 and gives me each one’s base address for hooking. Here’s an example of hooking mmap1–mmap4:

let hooked = false;
let mmap_history = {}
function hook_func_init(soName) {
    if (hooked) return;
    hooked = true;

    function hook_syscall() {
        function is_mmap1 (addr) {
            let byte_arr = [
                0xF0, 0x7B, 0xBF, 0xA9, 0x30, 0x01, 0x00, 0xB0, 0x11, 0x86, 
                0x42, 0xF9, 0x10, 0x22, 0x14, 0x91, 0x20, 0x02, 0x1F, 0xD6
            ]
            let offset = 0x440;
            for(let i = 0; i < byte_arr.length; i++) {
                if (addr.add(offset).add(i).readU8() != byte_arr[i]) return false;
            }
            return true;
        }
        
        function hook_mmap1(mmap_base) {
            Interceptor.attach(mmap_base.add(0xF9B0), {
                onEnter: function(args) {
                    this.sysno = args[7];
                    this.a0 = args[0]
                    this.a1 = args[1]
                    this.a2 = args[2]
                },
                onLeave: function(retval) {
                    if (this.sysno == 0xde) {
                        // console.log("[hook_mmap1_syscall] mmap addr: ", retval, "size: ", this.a1, "prot: ", this.a2);
                        mmap_history[retval] = this.a1;
                    }
    
                    if (this.sysno == 0xe2) {
                        console.log("[hook_mmap1_syscall] mprotect addr: ", this.a0, "size: ", this.a1 ,"prot: ", this.a2);
                        if (mmap_history[this.a0]) {
                            console.log(`\t[hook_mmap1_syscall] mmap addr: ${this.a0}  size: ${mmap_history[this.a0]}`);
                        }
                        if (is_mmap2(this.a0)) {
                            hook_mmap2(this.a0);
                        }
                        if (is_mmap3(this.a0)) {
                            hook_mmap3(this.a0);
                        }
                        if (is_mmap4(this.a0)) {
                            hook_mmap4(this.a0);
                        }
                    }
                }
            })
        }

        Interceptor.attach(base.add(0x5C84), {
            onEnter: function(args) {
                // console.log("[svc] sysno: ", args[7]);
                this.sysno = args[7];
                this.a0 = args[0]
                this.a1 = args[1]
                this.a2 = args[2]
            },
            onLeave: function(retval) {
                if (this.sysno == 0xde) {
                    // console.log("mmap addr: ", retval, "size: ", this.a1, "prot: ", this.a2);
                    mmap_history[retval] = this.a1;
                }

                if (this.sysno == 0xe2) {
                    console.log("[syscall] mprotect addr: ", this.a0, "size: ", this.a1 ,"prot: ", this.a2);
                    if (mmap_history[this.a0]) {
                        console.log(`\t[syscall] mmap addr: ${this.a0}  size: ${mmap_history[this.a0]}`);
                    }
                    if (is_mmap1(this.a0)) {
                        hook_mmap1(this.a0);
                    }
                }
            }
        })
    }

    var base = Module.findBaseAddress(soName);
    console.log("[hook_func_init] base: ", base);

    hook_syscall();

}

Analyzing the mmap13 module

The last module before the crash is mmap13, which most likely contains the Frida detection, so I focused on it.

The basics

First find the string-decryption function, whose signature is as follows — its return value is the decrypted string:

Fig. 8

Hooking it produces:

[hook_mmap13_decrypt_str] retval:  %s/lib
[hook_mmap13_decrypt_str] retval:  %s/lib
[hook_mmap13_decrypt_str] retval:  /lib
[hook_mmap13_decrypt_str] retval:  arm64
[hook_mmap13_decrypt_str] retval:  %s/%s
[hook_mmap13_decrypt_str] retval:  %s/%s
[hook_mmap13_decrypt_str] retval:  %s/%s
[hook_mmap13_decrypt_str] retval:  /proc/self/maps
[hook_mmap13_decrypt_str] retval:  %s/%s
[hook_mmap13_decrypt_str] retval:  %s/%s
[hook_mmap13_decrypt_str] retval:  %s/%s
[hook_mmap13_decrypt_str] retval:  %s/%s
[hook_mmap13_decrypt_str] retval:  %s/%s
[hook_mmap13_decrypt_str] retval:  %s/%s

The suspicious one is /proc/self/maps. Printing the backtrace shows mmap13!0xF348, and the cross-reference to that function is at 0x4D28.

Fig. 9

The bl sub_F2E8 is at 0x4D28, and with mmap13’s base of 0x7AB2D21D28 added, the absolute address is what it is.

Searching 0x7AB2D21D28 in log.txt and examining the call graph reveals the following sequence:

1. openat + lseek + read reads the data in /proc/self/maps.
2. vsnprintf concatenates the full paths of the app’s own 3 .so libraries, including lib__6dba__.so.

0x14043be0 (0x7ab2d2bbe0): sub_1404712c() {
    0x140470c4 (0x7ab2d2f0c4): sub_14047068() {
        0x14047080 (0x7ab2d2f080): [SVC] sysno(0x38) -> openat(-100, "/proc/self/maps") => fd: 0x27
    }
}
0x14043c1c (0x7ab2d2bc1c): sub_140439a0() {
    0x140439b4 (0x7ab2d2b9b4): sub_14045e70() {
    }
    0x140439d4 (0x7ab2d2b9d4): sub_1404709c() {
        0x140470c4 (0x7ab2d2f0c4): sub_14047068() {
            0x14047080 (0x7ab2d2f080): [SVC] sysno(0xde) -> mmap(0x0, 0x80000, 0x3) => mmap address: 0x7ab1660000
        }
    }
}
0x14043c40 (0x7ab2d2bc40): sub_14043a44() {
    0x14043aa8 (0x7ab2d2baa8): sub_1404709c() {
        0x140470c4 (0x7ab2d2f0c4): sub_14047068() {
            0x14047080 (0x7ab2d2f080): [SVC] sysno(0x3e) -> lseek
        }
    }
    0x14043ad4 (0x7ab2d2bad4): sub_1404709c() {
        0x140470c4 (0x7ab2d2f0c4): sub_14047068() {
            0x14047080 (0x7ab2d2f080): [SVC] sysno(0x3f) -> read(0x27, "12c00000-12c40000 rw-p 00000000 ", 0x80000) => real read bytes: 0xf96
        }
    }
    0x14043ad4 (0x7ab2d2bad4): sub_1404709c() {
        0x140470c4 (0x7ab2d2f0c4): sub_14047068() {
            0x14047080 (0x7ab2d2f080): [SVC] sysno(0x3f) -> read(0x27, "71124000-71125000 rw-p 0003d000 ", 0x7f06a) => real read bytes: 0xfb7
        }
    }
    
// ...

0x1403b658 (0x7ab2d23658): sub_14040bb8() {
    0x14040c48 (0x7ab2d28c48): sub_14040ab0() {
        0x14040af4 (0x7ab2d28af4): sub_14037590() {     // nglog: mmap13 => 0xBAF4
            0x14040af4 (0x7ab2d28af4): [ExternalCall] vsnprintf("/data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/lib__6dba__.so", 0x400, "%s/%s") => res: 0x4b
        }
			}
}

From this I guessed it might be checking whether its own .so has been hooked.

So I tried hooking mmap13’s vsnprintf, replacing lib__6dba__.so with another unhooked library, libgamecore.so (also one of the app’s own libraries).

function hook_vsnprintf () {
    Interceptor.attach(mmap_base.add(0xBBB8), {
        onEnter: function (args) {
            this.a0 = args[0];
        },
        onLeave: function (retval) {
            if (this.a0.readCString().indexOf("lib__6dba__.so") != -1) {
                console.log("replace!!!!!!!!!!!!")
                Memory.writeUtf8String(this.a0, this.a0.readCString().replace("lib__6dba__.so", "libgamecore.so"))
            }
            console.log("[mmap13_vsnprintf] this.a0: ", this.a0.readCString());
        }
    })
}

Before the replacement, vsnprintf outputs:

[mmap13_vsnprintf] this.a0:  /data/user/0/com.example.game/lib
[mmap13_vsnprintf] this.a0:  /data/user/0/com.example.game/lib
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libaudio.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libgamecore.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/lib__6dba__.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libaudio.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libgamecore.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/lib__6dba__.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libaudio.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libgamecore.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/lib__6dba__.so

After the replacement, vsnprintf outputs the following — note the two extra lines about libc.so:

[mmap13_vsnprintf] this.a0:  /data/user/0/com.example.game/lib
[mmap13_vsnprintf] this.a0:  /data/user/0/com.example.game/lib
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libaudio.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libgamecore.so
replace!!!!!!!!!!!!
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libgamecore.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libaudio.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libgamecore.so
replace!!!!!!!!!!!!
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libgamecore.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libaudio.so
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libgamecore.so
replace!!!!!!!!!!!!
[mmap13_vsnprintf] this.a0:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libgamecore.so
[mmap13_vsnprintf] this.a0:  /vendor/lib64/libc.so
[mmap13_vsnprintf] this.a0:  /system/lib64/libc.so

Using the same trick to replace libc.so with libz.so, the app finally stops crashing right after mmap13 — instead it creates more modules.

Fig. 10

Quick summary: mmap13 appears to first check whether lib__6dba__.so (the app’s own library) is hooked; if that passes, it checks whether libc.so (a system library) is hooked. Only when both pass does it create new modules for further checks; otherwise it makes the program exit by some means.

After manually patching mmap13, many new modules appear — created inside mmap3, indexed starting from 14, for mmap14–mmap30.

if (this.sysno == 0xe2) {
    console.log("[hook_mmap3_syscall] mprotect addr: ", this.a0, "size: ", this.a1 ,"prot: ", this.a2);
    if (mmap_history[this.a0]) {
        console.log(`\t[hook_mmap3_syscall] mmap addr: ${this.a0}  size: ${mmap_history[this.a0]}`);
        // after patch mmap13 detect, use this to dump new mmap module
        if (is_hook_mmap13) {
            saveData(`/data/data/com.example.game/mmap_${this.a0}_${mmap_history[this.a0]}_${idx++}.bin`, this.a0, mmap_history[this.a0].toInt32());
        }
    }
    // ...

Local lib detection

In the previous section I bypassed the hook detection in lib__6dba__.so via trace logs plus a bit of guesswork. This section digs into the actual detection mechanism.

Hooking mmap13’s syscall wrapper, printing the backtrace when the syscall is openat and the path contains lib__6dba__.so, then following it upward, I found that lib__6dba__.so is opened in mmap13!0x3BF0.

The detailed call chain (where ins addr is the instruction address and func addr is the function start address):

0x394C(ins addr) -> 0x485C(ins addr) -> 0x433C(ins addr) -> 0x3F7C(ins addr) -> 0x3BF0(func addr)

The logic around 0x394C (the instruction that calls sub_4684) is below; I’ll call its containing function mmap13_main.

Testing showed that after the “hook mmap13’s vsnprintf and replace lib__6dba__.so with the unhooked libgamecore.so” trick, the function sub_4C20 returns 1; otherwise it returns 0.

So sub_4C20 is either the actual detection function or one that processes the detection result. I’ll call sub_4C20 mb_detect_func.

Fig. 11

Diving into mb_detect_func and verifying by hooking, I find the key function get_so_info.

At first I thought get_so_info was the actual detection, because hooking showed it’s called 3 times, and hooking mmap13’s openat syscall showed it opening 3 of its own .so libraries, matching perfectly. So I guessed the first two calls returned a1 == 0 because I hadn’t hooked libaudio.so and libgamecore.so, while the third returned non-zero because lib__6dba__.so was hooked and detected.

Fig. 12

// hook mmap13 openat log:
[hook_mmap13_openat] a1:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libaudio.so
[hook_mmap13_openat] a1:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/libgamecore.so
[hook_mmap13_openat] a1:  /data/app/com.example.game-xxxxxxxxxxxx==/lib/arm64/lib__6dba__.so

// hook get_so_info log:
[mmap13_get_so_info] this.a1.readPointer:  0x0
[mmap13_get_so_info] this.a1.readPointer:  0x0
[mmap13_get_so_info] this.a1.readPointer:  0x7cfb84e000

But analyzing get_so_info in detail revealed it only parses and saves the info from /proc/pid/maps (so_info[0] holds the .so’s binary info). The first two a1 == 0 happened because those two libraries weren’t loaded yet at that moment.

Continuing down, so_info (so_img) is then passed into do_something1, whose return value is stored in dest, then compared with *(_DWORD*)(v8+0x3C). If they differ, it eventually leads to the wrong_branch.

So I guessed that *(_DWORD *)(v8 + 0x3C) is the hash of the original lib__6dba__.so’s .text segment, and dest is the hash of lib__6dba__.so’s .text segment computed from /proc/pid/maps.

Fig. 13

Entering do_something1, it first parses the relocation table via so_img, though I couldn’t see what it was for.

Fig. 14

Continuing down, there’s the key while loop.

Fig. 15

Inside, hash_sum is a pile of computation — likely computing something hash-like. Hooking it shows that args[0] once held lib__6dba__.so’s .text segment, args[1] is the .text size, and args[2] holds the result.

Fig. 16

I then found that for each of its own .so libraries, hash_sum is called twice (at two different locations) to compute hashes:

1. The first hashes the entire file. From the first part of the next image, 0x1860df is exactly the file size of lib__6dba__.so, and before this, openat opened lib__6dba__.so. The backtrace is at mmap13!0x3DF0.
2. The second hashes the .text segment. From the second part, 0xcaf4 is exactly the .text segment size of lib__6dba__.so, and before this, openat opened /proc/self/maps — so this part was obtained from there. The backtrace is at mmap13!0x6B98, which is the do_something1 mentioned above.

The first is probably an integrity check; the second is obviously the hook check. This makes ordinary IO redirection hard to bypass directly.

Fig. 17

Summary: for local libraries (the app’s own), it calls hash_sum to verify, comparing against a value precomputed and baked into the .so.

System lib detection

Only after passing the local-lib detection does it call check_libc to check libc.so (it appears to check only libc among the system libraries). The function below is mmap13_main.

Fig. 18

check_libc calls do_something2.

Fig. 19

Now let’s analyze do_something2 in detail.

First it calls parse_elf_data to parse a given .so; args[0] is the address of the libc.so image (which was opened and read via the openat syscall earlier). The parse result is stored in soinfo (not the linker’s soinfo).

Fig. 20

Then it decrypts a key string .text and passes it into get_section_info, which returns some data from libc.so’s .bss segment containing the specified section’s info — call it section_info.

For example, *(section_info+0x10) is the offset of the specified section.

Fig. 21

Next it walks maps_item (each line of /proc/pid/maps I call a maps_item), and only when reaching the segment right after libc.so’s .text does it satisfy the first if condition below.

On a normal phone that hasn’t run Frida, the second if condition holds (i.e. the segment after .text is at or beyond the end of .text), and it proceeds to the real libc check.

Fig. 22

When the second if condition is not satisfied, it goes to the path below, looping multiple times.

The first red box means the loop runs at most 10 times; if after walking the 10 maps_items after .text it still hasn’t found one beyond .text’s end, something is wrong, and it ultimately reaches the crash-inducing error branch.

A normal, non-Frida-interfered flow jumps straight to goto LABEL 49 at the second red box.

Fig. 23

goto LABEL 49 eventually leads here, calling do_check_libc for the real libc verification.

do_check_libc contains some key strings, shown below.

Fig. 24

As for the exact mechanism of do_check_libc, interested readers can analyze it themselves.

Bypassing all the hook detection

After bypassing the lib__6dba__.so and libc.so checks by hooking mmap13’s vsnprintf, it loads libaudio.so and libgamecore.so (the app’s own libraries), and I found these two libraries also contain a start function identical to lib__6dba__.so, along with the same mmap-module detection and the same local-lib and system-lib checks.

The good news is they roughly use the same mmap modules for detection; only the number of created modules differs. For example, the mmap11 created by libaudio.so is actually the mmap13 created by lib__6dba__.so.

Since the mmap modules call vsnprintf to build the full library paths, I can hook vsnprintf to change the path of a given library, redirecting it to another unhooked library to bypass the check. The concrete method is already given above, so I won’t repeat it.

Conclusion

This game’s protection is among the hardest I’ve encountered. The difficulty is that it’s very tedious, and it can only be debugged by hooking. But once you find the right approach, you can still analyze and solve it step by step — unlike some VMs where you don’t even know where to start.

I’ve only roughly analyzed one module here. If you’re interested, the other modules — about 29 in total — are also quite fun to dig into.

---

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch