About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

Bypassing an Unusual Anti-Debug & Anti-Hook in a Commercial Mobile-Game Packer

Preface

A friend in the chat group hit a game sample:

I opened it — symbols not stripped?? What kind of protection is this:

Protection that careless is basically inviting a crack, so I confidently said, “this one’s good for beginners!”

After digging deeper, it turned out more complex than expected, with unusual and uncommon detection methods. It’s a Japanese/Korean commercial protection called NHN Ent AppGuard. Interesting enough to warrant a writeup.

1. Packer analysis

Open the core libloader.so. IDA throws a bunch of warnings about invalid section headers. No matter — force your way in:

The code is greyed out — meaningless binary, probably encrypted. Look at the init array:

What’s the red one (the second)? Try to jump to it, but IDA says “Command JumpAsk failed” — the address isn’t in visible range?

Open the segment info in 010 Editor:

The code segment’s file size is 0x25c848, memory size 0x25c848, padding 0x10000 — normal.

The data segment’s file offset is 0x25ea50, but its memory offset is 0x35ea50 — 0x100000 bytes (100K) extra. Given the code segment’s 0x10000 padding, the data segment’s memory offset should be 0x26ea50. Clearly abnormal.

The init array’s second function address is 0x29c1ec — in neither the code nor the data segment, but in that extra padding. Bizarre.

So the first init function must handle this, or the linker errors out. Look at the first init function:

It opens /proc/self/maps with fopen, then uses strtok and strtoul to find the extra 100K mapped during linking and get its base address:

Then it mmaps that address as rwx, memcpys the code hidden in the data segment over, then mprotects it back to rx:

The memcpy runs 5 times, by source address, target address and length:

After copying, the second init function is ready in memory to execute.

The second init function does the decryption:

It first decrypts strings — a simple XOR over a few strings — then decrypts the section.

String decryption is simple; let’s look at section decryption.

Set a breakpoint on the linker’s callarray function and step into the second init function:

First, locate the encrypted data in the .so from X0; the start address is 0xab710:

Exactly the start of .text.

Then it XORs the bytes at .text+0x4 and .text+0xc to get the encrypted data length:

Then it mallocs a buffer based on X1 (the decrypted size), and memcpys the real encrypted data (starting at .text+0x10) over, using the computed length in x22:

Then it XORs the bytes at .text+0x8 and .text+0xc to compute the first key (w11), used later for decryption:

The second key sits at the end of the encrypted data — via x22, the byte right after the encrypted data is the second key (w25).

Then decryption: XOR each byte once; by current offset, odd bytes use key1, even bytes use key2:

After decryption the data starts with 0x78, 0x9c — clearly zlib-compressed:

Initialize zlib to version 1.2.7:

Then loop-decompress, and copy the decompressed code back to .text:

Once decompressed and copied, decryption is done:

So we can write our own encryption, modify the decompressed code, then recompress and re-encrypt it, to patch the original code. Encryption code:

import zlib
def encode(inName,outName):
    with open(inName,"rb") as f:
        data = f.read()
    #compress
    o1 = list(zlib.compress(data))
    #encrypt
    index = 0
    step = 0
    key1 = 0xEAB54A3C
    key0 = 0x5AB9C6D5
    lenraw = len(o1)
    out = []
    while index < lenraw:
        if lenraw - index >= 4:
            raw = getInt32(o1,index)
            if step == 0:
                addInt32(out,0,raw ^ key0)
                step = 1
            else:
                addInt32(out,0,raw ^ key1)
                step = 0
            index = index + 4
        else:
            while index<lenraw:
                out.append(data[index])
                index = index + 1
    addInt32(out,0,key0)
    with open(outName,"wb") as f:
        f.write(bytes(out))

Note: since the modified code’s compressed length may change, you need to adjust the bytes at .text+0x4, +0x8, and +0xc to match the decryption algorithm, and re-append key2 at the end of the encrypted data in little-endian.

At this point the packer’s decryption is fully understood, and we can customize code and encrypt it back.

2. Anti-debug

2.1 TracerPid anti-debug

With the packer understood, press F9 — IDA crashes. Clearly anti-debug exists, but there are no useful strings in the .so — probably encrypted.

The first step in studying any protection is finding the string decryption function. Once you have it, everything they do becomes clear.

To find the string decryption function, breakpoint fopen and trace back. For anti-debug, it usually opens /proc/self/status and reads TracerPid. Opening that file uses fopen. Since the .so has no plaintext /proc/self/status, it must be decrypted before being passed to fopen — i.e., the decrypt function is close to fopen.

By simple debugging, sub_C4A50 is the string decryption function:

Also a simple XOR.

Breakpoint sub_C4A50 and see what shows up.

It decrypts many common libc function names — not a good sign.

Then the familiar /proc/self/status appears.

Step in, and it encrypts TracerPid — the game is up.

Then the routine:

fopen /proc/self/status:

fgets loop read, strstr to check whether TracerPid was read:

If read, atoi to get TracerPid; if non-zero, enter the exit flow:

The exit flow is complex:

It eventually picks an exit method by argument: exit, a crafted crash, or svc exit_group. Not going deeper.

We just need to make atoi return 0.

But if it were that simple, this post wouldn’t exist.

2.2 Breakpoint-instruction detection

At first I didn’t know what exit flow followed atoi, so I set a breakpoint on exit:

But with the breakpoint set, exit got called before even reaching the TracerPid check — so there’s another check.

Through repeated debugging: setting a breakpoint on exit makes the process exit early, before the TracerPid check. Without the exit breakpoint, it proceeds to the TracerPid check and decides the exit flow by atoi.

We saw exit, _exit decrypted during the debug — follow it.

It decrypts not just exit, _exit, but also __exit, kill, tgkill.

And each time it decrypts a name, it dlsyms the function address:

Interesting.

Follow on: it calls dlclose to close the earlier dlopen’ed libc:

Then another BLR X8.

At some step, it passes exit’s address as an argument to a function:

It copies the first 8 bytes of exit to another address:

The copied result is exactly exit’s first 8 bytes.

The exit function:

Why extract the first 8 bytes? Interesting.

After copying, another function:

First set w8 = 0xd4200000:

Store W8 into var14, pass var14 as one argument and exit’s address as another into sub_71be98e690:

In sub_71be98e690, it compares whether exit’s first 4 bytes are 00 00 20 d4:

If not, returns non-zero; if so, returns 0.

Why compare 00 00 20 d4?

Because 00 00 20 d4 is the machine code of the arm64 breakpoint instruction (BRK).

If the first 8 bytes contain 00 00 20 d4, the protection enters the exit flow and calls exit.

It checks exit, _exit, __exit, kill, and tgkill’s first 8 bytes for a breakpoint instruction. Take _exit as an example:

_exit’s first 8 bytes were originally c8 0b 80 d2 01 00 00 d4:

After setting a breakpoint, memory holds 00 00 20 d4 01 00 00 d4.

So the moment you set a breakpoint, it’s detected and exits.

I’d heard of this anti-debug before, but this is the first time seeing it in the wild — impressive.

Besides exit, _exit, and so on, it also checks open, fopen, popen, read, pread, memcpy, memcmp, fgets, strcmp, strncmp, strstr, strlen, strcpy, sleep, mmap, munmap, __ptrace, ptrace, and fork for breakpoints, exiting if any is broken. The method is the same: decrypt the name, dlsym the address, check the first 8 bytes.

3. Anti-hook

With the breakpoint check and TracerPid bypassed, the friend says there’s still Frida detection.

Launch the app with Frida — it crashes immediately. Analysis shows it’s the same exit flow. But breaking on the string decrypt function shows no Frida-related strings — odd.

Launch with Frida, suspend the app 10s at start, and attach IDA to see what happens.

It enters the exit flow right after checking exit’s breakpoint — but we didn’t breakpoint exit!

Check what changed in exit for the Frida-launched app:

We didn’t hook any function, just launched — but Frida itself hooked exit.

It also hooked _exit, fork, and so on.

Since it exited after the exit-breakpoint check, I suspect it checks something more. Look closer.

After the breakpoint check, it enters sub_71bd56debc, passing the copied first 8 bytes:

Step in — it XORs all 8 copied bytes with 0x26:

Then it compares byte 7 with 0x39 and byte 8 with 0xf0:

Our encrypted byte 7 is exactly 0x39 and byte 8 exactly 0xf0. Match → exit flow → it destroys the same mutex twice via pthread_mutex_destroy to craft a crash.

Why compare 0x39 and 0xf0? The comparison is post-encryption. Pre-encryption: 0x39 ^ 0x26 = 0x1f, and 0xf0 ^ 0x26 = 0xd6.

So it’s really checking whether the original values are 0x1f and 0xd6. Why those?

Check the arm64 instruction manual for the BR (register branch) instruction:

There it is.

The br instruction’s high 16 bits are 0xd61f.

So the protection checks whether _exit’s second instruction is a br. Inline hooks usually make the first instruction a mov of a constant into a register, then a br register instruction as the second. Checking the second instruction’s high 16 bits against 0xd61f detects whether the function has been inline-hooked.

Because Frida hooked exit and _exit, the breakpoint check passed but the hook check failed, so it crafted a crash. The protection encrypts before comparing — a bit flashy.

The bypass is easy once you know this: change the compared 0x39 and 0xf0 (post-encryption) to other values. Then recompress, re-encrypt, and patch back into the .so.

So the packer’s scheme (multi-mapped memory, then decrypt + decompress + fill), anti-debug (TracerPid + breakpoint instruction), and anti-hook (br instruction check) are all bypassed. You can debug and hook the game freely.

4. Afterword

This time it was a Japanese game and packer. The APK uses NHN Ent AppGuard; not sure if anyone’s familiar with it. Essentially it’s an encrypted compression packer, which is uncommon in mobile-game protection. The breakpoint-instruction anti-debug and br-instruction anti-hook are also novel — not the usual playbook.

Hope this helps with learning detection, anti-debug, and bypass, and gives mobile-game protection vendors some outside perspective.

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch