Colin XiaoReverse Engineering & Security Research

Analyzing the ttEncrypt Algorithm with Unidbg

About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

Analyzing the ttEncrypt Algorithm with Unidbg

Why I did this

The honest answer: I wanted to kill some time one evening. But the real reason is that everyone around me is either already done analyzing it or in the middle of it. If I don’t keep up, I’ll get left behind. Since I’m just starting out, I picked something simple.

This is a small sample I looked at out of boredom one evening, so feel free to skip it.

Summary

I spent a few hours on a quick pass, so I didn’t fully trace the entire flow — just used some fast-analysis tricks to locate and verify the important parts. Toward the end I found the whole thing basically looks the same, like the JNI_OnLoad method at the very start. Given time I’d like to dig into the instruction execution too, to see how it passes args in/out and runs methods.

What you’ll get from this article:

  • Simple debugging with unidbg
  • Quick fixes for a .so’s methods
  • The AES key initialization flow

From there you can judge whether it’s worth reading on.

Target selection

Since I’m analyzing with unidbg, I can’t avoid the author’s small demo, which is:

  • unidbg-android/src/test/java/com/bytedance/frameworks/core/encrypt/TTEncrypt.java

When I first started out, I learned unidbg’s various hook scripts by playing with these demos. So consider this a long-overdue homework submission. Target confirmed — let’s start.

Environment

  • Dev environment: Unidbg — that’s all you need
  • Host: Windows 11
  • App version: 26.1.0 (sent to me, so no download link)

Emulated execution

For this “homework,” the first step is to copy the author’s demo and tweak it, then point the apk at our target version.

Since the version under analysis is 64-bit and loaded from the apk, the first step is to change for32Bit() to for64Bit(). Then pass the apk’s current path to createDalvikVM, change the loadLibrary argument to ttEncrypt, and set the second argument to true so it runs the .so’s init method.

Some of his hook demos and the IDA Pro debugging code aren’t needed right now, so delete them first.

After the changes it looks like this:

  • Then run it.
  • And it immediately blew up. Following the error, let’s check whether this dependency exists in the apk.
  • There’s no library named ttEncrypt, so let’s look at the phone.
  • Since the library is missing, and from the author’s code we know it calls ttEncrypt([BI)[B, let’s check whether the phone exports that symbol and try our luck.
  • Lucky guess — no need to analyze the apk at all. Today’s luck is good, keep going.
  • So we change the library name from ttEncrypt to Encryptor and try again.
  • Better this time: it can’t find the ttEncrypt([BI)[B method. Since we can locate the .so by symbol, the method must exist.
  • Why can’t it find it? Looking at the first three lines above, the class name has changed to com/bytedance/frameworks/encryptor/EncryptorUtil, no longer com/bytedance/frameworks/core/encrypt/TTEncryptUtils.
  • Fix it and try again.
  • First arg is 0x10 bytes; second arg is the byte[] length.
  • It runs!
  • And there’s a result.
  • Time to call it a night.

Of course it doesn’t end here — we still need to port the algorithm out.

Porting the algorithm

The entry point

The first step of porting is obvious: fire up IDA Pro.

  • Then drag the .so in for analysis.
  • As usual, let’s look at the exports first.
  • Only two exported functions: JNI_OnLoad and start.
  • We all know JNI_OnLoad is where dynamic registration happens, and start is the entry for directly launching the .so. Let’s see what’s in JNI_OnLoad.
  • Nothing. Let’s step into sub_da0.
  • Can’t read it. So let’s just glance at sub_da0’s control-flow graph.
  • A dense wall of basic blocks — enough to make anyone with a block-density phobia close the tab immediately.
  • Since I don’t want to find the dynamic-registration site by analyzing JNI_OnLoad, I’ll get the real entry from unidbg’s log instead:
Find native function Java_com_bytedance_frameworks_encryptor_EncryptorUtil_ttEncrypt => RX@0x40007a38[libEncryptor.so]0x7a38
  • This is a reminder that when you’re stuck, look at the logs — there’s usually a clue.
  • So the code entry point is 0x7a38.

Algorithm analysis

Fixing the .so methods

  • Opened, and it’s all red — and F5 won’t work. What now?
  • Relax, we just do a simple fix.
  • Starting at 0x7a38, select a region downward, pick a bit, then press p (this force-creates a function from the selected code).
  • There are still some red blocks below. Don’t panic — scroll down until you find the last instruction of the red block right before the RET instruction, i.e. at 0x7B58, and press e there.
  • Now it’s normal.
  • But scrolling up, 0x7B04 also has a red block.
  • Do the same thing — press e on this line. After fixing it looks like this.
  • Now we can happily press F5.
  • Some might say this is overkill — just press p directly at 0x7a38 and be done. True, this sample only needs a p at 0x7a38. But it’s worth learning the select-block fix anyway, because on later samples p alone won’t cut it; selecting a block works better.

Fixing the JNI method

  • Here I’ll just do it directly — for importing the structures, see other authors’ articles.
  • After fixing and reading IDA’s hints, we know the core method is sub_2AC4, and its arguments are, in order: bytes (input byte array), length (byte array size), output (our result), size (result size).
  • Let’s analyze sub_2ac4.
  • It puts our arguments into an array (a struct). Where:
  • There’s a long blob in .rodata we can’t read, so we’ll leave it for now.
  • Analyzing 2ac4, we see it further calls 2c18. Its first argument is 0xb090, the second is our passed args plus return value, the third looks like an index, the fourth is also unreadable (probably similar to the first, leave it), and the fifth looks like a callback method (this is important below).
  • We notice it’s very similar to JNI_OnLoad’s sub_da0 method.
  • This kind of code is unreadable without optimization, and I can’t write an optimizer. Since this path is blocked, let’s try another approach.

traceWrite

Since direct code analysis failed, all that’s left is tracing. This time I won’t trace code directly, but trace writes.

Why not trace code and reconstruct from the results? Because this is obfuscated code and the trace log would be enormous — so we need to simplify.

  • That trace log has over a hundred thousand lines — porting from it would be brutal. So I chose trace-write. Some might ask why not trace-read? We’re results-driven, so I trace the written output.
  • Since we’re tracing writes, we pick the address and size.
  • Back to sub_783a.
  • The malloc allocates length + 118 bytes, so we now know the size is length + 118, leaving just the address.
  • Testing shows the address after 0x7AA8 executes is fixed! So we can pin the address at 0x40358000 — a constant. (What if it’s not fixed? Just add a backend.hook_add_new to grab the actual value. time, rand, srand can all be fixed too — check the source or write a hook.)
  • Now we can happily write the trace-write:
// 0x10 is the passed-in byte[] size, known
emulator.traceWrite(0x40358000, 0x40358000 + 0x10 + 118);
  • Screen too small, had to screenshot twice.
  • From the trace we can see the data is composed of four parts and three methods: 0x2be0, 0x3c2c (PC is in libc.so, so look at LR to see where it returns to), and 0x2a58 (0x2a58 and 0x2a88 are probably in the same method).
74 63 05 10 ==>0x10056374
00 00 ==> 0x2be0

// 0x1c184
A0 BB 10 FA EC 38 50 03 EA 42 C6 94 15 ED CD 30（0x35038ecfa10bba0、0x30cded1594c642ea）
B6 5F F5 E8 25 43 A3 66 D4 B8 E7 FB D7 F3 89 77 （0x66a34325e8f55fb6、0x7789f3d7fbe7b8d4）

// 0x2a58
[01:48:41 773] Memory WRITE at 0x40358026, data size = 8, data value = 0xeb12c44a6b02c09, PC=RX@0x40002a58[libEncryptor.so]0x2a58, LR=RX@0x40002a4c[libEncryptor.so]0x2a4c
[01:48:41 773] Memory WRITE at 0x4035802e, data size = 8, data value = 0x632b08e36313439e, PC=RX@0x40002a58[libEncryptor.so]0x2a58, LR=RX@0x40002a4c[libEncryptor.so]0x2a4c
[01:48:41 773] Memory WRITE at 0x40358036, data size = 8, data value = 0x241e2a664df9f031, PC=RX@0x40002a58[libEncryptor.so]0x2a58, LR=RX@0x40002a4c[libEncryptor.so]0x2a4c
[01:48:41 773] Memory WRITE at 0x4035803e, data size = 8, data value = 0x2bee5c58693bd0fc, PC=RX@0x40002a58[libEncryptor.so]0x2a58, LR=RX@0x40002a4c[libEncryptor.so]0x2a4c
[01:48:41 773] Memory WRITE at 0x40358046, data size = 8, data value = 0x468e9bebbf9629eb, PC=RX@0x40002a58[libEncryptor.so]0x2a58, LR=RX@0x40002a4c[libEncryptor.so]0x2a4c
[01:48:41 773] Memory WRITE at 0x4035804e, data size = 8, data value = 0x2a698ed9942a3431, PC=RX@0x40002a58[libEncryptor.so]0x2a58, LR=RX@0x40002a4c[libEncryptor.so]0x2a4c
[01:48:41 773] Memory WRITE at 0x40358056, data size = 8, data value = 0x713bbf110c38b4, PC=RX@0x40002a58[libEncryptor.so]0x2a58, LR=RX@0x40002a4c[libEncryptor.so]0x2a4c
[01:48:41 773] Memory WRITE at 0x4035805e, data size = 8, data value = 0x217a8beac056e0c6, PC=RX@0x40002a58[libEncryptor.so]0x2a58, LR=RX@0x40002a4c[libEncryptor.so]0x2a4c
[01:48:41 773] Memory WRITE at 0x40358066, data size = 8, data value = 0x1e3380f1680f33dc, PC=RX@0x40002a58[libEncryptor.so]0x2a58, LR=RX@0x40002a4c[libEncryptor.so]0x2a4c
[01:48:41 773] Memory WRITE at 0x4035806e, data size = 8, data value = 0xe121d61a37fcf7dc, PC=RX@0x40002a58[libEncryptor.so]0x2a58, LR=RX@0x40002a4c[libEncryptor.so]0x2a4c
[01:48:41 773] Memory WRITE at 0x40358076, data size = 8, data value = 0x77f78e1f1dd196d1, PC=RX@0x40002a88[libEncryptor.so]0x2a88, LR=RX@0x40002a84[libEncryptor.so]0x2a84
[01:48:41 773] Memory WRITE at 0x4035807e, data size = 8, data value = 0x2e7d43264681b810, PC=RX@0x40002a88[libEncryptor.so]0x2a88, LR=RX@0x40002a84[libEncryptor.so]0x2a84


09 2C B0 A6 44 2C B1 0E 9E 43    .....w.,..D,...C
0030: 13 63 E3 08 2B 63 31 F0 F9 4D 66 2A 1E 24 FC D0    .c..+c1..Mf*.$..
0040: 3B 69 58 5C EE 2B EB 29 96 BF EB 9B 8E 46 31 34    ;iX\.+.).....F14
0050: 2A 94 D9 8E 69 2A B4 38 0C 11 BF 3B 71 00 C6 E0    *...i*.8...;q...
0060: 56 C0 EA 8B 7A 21 1E 33 80 F1 68 0F 33 DC DC F7    V...z!.3..h.3...
0070: FC 37 1A D6 21 E1 D1 96 D1 1D 1F 8E F7 77 10 B8    .7..!........w..
0080: 81 46 26 43 7D 2E                                  .F&C}.
  • So we focus on these three methods to complete the protocol port: 0x2be0, 0x3c2c (PC in libc.so, so look at LR), and 0x2a58 (0x2a58 and 0x2a88 should be in the same method).

0x2be0

  • First look at 0x2be0.
  • Looks a bit familiar.
  • Convert 268788596 to hex?
  • Isn’t that the first segment’s 0x10056374 and 0x0?
  • A third of the way there already. Luck is on our side.

0x3c2c

  • It returns to sub_2C18 and calls blr x20, and we don’t know what x20 holds, so let’s set a breakpoint and look.
  • Set a breakpoint at 0x3C28 and check x20:
        debugger.addBreakPoint(module, 0x3C28);
  • It calls sub_2c0c.
  • What does sub_2c0c do?
  • A trampoline: it passes x0 and x1 in, then calls x0 inside sub_2c0c with argument 1.
  • What are the values of x0 and x1?
  • It calls sub_2b40 with:
  • We don’t know what it is; let’s look at what sub_2b40 does.
  • Looks like dynamic memory allocation? That’s inconsistent with our earlier analysis. Could this execution not be the one we’re after?
  • Let’s run several times and see which methods get called.
  • It ran sub_2b5c.
  • Looks like time and random numbers.
  • It ran again? We don’t know how many times it runs before reaching our method, so let’s handle it with code.
backend.hook_add_new(new CodeHook() {
            @Override
            public void hook(Backend backend, long address, int size, Object user) {
                if (address == module.base + 0x3C28){
                    Number x0 = backend.reg_read(Arm64Const.UC_ARM64_REG_X0);
                    Number x20 = backend.reg_read(Arm64Const.UC_ARM64_REG_X20);
                    long m0 = x20.longValue()-module.base;
                    long m1 = x0.longValue()-module.base;
                    System.out.println("direct call -->" + Long.toHexString(m0));
                    System.out.println("indirect call -->" + Long.toHexString(m1));
                }
            }

            @Override
            public void onAttach(UnHook unHook) {

            }

            @Override
            public void detach() {

            }
        }, module.base, module.base + module.size, null);
  • The output:
direct call -->2c0c
indirect call -->2b40
direct call -->2c0c
indirect call -->2b5c
direct call -->2c0c
indirect call -->2b40
direct call -->2c0c
indirect call -->2b40
direct call -->2c0c
indirect call -->2b94
direct call -->2c0c
indirect call -->2b40
direct call -->2c0c
indirect call -->2ba8
direct call -->2c0c
indirect call -->2bb8
direct call -->2c0c
indirect call -->2bb8
direct call -->2c0c
indirect call -->2bc8
direct call -->2c0c
indirect call -->2be8
direct call -->2c0c
indirect call -->2c04
direct call -->2c0c
indirect call -->2c04
direct call -->2c0c
indirect call -->2c04
  • Let’s briefly analyze what these indirectly-called methods do, and print them all (I’ll only analyze the two core methods; the rest are left as an exercise).
  • After analyzing we can rewrite the code like this and check the output:
Map<Integer, String> map = new HashMap<>();
        map.put(0x2c0c, "trampoline, x0 = function to call, x1 = arg (pointer)");
        map.put(0x2b40, "malloc");
        map.put(0x2b5c, "time/srand/rand RNG functions");
        map.put(0x2b94, "jumps to another large function (same as the entry 0x2C18)");
        map.put(0x2ba8, "hash-family algorithm (6A58)");
        map.put(0x2bb8, "memcpy1");
        map.put(0x2bc8, "memcpy2");
        map.put(0x2be8, "aes-related function");
        map.put(0x2c04, "free");

        Backend backend = emulator.getBackend();
        backend.hook_add_new(new CodeHook() {
            @Override
            public void hook(Backend backend, long address, int size, Object user) {
                if (address == module.base + 0x3C28){
                    Number x0 = backend.reg_read(Arm64Const.UC_ARM64_REG_X0);
                    Number x20 = backend.reg_read(Arm64Const.UC_ARM64_REG_X20);
                    long m0 = x20.longValue()-module.base;
                    long m1 = x0.longValue()-module.base;
                    System.out.println("direct call -->" + Long.toHexString(m0)+ "\t" + map.get((int)m0));
                    System.out.println("indirect call -->" + Long.toHexString(m1)+ "\t" + map.get((int)m1));
                }
            }

            @Override
            public void onAttach(UnHook unHook) {

            }

            @Override
            public void detach() {

            }
        }, module.base, module.base + module.size, null);
  • Then the output:
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2b40	malloc
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2b5c	time/srand/rand RNG functions
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2b40	malloc
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2b40	malloc
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2b94	jumps to another large function (same as the entry 0x2C18)
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2b40	malloc
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2ba8	hash-family algorithm (6A58)
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2bb8	memcpy1
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2bb8	memcpy1
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2bc8	memcpy2
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2be8	aes-related function
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2c04	free
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2c04	free
direct call -->2c0c	trampoline, x0 = function to call, x1 = arg (pointer)
indirect call -->2c04	free
  • From this we discover another sub_2b94 that calls sub_45C8, similar to JNI_OnLoad’s sub_DA0 and ttEncrypt’s sub_2C18.
  • So we apply the same method to sub_45C8’s 0x55D8 (like sub_2C18, it has only one call site, so it’s quick to resolve), producing this full code (JNI_OnLoad untouched, since we haven’t seen anything related to it yet):
Map<Integer, String> map = new HashMap<>();
        map.put(0x2c0c, "trampoline, x0 = function to call, x1 = arg (pointer)");
        map.put(0x2b40, "malloc");
        map.put(0x2b5c, "time/srand/rand RNG functions");
        map.put(0x2b94, "jumps to another large function (same as the entry 0x2C18)");
        map.put(0x2ba8, "hash-family algorithm (6A58)");
        map.put(0x2bb8, "memcpy1");
        map.put(0x2bc8, "memcpy2");
        map.put(0x2be8, "aes-related function");
        map.put(0x2c04, "free");

        map.put(0x45bc, "trampoline, x0 = function to call, x1 = arg (pointer)");
        map.put(0x44c4, "malloc");
        map.put(0x44e0, "hash-family algorithm (6A58)");
        map.put(0x44f0, "memcpy");
        map.put(0x4500, "table lookup");
        map.put(0x45B4, "free");

        Backend backend = emulator.getBackend();
        backend.hook_add_new(new CodeHook() {
            @Override
            public void hook(Backend backend, long address, int size, Object user) {
                if (address == module.base + 0x3C28 || address == module.base + 0x55D8){
                    Number x0 = backend.reg_read(Arm64Const.UC_ARM64_REG_X0);
                    Number x20 = backend.reg_read(Arm64Const.UC_ARM64_REG_X20);
                    long m0 = x20.longValue()-module.base;
                    long m1 = x0.longValue()-module.base;
                    System.out.println("direct call -->" + Long.toHexString(m0)+ "\t" + map.get((int)m0));
                    System.out.println("indirect call -->" + Long.toHexString(m1)+ "\t" + map.get((int)m1));
                }
            }

            @Override
            public void onAttach(UnHook unHook) {

            }

            @Override
            public void detach() {

            }
        }, module.base, module.base + module.size, null);
  • From the code above we can see the pattern and the methods to focus on: 2be8, (2ba8 and 44E0 both call 6A58), and 2b5c. The rest are allocation/free and memcpy primitives (can be mapped).

2ba8, 44e0 (6A58)

  • Stepping in, it calls 6a58.
  • Inside 6a58 we see a length-4 array, the constant 0x5BE0CD19137E2179LL, and 128. Let’s check what 0x5BE0CD19137E2179LL is first.
  • Looks like sha512. Let’s check whether the array matches sha512’s constants.
  • They all match. Let’s debug.
  • The second argument 0x20 looks like the input length; let’s check its content.
  • From this output we can guess: first arg is the input byte array, second is the length, third is the output.
  • Time to write a hook.
// Dobby throws here — don't ask how I know.
IHookZz hookZz = HookZz.getInstance(emulator); // load HookZz, supports inline hook; docs at https://github.com/jmpews/HookZz
        hookZz.wrap(module.base + 0x6A58, new WrapCallback<HookZzArm64RegisterContext>() { // inline wrap the exported function
            @Override
            public void preCall(Emulator<?> emulator, HookZzArm64RegisterContext ctx, HookEntryInfo info) {
                UnidbgPointer x0 = ctx.getPointerArg(0);
                int x1 = ctx.getIntArg(1);
                UnidbgPointer x2 = ctx.getPointerArg(2);
                byte[] input = x0.getByteArray(0, x1);
                Inspector.inspect(input, "input(sha512)");
                ctx.push(x2);
                emulator.getUnwinder().unwind();
            }
            @Override
            public void postCall(Emulator<?> emulator, HookZzArm64RegisterContext ctx, HookEntryInfo info) {
                UnidbgPointer x2 = ctx.pop();
                int length = 0x40;
                Inspector.inspect(x2.getByteArray(0, length), "final result");
            }
        });
  • Now we have input and output, so we can verify whether our analysis is right.
>-----------------------------------------------------------------------------<
[02:55:46 216]input(sha512), md5=7e6de8344f55d460143710510155b657, hex=2ad949a94f9867924645bdfc952302dea3193a95cbe9913e3bbea28a659eaa90
size: 32
0000: 2A D9 49 A9 4F 98 67 92 46 45 BD FC 95 23 02 DE    *.I.O.g.FE...#..
0010: A3 19 3A 95 CB E9 91 3E 3B BE A2 8A 65 9E AA 90    ..:....>;...e...

>-----------------------------------------------------------------------------<
[02:55:46 234]final result, md5=bed9304b171af26ae94b5fcaccd1fd6b, hex=17b4b1b1094010ca669a8c2116ad14a25b40302b693684f23686cad7817f5295064c06ab34f5f55a8dfceee08bd2ae3a8cb12d52fbc7981e0b4ed5f995b921eb
size: 64
0000: 17 B4 B1 B1 09 40 10 CA 66 9A 8C 21 16 AD 14 A2    .....@..f..!....
0010: 5B 40 30 2B 69 36 84 F2 36 86 CA D7 81 7F 52 95    [@0+i6..6.....R.
0020: 06 4C 06 AB 34 F5 F5 5A 8D FC EE E0 8B D2 AE 3A    .L..4..Z.......:
0030: 8C B1 2D 52 FB C7 98 1E 0B 4E D5 F9 95 B9 21 EB    ..-R.....N....!.
^-----------------------------------------------------------------------------^
  • They match. Let’s analyze one more.

2b5c

  • Too easy — straight to code.
hookZz.wrap(module.base + 0x2B5C, new WrapCallback<HookZzArm64RegisterContext>() { // inline wrap the exported function
        @Override
        public void preCall(Emulator<?> emulator, HookZzArm64RegisterContext ctx, HookEntryInfo info) {
        UnidbgPointer x0 = ctx.getPointerArg(0);
        ctx.push(x0);
        }
        @Override
        public void postCall(Emulator<?> emulator, HookZzArm64RegisterContext ctx, HookEntryInfo info) {
        UnidbgPointer x0 = ctx.pop();
        Inspector.inspect(x0.getPointer(0).getByteArray(0, 0x100), "test");
        }
        });
  • There’s a reference elsewhere; let’s see where it’s used. We only have one sha512 — could it be it?
  • Indeed it is. Not wasted.

2be8

  • sub_2be8 calls sub_2948.
  • Seeing the 16-round loop below, still not obvious — click the first function sub_5DF4.
  • Experienced eyes will already know what this is.
  • If not, see all the different arrays and step in?
  • Look at this one.
  • An sbox! So it’s AES.
  • Then we can guess sub_5DF4 is initKey — it’s the only thing that runs that early.
  • sub_6380 seems useless, skip it.
  • This looks like XOR with the ctx’s sk.
  • An old friend again.
  • Now to confirm key & value — still needs debugging.
debugger.addBreakPoint(module, 0x2948);
  • As usual, check their actual values.
  • x0
  • Another reference — where from?
  • It’s from the sha512 result.
  • x1 –> 0x10
  • x2
  • Right after x0, also from the sha512 result.
  • Guess: x0 is key, x2 is iv.
  • x3
  • x4 –> 0x50
  • By common C++ conventions, address comes first, size second. So guess: x3 is the plaintext, x4 is the text size.
  • x5
  • The blank one looks like the output.
  • x6
  • Looks like the output length.
  • x7
  • All the necessary args seem present, no need to dig further.
  • Straight to the hook code.
hookZz.wrap(module.base + 0x2948, new WrapCallback<HookZzArm64RegisterContext>() {

            @Override
            public void preCall(Emulator<?> emulator, HookZzArm64RegisterContext ctx, HookEntryInfo info) {
                UnidbgPointer x0 = ctx.getPointerArg(0);
                int x1 = ctx.getIntArg(1);
                UnidbgPointer x2 = ctx.getPointerArg(2);
                UnidbgPointer x3 = ctx.getPointerArg(3);
                int x4 = ctx.getIntArg(4);

                byte[] key = x0.getByteArray(0, x1);
                byte[] iv = x2.getByteArray(0, x1);
                byte[] input = x3.getByteArray(0, x4);

                Inspector.inspect(key, "key");
                Inspector.inspect(iv, "iv");
                Inspector.inspect(input, "input(aes)");

                UnidbgPointer x5 = ctx.getPointerArg(5);
                int x6 = ctx.getPointerArg(6).getInt(0);
                ctx.push(x5, x6);
            }
            @Override
            public void postCall(Emulator<?> emulator, HookZzArm64RegisterContext ctx, HookEntryInfo info) {
                UnidbgPointer x5 = ctx.pop();
                int x6 = ctx.pop();
                Inspector.inspect(x5.getByteArray(0, x6), "aes result");
            }
        });
  • We have output! Let’s compare to see if we can decrypt — to confirm it’s AES.
  • Since it has both key and iv, it looks like CBC mode, so let’s take a guess and see if we get lucky.
  • Guessed it on the first try?

Putting it together

  • At this point it’s a bit dizzying — so many methods, three of which look important. How to proceed? Answer: stay results-driven and work backwards.
  • We’ve analyzed three methods, so let’s check which of them are actually used.
The last segment
  • Our result’s last segment looks like the AES-encrypted part.
  • Going back, what is AES’s input?
  • AES’s input:
  • It’s passed in here.
  • Looks like the concatenation of sha512(bytes) + bytes.
The middle segment
  • Our middle segment is also referenced elsewhere.
  • It’s randomly generated right here.

The last open question: how are AES’s key/iv generated?

  • It has a reference too! Where?
  • It’s a sha512 value.
  • What’s sha512’s input?
  • Another reference — what is it?
  • 5C A5 E7 F7 3C 2B 01 CE CA BB DE C0 5C 8E A2 B9 looks familiar — wasn’t it the random value from before?
  • And the other half of the generated value?
0040: 4D D4 C2 E6 B8 31 62 09 0E 52 B3 C7 A6 73 3B A4    M....1b..R...s;.
0050: 1C B2 46 2B 82 9A B5 8A 19 6B 39 DB 57 17 75 24    ..F+.....k9.W.u$
0060: F4 9B AF 7F 08 E8 D6 8D 26 A7 2E 37 C1 A9 5A 2F    ........&..7..Z/
0070: 1F 05 A5 18 92 AE F2 94 97 32 B6 2A 38 AA DD 58    .........2.*8..X
  • After many tests, it turns out to be a fixed value.

Final summary

  • The whole flow is basically analyzed at this point.
  • I didn’t write the final logic code, and I can’t — I have no business need for it. If you need it, just code it up; the whole flow is clear. Analysis took four to five hours, and writing it up took another four. It’s 4:01 AM as I type this.
  • If anything’s wrong, corrections are welcome.

2023-12-20 10:34 update

  • Per expert feedback, that value is not fixed — I missed something. It looked fixed only because our input was fixed. The logs show we skipped one more method: the one labeled “table lookup”.
  • Let’s catch this escapee and see what it is.
debugger.addBreakPoint(module, 0x4500);
  • Also looks like a pointer — let’s look at what it points to.
  • There’s a reference; where is it used?
  • The sha512 result!
  • What’s sha512’s input text?
  • Another reference.
  • Still the random value. So we know its input is the sha512 of the random value.
  • Let’s briefly analyze this method: it takes one result argument, then extracts three values from it:
0-->80 60 35 40 
1-->00 00 00 00 
2-->40 00 00 00

v1 = result[2];
v2 = *result;
v3 = result[1];

so v1=0x40, v2=0x40356080, v3=0
  • Looking at it this way, the first if ( v1 ) is true, and the second if ( v1 < 0x20 ) is false.
  • Scrolling down, only here does it write to the address — let’s debug it.
  • The corresponding assembly:
  • Set a breakpoint and look.
b0x4560
>>> q0=0xa43b73a6c7b3520e096231b8e6c2d44d(1.8056244479338115E-263, -3.7768713601031674E-134) q1=0x24751757db396b198ab59a822b46b21c(-4.496232995265719E-257, 4.642831181428702E-133)
  • What are these q0 and q1? Aren’t they the “fixed value” we mentioned above?
  • Run once more and see:
q0=0x2f5aa9c1372ea7268dd6e8087faf9bf4(-5.367597597728704E-242, 1.4054386263309782E-80) q1=0x58ddaa382ab6329794f2ae9218a5051f(-9.09210480120694E-208, 1.1969099196948154E120)
  • Run to the end and combine with the “fixed value” from above:
0040: 4D D4 C2 E6 B8 31 62 09 0E 52 B3 C7 A6 73 3B A4    M....1b..R...s;.
0050: 1C B2 46 2B 82 9A B5 8A 19 6B 39 DB 57 17 75 24    ..F+.....k9.W.u$
0060: F4 9B AF 7F 08 E8 D6 8D 26 A7 2E 37 C1 A9 5A 2F    ........&..7..Z/
0070: 1F 05 A5 18 92 AE F2 94 97 32 B6 2A 38 AA DD 58    .........2.*8..X
  • It matches perfectly. So that’s where it comes from — the escapee is caught.

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch