Inside ttEncrypt - Tracing a VM-Protected JNI Cipher Down to SHA-512 and AES
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
Inside ttEncrypt - Tracing a VM-Protected JNI Cipher Down to SHA-512 and AES
ttEncrypt is a JNI native method that encrypts an HTTP request body. Its Java signature is deceptively simple:
public static native byte[] ttEncrypt(byte[] bArr, int i);Pass in a byte array and a length, get a byte array back. The native side lives in libEncryptor.so, and it’s wrapped in a custom virtual machine. The punchline is that the VM itself doesn’t do the cryptography — it just orchestrates calls to SHA-512 and AES. You don’t have to fully de-virtualize it to recover the algorithm. Here’s the whole trace.
Finding the native implementation
Loading the .so in IDA shows no direct JNI binding, so I hooked RegisterNatives to catch where the method is registered at runtime:
Interceptor.attach(addrRegisterNatives, {
onEnter: function (args) {
console.log("[RegisterNatives] method_count:", args[3], DebugSymbol.fromAddress(this.returnAddress));
var env = args[0];
var java_class = args[1];
var class_name = Java.vm.tryGetEnv().getClassName(java_class);
var methods_ptr = ptr(args[2]);
var method_count = parseInt(args[3]);
for (var i = 0; i < method_count; i++) {
var name_ptr = Memory.readPointer(methods_ptr.add(i * Process.pointerSize * 3));
var sig_ptr = Memory.readPointer(methods_ptr.add(i * Process.pointerSize * 3 + Process.pointerSize));
var fnPtr_ptr = Memory.readPointer(methods_ptr.add(i * Process.pointerSize * 3 + Process.pointerSize * 2));
var name = Memory.readCString(name_ptr);
var sig = Memory.readCString(sig_ptr);
var find_module = Process.findModuleByAddress(fnPtr_ptr);
console.log("[RegisterNatives] java_class:", class_name, "name:", name, "sig:", sig, "fnPtr:", fnPtr_ptr, "module_name:", find_module.name, "module_base:", find_module.base, "offset:", ptr(fnPtr_ptr).sub(find_module.base));
}
},
onLeave: function (retval) { }
});Output (class name anonymized):
[RegisterNatives] method_count: 0x1 0x76d0acbd6c libEncryptor.so!0xd6c
[RegisterNatives] java_class: com.[redacted].frameworks.encryptor.EncryptorUtil name: ttEncrypt sig: ([BI)[B fnPtr: 0x76d0ad2d88 module_name: libEncryptor.so module_base: 0x76d0acb000 offset: 0x7d88The native implementation is at file offset 0x7d88.
sub_7D88 — the JNI wrapper
The decompiled entry point (with variable types restored):
jbyteArray __fastcall sub_7D88(JNIEnv *env, jclass thiz, jbyteArray jarray, jint size)
{
unsigned __int64 v4; // x25
int v5; // w24
jbyteArray v6; // x19
JNIEnv *v7; // x20
JNIEnv *v8; // x23
jbyte *v9; // x21
__int64 v10; // x22
__int64 v11; // [xsp+0h] [xbp-50h]
__int64 v12; // [xsp+8h] [xbp-48h]
v4 = _ReadStatusReg(ARM64_SYSREG(3, 3, 13, 0, 2));
v5 = size;
v6 = jarray;
v7 = env;
v8 = 0LL;
v12 = *(_QWORD *)(v4 + 40);
if ( jarray && (signed int)size >= 1 )
{
env = (JNIEnv *)(*env)->GetByteArrayElements(env, jarray, 0LL);
v9 = (jbyte *)env;
if ( env )
{
v11 = v5 + 118;
v10 = malloc(v5 + 118);
if ( v10 )
{
sub_2BD4((__int64)v9, v5, v10, (__int64)&v11);
if ( v11 )
{
v8 = (JNIEnv *)((__int64 (__fastcall *)(JNIEnv *))(*v7)->NewByteArray)(v7);
((void (__fastcall *)(JNIEnv *, JNIEnv *, _QWORD, _QWORD, __int64))(*v7)->SetByteArrayRegion)(
v7,
v8,
0LL,
(unsigned int)v11,
v10);
}
else
{
v8 = 0LL;
}
((void (__fastcall *)(JNIEnv *, jbyteArray, jbyte *, _QWORD))(*v7)->ReleaseByteArrayElements)(v7, v6, v9, 0LL);
env = (JNIEnv *)free(v10);
}
else
{
env = (JNIEnv *)((__int64 (__fastcall *)(JNIEnv *, jbyteArray, jbyte *, _QWORD))(*v7)->ReleaseByteArrayElements)(
v7,
v6,
v9,
0LL);
v8 = 0LL;
}
}
else
{
v8 = 0LL;
}
}
if ( *(_QWORD *)(v4 + 40) == v12 )
env = v8;
return (jbyteArray)env;
}It grabs the jbyteArray, turns it into a jbyte *, allocates a fresh size + 118 buffer, and hands it to sub_2BD4. If the resulting length is positive it copies the result back into a new jbyteArray; otherwise it returns null. So sub_2BD4 is where the real work happens, and the ciphertext is size + 118 bytes — 118 bytes of fixed overhead.
sub_2BD4 — the trampoline into the VM
void __fastcall sub_2BD4(jbyte *a1, jsize a2, char *a3, int *a4)
{
unsigned __int64 v4; // x19
jbyte *v5; // [xsp+8h] [xbp-528h]
jsize v6; // [xsp+10h] [xbp-520h]
char *v7; // [xsp+18h] [xbp-518h]
int *v8; // [xsp+20h] [xbp-510h]
__int64 (__fastcall *v9)(__int64 (__fastcall *)(__int64), __int64); // [xsp+28h] [xbp-508h]
__int64 *v10; // [xsp+30h] [xbp-500h]
__int64 v11; // [xsp+510h] [xbp-20h]
__int64 v12; // [xsp+518h] [xbp-18h]
v4 = _ReadStatusReg(ARM64_SYSREG(3, 3, 13, 0, 2));
v12 = *(_QWORD *)(v4 + 40);
v5 = a1;
v6 = a2;
v7 = a3;
v8 = a4;
v9 = sub_2D1C;
v10 = &v11;
sub_2D28((unsigned int *)&unk_B2C0, (__int64)&v5, 0LL, (__int64)&off_21CE0, (__int64)&v9);
*(_QWORD *)(v4 + 40);
}sub_2BD4 just marshals the four caller arguments and jumps into sub_2D28. Looking at the assembly to see how the arguments actually travel:
sub_2D28 takes five arguments:
0xB2C0— a data-segment address that looks like a bytecode sequence.- An argument pointer — a struct holding the four original parameters (
jbyte *,jsize,char *,int *). 0.0x21CE0— what looks like an address array (a function table).- An address holding
sub_2D1Candsp + 0x510.
That’s a textbook VM dispatcher signature: a bytecode pointer, a register/argument block, and a table of handler functions.
sub_2D28 — the VM interpreter
sub_2D28 decompiles into a single enormous switch. The core loop fetches a 32-bit word of bytecode and decodes its fields by shifting and masking:
unsigned int *__fastcall sub_2D28(unsigned int *result, __int64 a2, __int64 a3, __int64 a4, __int64 a5)
{
// ... (hundreds of locals)
v6 = *(unsigned int **)a5;
v5 = *(_QWORD *)(a5 + 8);
*(_QWORD *)(v5 - 56) = 0LL;
*(_QWORD *)(v5 - 304) = 0LL;
v7 = (unsigned int **)(v5 - 312);
v8 = (_QWORD *)(v5 - 272);
v9 = (_QWORD *)(v5 - 264);
*(_QWORD *)(v5 - 72) = (v5 - 328) & 0xFFFFFFFFFFFFFFF0LL;
*(_QWORD *)(v5 - 32) = 0LL;
*(_QWORD *)(v5 - 272) = a2;
*(_QWORD *)(v5 - 264) = a3;
*(_QWORD *)(v5 - 256) = a4;
*(_QWORD *)(v5 - 248) = v6;
*(_QWORD *)(v5 - 312) = result;
*(_QWORD *)(v5 - 8) = result;
if ( result )
{
v10 = 0LL;
v11 = (unsigned __int64 *)(v5 - 40);
v12 = (unsigned __int64 *)(v5 - 48);
while ( 1 )
{
v13 = *result; // fetch the current bytecode word
if ( v10 == 2 )
{
v10 = 3LL;
*(_QWORD *)(v5 - 32) = 3LL;
}
v14 = v13 & 0x10000000;
v15 = v13 & 0x20000000;
v16 = v13 & 0x40000000;
v17 = (v13 >> 11) & 2 | (v13 >> 31) | (v13 >> 11) & 4 | (v13 >> 11) & 8 | (v13 >> 11) & 0x10;
v18 = ((v13 & 0x10000000) >> 26) & 0xFFFFFFFC | (v13 >> 26) & 3 | ((v13 & 0x20000000) >> 26);
v19 = (v13 >> 21) & 0x1F;
v20 = (v13 >> 16) & 0x1F;
// ... v21/v22/v23 mask further operand bits ...
switch ( v17 )
{
case 0u: case 2u: case 8u: case 0x12u: case 0x1Fu: case 0x21u: case 0x22u: case 0x27u:
case 0x2Bu: case 0x30u: case 0x31u: case 0x33u: case 0x37u:
// load family: signed/unsigned loads at byte/word/dword/qword widths, with bitfield extraction
...
case 3u: case 5u: case 0xFu: case 0x1Au: case 0x2Cu: case 0x2Du: case 0x3Au: case 0x3Fu:
// compare / conditional-branch family
...
case 4u: case 0x20u: case 0x38u: case 0x3Eu:
// arithmetic/bitwise on the register file (XOR / AND / OR, immediate forms)
...
case 0xAu: case 0xBu: case 0xEu: case 0x14u: case 0x16u: case 0x24u: case 0x36u: case 0x3Bu:
// store family: byte/word/dword/qword stores with bitfield merge
...
case 0xDu: case 0x15u: case 0x1Bu: case 0x28u:
// add-immediate
...
case 0x10u:
// bitfield insert/extract and byte-swap (__ROR8__ etc.)
...
case 0x11u: case 0x2Au:
// branch / PC manipulation
...
case 0x17u:
// compare-and-branch, signed/unsigned
...
case 0x29u:
// arithmetic: add/sub/mul, including 64x64->128 multiply (v11/v12 hold hi/lo)
...
case 0x2Fu:
// bitwise and shifts: XOR/AND/OR/NOT, logical and arithmetic shifts, rotates
...
}
}
}
return result;
}A few things jump out from the decode logic:
- Registers live on the stack.
v5 - 304is a register file indexed by 5-bit register selectors (32 registers), andv5 - 40/v5 - 48are the high/low halves of a 64×64→128 multiply result. - The bytecode word packs everything into 32 bits. The opcode is spread across several high bits (
v17mixes>>11,>>31, and flag bits), while two 5-bit fields (v19,v20) select source and destination registers, and immediates are reconstructed from several scattered bit ranges. - The instruction set is a full mini-ISA. Loads and stores at four widths with bitfield merge, arithmetic including 128-bit multiply, full bitwise/rotate/shift coverage, and several compare-and-branch variants. It’s a real CPU, not a toy.
The key observation — the VM calls out, it doesn’t compute
The important part isn’t any single case. It’s that throughout the interpreter, operand values feed into calls through the function table at 0x21CE0, and the argument block a2 is what carries data in and out. The bytecode sequences stored at 0xB2C0 don’t inline a cipher — they orchestrate calls to external functions.
That matches the cleanup: sub_2D1C (passed in through argument 5) is the callback invoked when the VM finishes. The VM is a scheduler around host functions, not an in-VM implementation of the encryption.
So instead of fully de-virtualizing sub_2D28 (thousands of lines of switch), I just hooked the external functions the VM calls — SHA-512 and AES — and reconstructed the algorithm directly from what flows through them.
Reconstructing the algorithm
import os
import hashlib
from Crypto.Cipher import AES
from Crypto.Util.Padding import pad, unpad
def get_hash_key():
key1 = [0x1F, 0xDD, 0xA8, 0x33, 0x88, 0x07, 0xC7, 0x31, 0xB1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xEC, 0x5F,
0x60, 0x51, 0x7F, 0xA9, 0x19, 0xB5, 0x4A, 0x0D, 0x2D, 0xE5, 0x7A, 0x9F, 0x93, 0xC9, 0x9C, 0xEF,
0xA0, 0xE0, 0x3B, 0x4D, 0xAE, 0x2A, 0xF5, 0xB0, 0xC8, 0xEB, 0xBB, 0x3C, 0x83, 0x53, 0x99, 0x61,
0x17, 0x2B, 0x04, 0x7E, 0xBA, 0x77, 0xD6, 0x26, 0xE1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0C, 0x7D]
key2 = [0x52, 0x09, 0x6A, 0xD5, 0x30, 0x36, 0xA5, 0x38, 0xBF, 0x40, 0xA3, 0x9E, 0x81, 0xF3, 0xD7, 0xFB,
0x7C, 0xE3, 0x39, 0x82, 0x9B, 0x2F, 0xFF, 0x87, 0x34, 0x8E, 0x43, 0x44, 0xC4, 0xDE, 0xE9, 0xCB,
0x54, 0x7B, 0x94, 0x32, 0xA6, 0xC2, 0x23, 0x3D, 0xEE, 0x4C, 0x95, 0x0B, 0x42, 0xFA, 0xC3, 0x4E,
0x08, 0x2E, 0xA1, 0x66, 0x28, 0xD9, 0x24, 0xB2, 0x76, 0x5B, 0xA2, 0x49, 0x6D, 0x8B, 0xD1, 0x25]
hash_key = bytes()
for i in range(len(key1)):
hash_key += (key1[i] ^ key2[i]).to_bytes(1, 'little')
return hash_key
def sha512(data):
sha512_data = hashlib.sha512()
sha512_data.update(data)
return sha512_data.digest()
# the inner VM function derives key and iv from the random input
def get_aes_key_and_iv(random_data):
random_data_sha512 = sha512(random_data) # sha512 of the random bytes
hash_key = get_hash_key() # key computed by XOR, length 0x40
data = random_data_sha512 + hash_key # total length 0x80
key_iv_hash = sha512(data) # sha512 of the combined buffer
return key_iv_hash[0:16], key_iv_hash[16:32]
def aes_encrypt(data, key, iv):
cipher = AES.new(key, AES.MODE_CBC, iv)
padded_data = pad(data, AES.block_size)
encrypted = cipher.encrypt(padded_data)
return encrypted
def tt_encrypt(data):
magic = b"\x74\x63\x05\x10\x00\x00"
random_bytes = os.urandom(32)
aes_key, aes_iv = get_aes_key_and_iv(random_bytes)
data_sha512 = sha512(data)
raw_data = data_sha512 + data
aes_data = aes_encrypt(raw_data, aes_key, aes_iv)
return magic + random_bytes + aes_data
if __name__ == "__main__":
test_data = b"hello!!!"
print(tt_encrypt(test_data))So ttEncrypt is, end to end:
- Generate 32 random bytes.
- Derive the AES key and IV:
sha512(sha512(random) || hash_key)split into[0:16]and[16:32]. hash_keyiskey1 XOR key2, two 64-byte constants baked into the binary.- Prepend
sha512(plaintext)to the plaintext (integrity check), AES-CBC-encrypt the result with PKCS7 padding. - Return
magic (6 bytes) || random (32 bytes) || ciphertext.
That size + 118 we saw back in sub_7D88 is just 6 + 32 + 64 (the sha512 prefix) + 16 (PKCS7 padding to the block boundary) = 118 bytes of fixed overhead. It lines up exactly.
Wrapping up
The lesson here is a general one for VM-protected native code: don’t de-virtualize more than you have to. sub_2D28 is a full custom CPU — a 32-bit bytecode word, a stack register file, loads/stores at four widths, 128-bit multiply, full bitwise and shift coverage, compare-and-branch — and fully reversing it would be a multi-day project. But because the VM’s job was to call out to SHA-512 and AES rather than compute the cipher inline, hooking those two host functions was enough to recover the entire algorithm.
It’s also a reminder that a VM wrapper doesn’t automatically mean strong cryptography. The actual protection here reduces to standard primitives with keys derived from constants and randomness that travels alongside the ciphertext — the VM is obfuscation of control flow, not of cryptographic strength.
The VM itself, though, is worth studying as a foundation: the same bytecode format and dispatch structure shows up, in more complex forms, in the app’s other signature algorithms. Once you can read this one, the rest of the family falls into place.
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch