Three Ways to Protect Android Native Libraries - Function Encryption, Program Header Obfuscation, and a Custom ELF Format
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
Here are three ways to protect an Android native library (.so) that I implemented over about two weeks. Working through them taught me a lot about ELF internals and the Android loader. The first two build on an earlier writeup on SO-encryption anti-analysis; the third — a custom ELF format plus a custom linker — is my own extension.
Before reading, it helps to be familiar with the ELF format and the Android SO loading flow.
1. Technique one: encrypting a function
We start from this source, compiled into libmathlib.so, and we want to protect int mymyadd(int a, int b):
#include <android/log.h>
#define LOG_TAG "MathLib"
#define LOGD(...) __android_log_print(ANDROID_LOG_DEBUG, LOG_TAG, __VA_ARGS__)
extern "C"
int mymyadd(int a, int b) {
int result = a + b;
LOGD("Add: %d + %d = %d", a, b, result);
return result;
}
extern "C"
int subtract(int a, int b) {
int result = a - b;
LOGD("Subtract: %d - %d = %d", a, b, result);
return result;
}1.1 Effect
Before encryption:
After encryption, IDA naturally can no longer recognize the function:
1.2 Principle
To encrypt a function, you first need to locate it in the ELF file and know its size. The cleanest way is to mirror what the dynamic linker does when resolving dlsym.
dlsym() resolves a symbol through the GNU hash table (or the classic ELF hash table when GNU hash is absent). The linker’s lookup chain is dlsym() → __loader_dlsym() → dlsym_impl() → do_dlsym(), which ends at find_symbol_by_name():
// bionic/linker/linker_soinfo.cpp
const ElfW(Sym)* soinfo::find_symbol_by_name(SymbolName& symbol_name,
const version_info* vi) const {
return is_gnu_hash() ? gnu_lookup(symbol_name, vi) : elf_lookup(symbol_name, vi);
}The GNU-hash variant:
// bionic/linker/linker_soinfo.cpp
const ElfW(Sym)* soinfo::gnu_lookup(SymbolName& symbol_name, const version_info* vi) const {
const uint32_t hash = symbol_name.gnu_hash();
constexpr uint32_t kBloomMaskBits = sizeof(ElfW(Addr)) * 8;
const uint32_t word_num = (hash / kBloomMaskBits) & gnu_maskwords_;
const ElfW(Addr) bloom_word = gnu_bloom_filter_[word_num];
const uint32_t h1 = hash % kBloomMaskBits;
const uint32_t h2 = (hash >> gnu_shift2_) % kBloomMaskBits;
// test against bloom filter
if ((1 & (bloom_word >> h1) & (bloom_word >> h2)) == 0) {
return nullptr;
}
// bloom test says "probably yes"...
uint32_t n = gnu_bucket_[hash % gnu_nbucket_];
if (n == 0) {
return nullptr;
}
const ElfW(Versym) verneed = find_verdef_version_index(this, vi);
const ElfW(Versym)* versym = get_versym_table();
do {
ElfW(Sym)* s = symtab_ + n;
if (((gnu_chain_[n] ^ hash) >> 1) == 0 &&
check_symbol_version(versym, n, verneed) &&
strcmp(get_string(s->st_name), symbol_name.get_name()) == 0 &&
is_symbol_global_and_defined(this, s)) {
return symtab_ + n;
}
} while ((gnu_chain_[n++] & 1) == 0);
return nullptr;
}The GNU hash of a symbol name is straightforward to compute:
static uint32_t gnu_hash(const char *s0)
{
const unsigned char *s = (void *)s0;
uint_fast32_t h = 5381;
for (; *s; s++)
h += h*32 + *s;
return h;
}The rest of the metadata (gnu_maskwords_, gnu_bloom_filter_, symtab_, etc.) comes from the dynamic section. The linker walks the program header table to find PT_DYNAMIC, then reads the DT_GNU_HASH, DT_SYMTAB, DT_STRTAB, and DT_STRSZ entries:
// bionic/linker/linker.cpp
case DT_GNU_HASH:
gnu_nbucket_ = reinterpret_cast<uint32_t*>(load_bias + d->d_un.d_ptr)[0];
// skip symndx
gnu_maskwords_ = reinterpret_cast<uint32_t*>(load_bias + d->d_un.d_ptr)[2];
gnu_shift2_ = reinterpret_cast<uint32_t*>(load_bias + d->d_un.d_ptr)[3];
gnu_bloom_filter_ = reinterpret_cast<ElfW(Addr)*>(load_bias + d->d_un.d_ptr + 16);
gnu_bucket_ = reinterpret_cast<uint32_t*>(gnu_bloom_filter_ + gnu_maskwords_);
// amend chain for symndx = header[1]
gnu_chain_ = gnu_bucket_ + gnu_nbucket_ -
reinterpret_cast<uint32_t*>(load_bias + d->d_un.d_ptr)[1];
if (!powerof2(gnu_maskwords_)) {
DL_ERR("invalid maskwords for gnu_hash = 0x%x, in \"%s\" expecting power to two",
gnu_maskwords_, get_realpath());
return false;
}
--gnu_maskwords_;
flags_ |= FLAG_GNU_HASH;
break;1.3 Implementation
With the symbol located, we can encrypt the function bytes in place. harden_handle reads the ELF header, program headers, and dynamic section, resolves the target symbol through GNU hash, then RC4-encrypts FUNC_SIZE bytes at the symbol’s st_value:
int harden_handle(char *src_path, char *name, char *dst_path, int mode) {
Elf64_Ehdr header_;
Elf64_Phdr phdr_;
Elf64_Dyn dyn_;
int dyn_off;
int dyn_size;
int dyn_count;
Elf64_Addr dyn_symtab;
Elf64_Addr dyn_strtab;
Elf64_Addr dyn_gnuhash;
int dyn_strsz;
uint32_t symndex;
uint32_t gnu_nbucket_;
uint32_t gnu_maskwords_;
uint32_t gnu_shift2_;
Elf64_Addr *gnu_bloom_filter_;
uint32_t* gnu_bucket_;
uint32_t* gnu_chain_;
bool has_gnu_hash = false;
int fd = open(src_path, O_RDONLY);
if (fd == -1) {
LOGE("error opening source file");
return -1;
}
// read ELF header
int ret = read(fd, &header_, sizeof(header_));
if (ret < 0) {
LOGE("error read file!");
}
// read program headers, find PT_DYNAMIC
lseek(fd, header_.e_phoff, SEEK_SET);
for (int i = 0; i < header_.e_phnum; i++) {
ret = read(fd, &phdr_, sizeof(phdr_));
if (ret < 0) {
LOGE("error read file!");
}
if (phdr_.p_type != PT_DYNAMIC) {
continue;
}
dyn_off = phdr_.p_offset;
dyn_size = phdr_.p_filesz;
dyn_count = phdr_.p_memsz / (8 * 2);
}
// read dynamic section
lseek(fd, dyn_off, SEEK_SET);
for (int i = 0; i < dyn_count; i++) {
ret = read(fd, &dyn_, sizeof(dyn_));
if (ret < 0) {
LOGI("error read file!");
}
switch (dyn_.d_tag) {
case DT_GNU_HASH:
dyn_gnuhash = dyn_.d_un.d_ptr;
break;
case DT_SYMTAB:
dyn_symtab = dyn_.d_un.d_ptr;
break;
case DT_STRTAB:
dyn_strtab = dyn_.d_un.d_ptr;
break;
case DT_STRSZ:
dyn_strsz = dyn_.d_un.d_val;
break;
}
}
char *dynstr = (char*) malloc(dyn_strsz);
if (dynstr == NULL) {
LOGE("malloc failed");
}
lseek(fd, dyn_strtab, SEEK_SET);
ret = read(fd, dynstr, dyn_strsz);
if (ret < 0) {
LOGE("read .dynstr failed");
}
// GNU hash bucket lookup
gnu_nbucket_ = dyn_gnuhash[0];
uint32_t symndx = dyn_gnuhash[1];
gnu_maskwords_ = dyn_gnuhash[2];
gnu_shift2_ = dyn_gnuhash[3];
gnu_bloom_filter_ = reinterpret_cast<Elf64_Addr *>(dyn_gnuhash + 16);
gnu_bucket_ = reinterpret_cast<uint32_t*>(gnu_bloom_filter_ + gnu_maskwords_);
gnu_chain_ = gnu_bucket_ + gnu_nbucket_ - symndx;
uint32_t hash = gnu_hash(name);
uint32_t h2 = hash >> gnu_shift2_;
uint32_t bloom_mask_bits = sizeof(Elf64_Addr) * 8;
uint32_t word_num = (hash / bloom_mask_bits) & gnu_maskwords_;
uint32_t val = hash % gnu_nbucket_;
uint32_t n = gnu_bucket_[hash % gnu_nbucket_];
if (n == 0) {
printf("symbol '%s' hash bucket is empty\n", name);
return false;
}
Elf64_Sym s;
uint32_t chain;
do {
lseek(fd, dyn_symtab + n * sizeof(Elf64_Sym), SEEK_SET);
ret = read(fd, &s, sizeof(Elf64_Sym));
if (ret < 0) {
LOGI("read gnuhash failed");
}
LOGI("name = %d %s", s.st_name, dynstr + s.st_name);
lseek(fd, reinterpret_cast<uint64_t>(gnu_chain_ + n), SEEK_SET);
ret = read(fd, &chain, sizeof(chain));
if (ret < 0) {
LOGI("read gnuhash failed");
}
if (((chain ^ hash) >> 1) == 0 && strcmp(dynstr + s.st_name, name) == 0) {
LOGI("found function(%s) at %p(%zd)", name, reinterpret_cast<void*>(s.st_value), static_cast<size_t>(s.st_size));
break;
}
n++;
lseek(fd, reinterpret_cast<uint64_t>(gnu_chain_ + n), SEEK_SET);
ret = read(fd, &chain, sizeof(chain));
if (ret < 0) {
LOGI("read gnuhash failed");
}
} while ((chain & 1) == 0);
uint32_t size = get_file_size(src_path);
char *file_buf = (char *)malloc(size);
if (file_buf == NULL) {
LOGI("file buf malloc failed");
}
lseek(fd, 0, SEEK_SET);
ret = read(fd, file_buf, size);
if (ret < 0) {
LOGI("read file buf failed");
}
close(fd);
// modify and save the so
char save_path[128] = {0};
fd = open(dst_path, O_RDWR | O_CREAT, 0644);
if (fd == -1) {
LOGI("error opening file, %s", dst_path);
return -1;
}
char *encrypt_buf = (char *)malloc(s.st_size);
encrypt((unsigned char *) RC4_KEY, reinterpret_cast<unsigned char *>(encrypt_buf),
reinterpret_cast<unsigned char *>(&file_buf[s.st_value]), FUNC_SIZE);
memcpy(&file_buf[s.st_value], encrypt_buf, FUNC_SIZE);
if (mode == MODE_DUPLEX) {
for (int i = 0; i < header_.e_phnum; i++) {
file_buf[header_.e_phoff + i * sizeof(phdr_)] = file_buf[header_.e_phoff + i * sizeof(phdr_)] ^ XOR_MAGIC;
file_buf[header_.e_phoff + i * sizeof(phdr_) + 1] = file_buf[header_.e_phoff + i * sizeof(phdr_) + 1] ^ XOR_MAGIC;
file_buf[header_.e_phoff + i * sizeof(phdr_) + 2] = file_buf[header_.e_phoff + i * sizeof(phdr_) + 2] ^ XOR_MAGIC;
file_buf[header_.e_phoff + i * sizeof(phdr_) + 3] = file_buf[header_.e_phoff + i * sizeof(phdr_) + 3] ^ XOR_MAGIC;
}
}
write(fd, file_buf, size);
free(encrypt_buf);
close(fd);
return 0;
}Note the original author hardcoded FUNC_SIZE. If you don’t want that, you have to parse the ELF dynamic section yourself to walk the symbol table and recover each function’s size — the same work the encryptor already does.
1.4 Runtime call flow
At runtime, the function is decrypted on demand, executed, then re-encrypted:
dlopenthe library anddlsymthe function address.- Change memory permissions, decrypt over the function range, restore permissions.
- Call the function.
- Change permissions, re-encrypt over the function range, restore permissions.
extern "C"
JNIEXPORT void JNICALL
Java_com_example_mylinkerwith21_MainActivity_loadSO(JNIEnv *env, jobject thiz,
jstring jEncryptedFileDirectoryPath) {
void *lib;
uint64_t start;
uint64_t end;
ssize_t page;
FUNC lib_func = NULL;
RC4_CTX ctx;
char buf[512] = {0};
int result;
LOGI("loadSO called!");
const char *encryptedFileDirectoryPath = env->GetStringUTFChars(jEncryptedFileDirectoryPath, nullptr);
if (encryptedFileDirectoryPath == nullptr) {
LOGE("Failed to get string characters for encrypted file directory path");
return;
}
const char* encryptedFileName = "libmathlib_encrypt.so";
size_t requiredPathSize = strlen(encryptedFileDirectoryPath) + 1 + strlen(encryptedFileName) + 1;
char* encryptedLoadPath = (char*)malloc(requiredPathSize);
if (encryptedLoadPath == nullptr) {
LOGE("Failed to allocate memory for load path");
env->ReleaseStringUTFChars(jEncryptedFileDirectoryPath, encryptedFileDirectoryPath);
return;
}
sprintf(encryptedLoadPath, "%s/%s", encryptedFileDirectoryPath, encryptedFileName);
LOGI("Loading from path: %s", encryptedLoadPath);
lib = dlopen(encryptedLoadPath, RTLD_LAZY);
free(encryptedLoadPath);
env->ReleaseStringUTFChars(jEncryptedFileDirectoryPath, encryptedFileDirectoryPath);
if (lib == NULL) {
LOGE("%s dlopen failed: %s\n", encryptedFileName, dlerror());
return;
}
lib_func = reinterpret_cast<FUNC>(dlsym(lib, "mymyadd"));
if (!lib_func) {
LOGI("can't find module symbol 'mymyadd': %s\n", dlerror());
dlclose(lib);
return;
}
LOGI("loadso lib_func = %p", lib_func);
parse_maps_for_lib("libmathlib_encrypt.so", &start, &end);
if (start == 0 && end == 0) {
LOGI("Failed to find memory map for libmathlib_encrypt.so");
dlclose(lib);
return;
}
page = end - start;
// mprotect requires page-aligned address and size
if (mprotect((void*)start, page, PROT_READ | PROT_WRITE | PROT_EXEC) == -1) {
LOGI("mprotect failed: %s", strerror(errno));
dlclose(lib);
return;
}
// decrypt and overwrite
rc4_init(&ctx, (unsigned char *) RC4_KEY, strlen(reinterpret_cast<const char *>(RC4_KEY)));
rc4_run(&ctx, reinterpret_cast<unsigned char *>(buf), reinterpret_cast<unsigned char *>(lib_func), FUNC_SIZE);
memcpy((void*)lib_func, buf, FUNC_SIZE);
// restore permissions
if (mprotect((void*)start, page, PROT_READ | PROT_EXEC) == -1) {
LOGI("mprotect restore failed: %s", strerror(errno));
}
// call the function
result = lib_func(1, 2);
LOGI("loadso add result = %d", result);
// re-encrypt
if (mprotect((void*)start, page, PROT_READ | PROT_WRITE | PROT_EXEC) == -1) {
LOGI("mprotect re-encrypt failed: %s", strerror(errno));
} else {
rc4_init(&ctx, (unsigned char *) RC4_KEY, strlen(reinterpret_cast<const char *>(RC4_KEY)));
rc4_run(&ctx, reinterpret_cast<unsigned char *>(buf), reinterpret_cast<unsigned char *>(lib_func), FUNC_SIZE);
memcpy((void*)lib_func, buf, FUNC_SIZE);
if (mprotect((void*)start, page, PROT_READ | PROT_EXEC) == -1) {
LOGI("mprotect restore after re-encrypt failed: %s", strerror(errno));
}
}
dlclose(lib);
}2. Technique two: XOR the program-header type
2.1 Effect
2.2 Principle
On top of technique one, each program header’s p_type is XORed with a magic constant. The code is the MODE_DUPLEX branch already shown above.
The catch: this only modifies the program header table, not the section header table, so IDA can still open the file and recover most information statically through the section headers. That’s the motivation for technique three.
3. Technique three: a custom ELF format and a custom linker
3.1 Effect
A file packed this way is unparseable by IDA:
Left is the standard unencrypted so; right is the custom-format, RC4-encrypted so:
And after RC4 decryption:
3.2 Implementation
The custom file header doesn’t have to match the standard ELF header. I kept it mostly identical for convenience — in practice you can reorder fields, compute offsets on the fly, or drop unused parts entirely. The magic number is replaced with .csf, and since we implement the linker ourselves, there’s no external validation to satisfy.
The packing logic (do_pack) copies the original file, writes a custom file header at the front, RC4-encrypts the ELF header and program header table into a trailing area, RC4-encrypts every PT_LOAD segment in place, and zeroes out the original header and program header table. The unpacker (unpack / restore_elf) reverses it:
#pragma pack(push, 1)
typedef struct custom_elf64_file{
Elf64_Off elf_header_off;
Elf64_Half elf_header_size;
Elf64_Off elf_program_header_table_off;
Elf64_Half elf_program_header_table_num;
Elf64_Half elf_program_header_table_size;
} Custom_Elf64_File;
#pragma pack(pop)
typedef struct
{
unsigned char e_ident[EI_NIDENT];
Elf64_Half e_type;
Elf64_Half e_machine;
Elf64_Word e_version;
Elf64_Addr e_entry;
Elf64_Off e_phoff;
Elf64_Off e_shoff;
Elf64_Word e_flags;
Elf64_Half e_ehsize;
Elf64_Half e_phentsize;
Elf64_Half e_phnum;
Elf64_Half e_shentsize;
Elf64_Half e_shnum;
Elf64_Half e_shstrndx;
} Custom_Elf64_Ehdr;
int do_pack(char *inputfile_buffer, size_t inputfile_size, char *outfile_buffer, size_t outfile_size)
{
if (NULL == inputfile_buffer || 0 == inputfile_size || NULL == outfile_buffer) {
return -1;
}
Elf64_Ehdr* orig_ehdr = (Elf64_Ehdr *)inputfile_buffer;
if (memcmp(orig_ehdr->e_ident, ELFMAG, SELFMAG) != 0) {
return -2;
}
size_t phdr_table_size = orig_ehdr->e_phnum * orig_ehdr->e_phentsize;
size_t enc_ehdr_offset = inputfile_size;
size_t enc_phdr_offset = enc_ehdr_offset + orig_ehdr->e_ehsize;
size_t final_size = enc_phdr_offset + phdr_table_size;
if (final_size > outfile_size) {
return -3;
}
memcpy(outfile_buffer, inputfile_buffer, inputfile_size);
// build the custom file header
Custom_Elf64_File custom_file = {0};
custom_file.elf_header_off = enc_ehdr_offset;
custom_file.elf_header_size = orig_ehdr->e_ehsize;
custom_file.elf_program_header_table_off = enc_phdr_offset;
custom_file.elf_program_header_table_num = orig_ehdr->e_phnum;
custom_file.elf_program_header_table_size = phdr_table_size;
Custom_Elf64_Ehdr custom_ehdr = {0};
memcpy(&custom_ehdr, orig_ehdr, sizeof(Elf64_Ehdr));
// change the magic number, keep the rest of e_ident
memcpy(custom_ehdr.e_ident, ".csf", 4);
// encrypt and append the modified ELF header
encrypt((unsigned char *)RC4_KEY, RC4_KEY_LEN,
reinterpret_cast<unsigned char *>(outfile_buffer + enc_ehdr_offset),
reinterpret_cast<unsigned char *>(&custom_ehdr),
orig_ehdr->e_ehsize);
// encrypt and append the program header table
encrypt((unsigned char *)RC4_KEY, RC4_KEY_LEN,
reinterpret_cast<unsigned char *>(outfile_buffer + enc_phdr_offset),
reinterpret_cast<unsigned char *>(inputfile_buffer + orig_ehdr->e_phoff),
phdr_table_size);
// encrypt every PT_LOAD segment in place
Elf64_Phdr *phdr = (Elf64_Phdr *)(inputfile_buffer + orig_ehdr->e_phoff);
for (int i = 0; i < orig_ehdr->e_phnum; i++) {
if (phdr[i].p_type == PT_LOAD) {
if (phdr[i].p_offset + phdr[i].p_filesz > inputfile_size) {
return -4;
}
LOGI("PT_LOAD #%d offset=%llu size=%llu", i, phdr[i].p_offset, phdr[i].p_filesz);
encrypt((unsigned char *)RC4_KEY, RC4_KEY_LEN,
reinterpret_cast<unsigned char *>(outfile_buffer + phdr[i].p_offset),
reinterpret_cast<unsigned char *>(outfile_buffer + phdr[i].p_offset),
phdr[i].p_filesz);
}
}
// zero the original ELF header and program header table
memset(outfile_buffer, 0, sizeof(Elf64_Ehdr));
memset(outfile_buffer + orig_ehdr->e_phoff, 0, phdr_table_size);
// write the custom file header at the front
memcpy(outfile_buffer, &custom_file, sizeof(Custom_Elf64_File));
return 0;
}
int unpack(char *elf_pack_data, size_t file_size, Custom_Elf64_File my_elf64_file)
{
if (!elf_pack_data) {
return -1;
}
if (my_elf64_file.elf_header_size == 0 ||
my_elf64_file.elf_program_header_table_size == 0 ||
my_elf64_file.elf_program_header_table_num == 0) {
return -2;
}
if (my_elf64_file.elf_header_off >= file_size ||
my_elf64_file.elf_program_header_table_off >= file_size) {
return -3;
}
Custom_Elf64_File original_custom_file;
memcpy(&original_custom_file, elf_pack_data, sizeof(Custom_Elf64_File));
char *decrypted_ehdr = (char *)malloc(my_elf64_file.elf_header_size);
if (!decrypted_ehdr) {
return -4;
}
char *decrypted_phdr = (char *)malloc(my_elf64_file.elf_program_header_table_size);
if (!decrypted_phdr) {
free(decrypted_ehdr);
return -5;
}
// decrypt the ELF header
decrypt((unsigned char *)RC4_KEY, RC4_KEY_LEN,
reinterpret_cast<unsigned char *>(decrypted_ehdr),
reinterpret_cast<unsigned char *>(elf_pack_data + my_elf64_file.elf_header_off),
my_elf64_file.elf_header_size);
Elf64_Ehdr *decrypted_elf_header = (Elf64_Ehdr *)decrypted_ehdr;
if (memcmp(decrypted_elf_header->e_ident, ".csf", 4) != 0) {
free(decrypted_ehdr);
free(decrypted_phdr);
return -6;
}
// decrypt the program header table
decrypt((unsigned char *)RC4_KEY, RC4_KEY_LEN,
reinterpret_cast<unsigned char *>(decrypted_phdr),
reinterpret_cast<unsigned char *>(elf_pack_data + my_elf64_file.elf_program_header_table_off),
my_elf64_file.elf_program_header_table_size);
Elf64_Phdr *decrypted_ph_table = (Elf64_Phdr *)decrypted_phdr;
if (decrypted_elf_header->e_phnum != my_elf64_file.elf_program_header_table_num) {
free(decrypted_ehdr);
free(decrypted_phdr);
return -7;
}
for (int i = 0; i < my_elf64_file.elf_program_header_table_num; i++) {
Elf64_Phdr *current_phdr = &decrypted_ph_table[i];
if (current_phdr->p_type == PT_LOAD) {
if (current_phdr->p_offset + current_phdr->p_filesz > file_size) {
free(decrypted_ehdr);
free(decrypted_phdr);
return -8;
}
}
}
// decrypt all LOAD segments
for (int i = 0; i < my_elf64_file.elf_program_header_table_num; i++) {
Elf64_Phdr *current_phdr = &decrypted_ph_table[i];
if (current_phdr->p_type == PT_LOAD && current_phdr->p_filesz > 0) {
decrypt((unsigned char *)RC4_KEY, RC4_KEY_LEN,
reinterpret_cast<unsigned char *>(elf_pack_data + current_phdr->p_offset),
reinterpret_cast<unsigned char *>(elf_pack_data + current_phdr->p_offset),
current_phdr->p_filesz);
}
}
// store the decrypted header and program header table back for restore
memcpy(elf_pack_data + my_elf64_file.elf_header_off, decrypted_ehdr, my_elf64_file.elf_header_size);
memcpy(elf_pack_data + my_elf64_file.elf_program_header_table_off, decrypted_phdr, my_elf64_file.elf_program_header_table_size);
// restore the original custom file header
memcpy(elf_pack_data, &original_custom_file, sizeof(Custom_Elf64_File));
free(decrypted_ehdr);
free(decrypted_phdr);
return 0;
}
int restore_elf(char *decrypted_data, size_t file_size, Custom_Elf64_File my_elf64_file,
char **restored_data, size_t *restored_size)
{
if (!decrypted_data || !restored_data || !restored_size) {
return -1;
}
Elf64_Ehdr *decrypted_elf_header = (Elf64_Ehdr *)(decrypted_data + my_elf64_file.elf_header_off);
Elf64_Phdr *decrypted_phdr_table = (Elf64_Phdr *)(decrypted_data + my_elf64_file.elf_program_header_table_off);
if (memcmp(decrypted_elf_header->e_ident, ".csf", 4) != 0) {
return -2;
}
// compute the restored file size from segment and section table ends
size_t max_segment_end = 0;
for (int i = 0; i < my_elf64_file.elf_program_header_table_num; i++) {
Elf64_Phdr *current_phdr = &decrypted_phdr_table[i];
size_t segment_end = current_phdr->p_offset + current_phdr->p_filesz;
if (segment_end > max_segment_end) {
max_segment_end = segment_end;
}
}
size_t section_table_end = 0;
if (decrypted_elf_header->e_shoff > 0 && decrypted_elf_header->e_shnum > 0) {
section_table_end = decrypted_elf_header->e_shoff +
decrypted_elf_header->e_shnum * decrypted_elf_header->e_shentsize;
}
size_t new_file_size = (max_segment_end > section_table_end) ? max_segment_end : section_table_end;
size_t min_size = decrypted_elf_header->e_phoff +
decrypted_elf_header->e_phnum * decrypted_elf_header->e_phentsize;
if (new_file_size < min_size) {
new_file_size = min_size;
}
if (new_file_size > my_elf64_file.elf_header_off && my_elf64_file.elf_header_off > min_size) {
new_file_size = my_elf64_file.elf_header_off;
}
*restored_data = (char *)malloc(new_file_size);
if (!*restored_data) {
return -3;
}
memcpy(*restored_data, decrypted_data, new_file_size);
Elf64_Ehdr restored_ehdr = {0};
memcpy(&restored_ehdr, decrypted_elf_header, sizeof(Elf64_Ehdr));
// restore the standard ELF magic
memcpy(restored_ehdr.e_ident, ELFMAG, SELFMAG);
memcpy(*restored_data, &restored_ehdr, sizeof(Elf64_Ehdr));
Elf64_Off phdr_restore_offset = restored_ehdr.e_phoff;
if (phdr_restore_offset < sizeof(Elf64_Ehdr) || phdr_restore_offset >= new_file_size) {
phdr_restore_offset = sizeof(Elf64_Ehdr);
((Elf64_Ehdr*)(*restored_data))->e_phoff = phdr_restore_offset;
}
size_t phdr_table_size = decrypted_elf_header->e_phnum * decrypted_elf_header->e_phentsize;
if (phdr_restore_offset + phdr_table_size > new_file_size) {
free(*restored_data);
*restored_data = NULL;
*restored_size = 0;
return -4;
}
memcpy(*restored_data + phdr_restore_offset, decrypted_phdr_table, phdr_table_size);
*restored_size = new_file_size;
Elf64_Ehdr *final_ehdr = (Elf64_Ehdr *)(*restored_data);
if (memcmp(final_ehdr->e_ident, ELFMAG, SELFMAG) != 0) {
free(*restored_data);
*restored_data = NULL;
*restored_size = 0;
return -5;
}
return 0;
}At this point you have two choices: restore to a standard ELF and load it, or load the custom format directly. I chose the latter. If you’re learning a custom-format loader, it’s easier to first restore to a standard ELF, get that loading, then adapt.
3.3 Principle: a custom loader, following the Android SO flow
The custom loader mirrors the Android linker’s ElfReader flow, with a few deliberate changes.
The key insight is that the custom format’s ELF header lives at the end of the file, not offset 0. DIY_Read keeps file_offset_ at 0 (so later memory mapping lines up) and stashes the custom header offset in origin_file_size_:
bool ElfReader::DIY_Read(const char* name, int fd, off64_t file_offset, off64_t file_size) {
if (did_read_) {
return true;
}
name_ = name;
fd_ = fd;
file_offset_ = 0;
file_size_ = file_size;
origin_file_size_ = file_offset;
if (DIY_ReadElfHeader() &&
DIY_VerifyElfHeader() &&
ReadProgramHeaders() &&
ReadSectionHeaders() &&
DIY_ReadDynamicSection()) {
did_read_ = true;
}
return did_read_;
}DIY_ReadElfHeader then uses that saved offset to read the header from the right place:
bool ElfReader::DIY_ReadElfHeader() {
LOGI("enter DIY_ReadElfHeader");
ssize_t rc = TEMP_FAILURE_RETRY(pread64(fd_, &header_, sizeof(header_), origin_file_size_));
if (rc < 0) {
LOGE("can't read file \"%s\": %s", name_.c_str(), strerror(errno));
return false;
}
if (rc != sizeof(header_)) {
LOGE("\"%s\" is too small to be an ELF executable: only found %zd bytes", name_.c_str(),
static_cast<size_t>(rc));
return false;
}
return true;
}Header verification is the same as the original except the magic is .csf (and could be skipped entirely). Program-header and section-header reads are unchanged from AOSP, since the custom file’s offsets still point at the original tables.
The one genuinely tricky spot is FindPhdr(). When a PT_PHDR segment exists, its p_vaddr/p_offset describe the original program header table location, so si_->phdr = elfreader->loaded_phdr() ends up pointing at the wrong memory. My fix is to detect the custom format and set loaded_phdr_ directly:
bool ElfReader::FindPhdr() {
const ElfW(Phdr)* phdr_limit = phdr_table_ + phdr_num_;
if (memcmp(header_.e_ident, reinterpret_cast<const void *>(DIY_ELFMAG), SELFMAG) == 0) {
loaded_phdr_ = phdr_table_;
return true;
}
// If there is a PT_PHDR, use it directly.
for (const ElfW(Phdr)* phdr = phdr_table_; phdr < phdr_limit; ++phdr) {
if (phdr->p_type == PT_PHDR) {
return CheckPhdr(load_bias_ + phdr->p_vaddr);
}
}
// Otherwise, check the first loadable segment.
for (const ElfW(Phdr)* phdr = phdr_table_; phdr < phdr_limit; ++phdr) {
if (phdr->p_type == PT_LOAD) {
if (phdr->p_offset == 0) {
ElfW(Addr) elf_addr = load_bias_ + phdr->p_vaddr;
const ElfW(Ehdr)* ehdr = reinterpret_cast<const ElfW(Ehdr)*>(elf_addr);
ElfW(Addr) offset = ehdr->e_phoff;
return CheckPhdr(reinterpret_cast<ElfW(Addr)>(ehdr) + offset);
}
break;
}
}
DL_ERR("can't find loaded phdr for \"%s\"", name_.c_str());
return false;
}3.4 The shell-and-swap trick
The last step reuses the system’s soinfo machinery. We dlopen an empty shell so, resolve its soinfo, then overwrite its fields with our custom ELF’s data — a “borrow the shell, swap the body” approach. The catch is that soinfo_from_handle is not exported, so we have to implement it.
It only needs one unknown global, g_soinfo_handles_map, so we parse the linker64 ELF to resolve that symbol’s offset (this makes it robust across Android versions):
uintptr_t get_soinfo_handles_map_offset(uint64_t start, uint64_t end) {
int fd = open("/system/bin/linker64", O_RDONLY);
if (fd < 0) {
LOGI("Failed to open linker64: %s", strerror(errno));
return 0;
}
struct stat sb;
fstat(fd, &sb);
void* file_start = mmap(NULL, sb.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
if (file_start == MAP_FAILED) {
LOGI("mmap failed: %s", strerror(errno));
close(fd);
return 0;
}
Elf64_Ehdr header;
memcpy(&header, file_start, sizeof(Elf64_Ehdr));
int secoff = header.e_shoff;
int secsize = header.e_shentsize;
int secnum = header.e_shnum;
int secstr = header.e_shstrndx;
Elf64_Shdr strtab;
memcpy(&strtab, (char*)file_start + secoff + secstr * secsize, sizeof(Elf64_Shdr));
int strtaboff = strtab.sh_offset;
char* strtabchar = new char[strtab.sh_size];
memcpy(strtabchar, (char*)file_start + strtaboff, strtab.sh_size);
Elf64_Shdr enumsec;
int symtab_off = 0, symtab_size = 0;
int strtab_off = 0, strtab_size = 0;
for (int n = 0; n < secnum; n++) {
memcpy(&enumsec, (char*)file_start + secoff + n * secsize, sizeof(Elf64_Shdr));
if (strcmp(&strtabchar[enumsec.sh_name], ".symtab") == 0) {
symtab_off = enumsec.sh_offset;
symtab_size = enumsec.sh_size;
LOGI("Found .symtab: offset=0x%x, size=%d", symtab_off, symtab_size);
}
if (strcmp(&strtabchar[enumsec.sh_name], ".strtab") == 0) {
strtab_off = enumsec.sh_offset;
strtab_size = enumsec.sh_size;
LOGI("Found .strtab: offset=0x%x, size=%d", strtab_off, strtab_size);
}
}
delete[] strtabchar;
if (symtab_off == 0 || strtab_off == 0) {
LOGI("Required sections not found");
munmap(file_start, sb.st_size);
close(fd);
return 0;
}
uintptr_t symbol_offset = 0;
char* symbol_strtab = new char[strtab_size];
memcpy(symbol_strtab, (char*)file_start + strtab_off, strtab_size);
for (int n = 0; n < symtab_size; n += sizeof(Elf64_Sym)) {
Elf64_Sym sym;
memcpy(&sym, (char*)file_start + symtab_off + n, sizeof(Elf64_Sym));
if (sym.st_name != 0) {
const char* name = symbol_strtab + sym.st_name;
if (strstr(name, "soinfo_handles_map")) {
symbol_offset = sym.st_value;
LOGI("Found symbol %s: offset=0x%lx", name, symbol_offset);
break;
}
}
}
delete[] symbol_strtab;
munmap(file_start, sb.st_size);
close(fd);
return symbol_offset;
}Then resolve the shell’s soinfo and fill it with our data:
FILE *fp = fopen("/proc/self/maps", "r");
if (!fp) {
LOGI("Failed to open /proc/self/maps");
close(fd);
return;
}
parse_maps_for_lib("linker64", &start, &end);
LOGI("linker64 start = 0x%lx end = 0x%lx", start, end);
uintptr_t sym_offset = get_soinfo_handles_map_offset(start, end);
if (sym_offset == 0) {
LOGE("Failed to get offset");
return;
} else {
LOGI("soinfo_handles_map offset = 0x%lx", sym_offset);
}
soinfo_handles_map = reinterpret_cast<std::unordered_map<uintptr_t, soinfo*>*>((char*)start + sym_offset);
if (soinfo_handles_map == nullptr) {
LOGI("failed to get soinfo_handles_map");
return;
}
void *handle = dlopen(path_fake, RTLD_LAZY); // open the shell so, get its soinfo
if (handle == NULL) {
LOGI("%s dlopen failed: %s\n", path_fake, dlerror());
fclose(fp);
close(fd);
return;
}
si_ = soinfo_from_handle(soinfo_handles_map, static_cast<std::unordered_map<uintptr_t, soinfo*>*>(handle));
LOGI("si_ %p\n", si_);
// backfill the soinfo with our custom ELF's data
si_->base = elfreader->load_start();
si_->size = elfreader->load_size();
si_->load_bias = elfreader->load_bias();
si_->phnum = elfreader->phdr_count();
si_->phdr = elfreader->loaded_phdr();The pre-link and link steps after this are the same as the original flow. Here’s the loader’s log output:
3.5 Verifying in the soinfo list
We use libjoke.so as the shell, so we look for it in the soinfo list via Frida. The upper half shows the soinfo-list traversal; the lower half is the first 64 bytes of the file — exactly the RC4-decrypted data from section 3.1:
Summary
These three techniques compose — they aren’t mutually exclusive. Once you’ve gone to the trouble of a custom linker, the protected so can also live somewhere less obvious. I left call_constructors unimplemented; that’s a natural extension if you want to try it.
The full code isn’t pasted here in its entirety — it’s littered with debug output from the development process, and it’s messy. The interesting parts are the three techniques above.
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch