Colin XiaoReverse Engineering & Security Research

ART Internals and How They Power Android Security

About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

ART Internals and How They Power Android Security

Why this matters

Most Android security problems trace back to the same place: the Java virtual machine that runs every app. Packed apps need unpacking. Hook frameworks like Frida and Xposed need to hijack method execution. Detection tools need to see how those hooks work — and whether you could build your own. All of those answers live inside Android’s VM implementation.

Android has shipped two VMs: the early Dalvik, and the current ART (Android Runtime). This writeup focuses on ART, but I’ll spend a few paragraphs on Dalvik for the history. Everything is based on AOSP android-12.0.0_r11.

The Java VM

Java is a cross-platform language: the system actually runs Java bytecode, interpreted by a VM. If you’ve ever read a walkthrough of how a Python VM works, you already know the shape of an interpreter — a loop that decodes each instruction and dispatches it through a giant switch-case by instruction name. A JVM is the same idea, but heavily optimized, most notably with JIT: bytecode compiled to native platform code at runtime to speed up later runs.

Android apps are written in Java/Kotlin, so there’s a bytecode-interpreting VM at runtime. But Android doesn’t use a standard JVM — it compiles Java to Dalvik bytecode and runs it on an in-house VM. Partly licensing, partly genuine fit for mobile. The instruction sets differ in a fundamental way:

  • Dalvik instructions are register-based, 1–2 bytes each.
  • JVM instructions are stack-based, 1 byte each.

Dalvik trades memory for speed — more operands encoded per instruction, fewer instructions overall — which is why it can outrun the Oracle JVM on the same workload.

Startup

Interpreting Java isn’t slow; starting a Java VM is. If every app had to boot and initialize a VM from scratch, launch latency would be unacceptable. The fix is zygote: app processes are forked from the zygote process, so a child inherits its parent’s address space. Read-only pages are shared directly; writable data is mapped lazily through copy-on-write. Check /proc/self/maps for zygote and any of its children and you’ll see most system-library mappings are identical — that’s the whole trick.

Zygote itself is a native program. At the end of its main:

int main(int argc, char* const argv[]) {
    // ...
    if (zygote) {
        runtime.start("com.android.internal.os.ZygoteInit", args, zygote);
    } else if (className) {
        runtime.start("com.android.internal.os.RuntimeInit", args, zygote);
    } else {
        fprintf(stderr, "Error: no class name or --zygote supplied.\n");
        // ....
    }
}

That’s in frameworks/base/cmds/app_process/app_main.cpp. runtime.start starts the VM and hands control to the corresponding Java function:

void AndroidRuntime::start(const char* className, const Vector<String8>& options, bool zygote)
{
    /* start the virtual machine */
    JniInvocation jni_invocation;
    jni_invocation.Init(NULL);
    JNIEnv* env;
    if (startVm(&mJavaVM, &env, zygote) != 0) {
        return;
    }
    onVmCreated(env);
    /*
     * Register android functions.
     */
    if (startReg(env) < 0) {
        ALOGE("Unable to register all android natives\n");
        return;
    }

     /*
     * Start VM.  This thread becomes the main thread of the VM, and will
     * not return until the VM exits.
     */
    jclass startClass = env->FindClass(slashClassName);
        jmethodID startMeth = env->GetStaticMethodID(startClass, "main", "([Ljava/lang/String;)V");
    env->CallStaticVoidMethod(startClass, startMeth, strArray);
}

The key point: the VM is created once, in zygote, and inherited by every forked child. An app doesn’t re-boot a VM on launch — it reuses the inherited one and does lightweight init.

The interface

“Java VM” here is an umbrella term for Dalvik and ART. For applications they’re interchangeable, so both expose the same interface. That interface has two halves:

  1. The app-facing interface — the JavaVM / JNIEnv structs with FindClass, GetMethodID, CallVoidMethod, and friends.
  2. The system-facing interface — what the system uses to initialize and create the VM in the first place.

JniInvocation.Init sets up that second half. It reads the system property persist.sys.dalvik.vm.lib.2 to pick the VM library — libdvm.so for Dalvik, libart.so for ART — then dlopens it and dlsyms three symbols that abstract the whole VM:

  • JNI_GetDefaultJavaVMInitArgs — default JVM init args.
  • JNI_CreateJavaVM — create the VM.
  • JNI_GetCreatedJavaVMs — get the already-created VM instance(s).

AndroidRuntime::startVm ultimately calls JNI_CreateJavaVM to create the VM. These three functions are the abstraction seam that let ART replace Dalvik silently — and will let some future VM replace ART the same way.

Two consequences worth remembering: the VM is created exactly once, in zygote, and every Java process is a fork of zygote. Because each child owns an independent VM copy, one process’s VM crash doesn’t take down the others — that’s your runtime isolation.

Dalvik

Dalvik was the original Android VM, and its retirement was made official with Android 5.0:

commit 870b4f2d70d67d6dbb7d0881d101c61bed8caad2
Author: Brian Carlstrom <bdc@google.com>
Date:   Tue Aug 5 12:46:17 2014 -0700

    Dalvik is dead, long live Dalvik!

ART replaced it, but Dalvik’s simplicity is still the best way to understand how Java code runs. When zygote creates the VM, the first Java function it calls is com.android.internal.os.ZygoteInit.main, a static method — so natively it goes through the JNI call CallStaticVoidMethod. The chain:

method file
JNIEnv.CallStaticVoidMethod dalvik/libnativehelper/include/nativehelper/jni.h
JNINativeInterface.CallStaticVoidMethodV dalvik/vm/Jni.c
Jni.dvmCallMethodV dalvik/vm/interp/Stack.c
Stack.dvmInterpret dalvik/vm/interp/Interp.c
dvmInterpretStd dalvik/vm/mterp/out/InterpC-portstd.c (generated)

Dalvik supports three execution modes:

  • kExecutionModeInterpPortable — portable; runs anywhere; entry dvmInterpretStd.
  • kExecutionModeInterpFast — fast; optimized per-platform; entry dvmMterpStd.
  • kExecutionModeJit — JIT; compiles to native at runtime; entry also dvmMterpStd.

The portable-mode interpreter, in its essentials:

#define INTERP_FUNC_NAME dvmInterpretStd
bool INTERP_FUNC_NAME(Thread* self, InterpState* interpState) {
    // ...

    /* core state */
    const Method* curMethod;    // method we're interpreting
    const u2* pc;               // program counter
    u4* fp;                     // frame pointer
    u2 inst;                    // current instruction

    /* copy state in */
    curMethod = interpState->method;
    pc = interpState->pc;
    fp = interpState->fp;
    retval = interpState->retval;   /* only need for kInterpEntryReturn? */

    methodClassDex = curMethod->clazz->pDvmDex;

    while (1) {
        /* fetch the next 16 bits from the instruction stream */
        inst = FETCH(0);
        switch (INST_INST(inst)) {
            HANDLE_OPCODE(OP_INVOKE_DIRECT /*vB, {vD, vE, vF, vG, vA}, meth@CCCC*/)
                GOTO_invoke(invokeDirect, false);
            OP_END
            HANDLE_OPCODE(OP_RETURN /*vAA*/)
            HANDLE_OPCODE(...)
        }
    }

    /* export state changes */
    interpState->method = curMethod;
    interpState->pc = pc;
    interpState->fp = fp;

    /* debugTrackedRefStart doesn't change */
    interpState->retval = retval;   /* need for _entryPoint=ret */
    interpState->nextMode =
        (INTERP_TYPE == INTERP_STD) ? INTERP_DBG : INTERP_STD;
    return true;
}

There it is: a big switch/case that walks from the program counter, reads 4-byte-aligned bytecode, and dispatches by opcode until the method returns or throws. “Portable” because it’s pure interpretation — no up-front optimization, no JIT — so it runs anywhere a C compiler targets.

Even though Dalvik is gone, its bytecode format survives. Today’s Android still compiles Java to DEX files. The format is documented in the Dalvik Executable format spec, and the instructions in Dalvik bytecode.

ART

ART — Android Runtime — is the high-performance successor. My focus here is how ART executes Java code. First we need the ahead-of-time (AOT) pipeline and the intermediate files it produces.

Ahead-of-time compilation

When you build an app in Android Studio, the Java compiler turns .java into Java bytecode (.class), then dx (now d8) converts that to Dalvik bytecode and packs all classes into one DEX, which is zipped with resources into the .apk.

At install time, PackageManager unpacks the APK and calls out to a helper binary to optimize the bytecode. Dalvik used dexopt; ART uses dex2oat.

  • dexopt produces an odex (optimized-dex) — optimized Dalvik bytecode, “quickened” dex.
  • dex2oat is LLVM-based and produces native code for the target platform, stored as oat (Ahead-Of-Time). An oat file is actually an ELF file, with the original DEX content embedded in an oatdata section.

From Android 8 onward the OAT file was split in two: the OAT stays ELF, but the original DEX content moved into a VDEX file with its own format. The overall flow:

Java code processing flow

dex2oat writes optimized code under the app’s /data/app path; system apps land in /data/dalvik-cache/. For a system app you get three files:

$ ls -l | grep Settings.apk
-rw-r----- 1 system system           77824 2021-12-10 10:33 system_ext@priv-app@Settings@Settings.apk@classes.art
-rw-r----- 1 system system          192280 2021-11-19 12:50 system_ext@priv-app@Settings@Settings.apk@classes.dex
-rw-r----- 1 system system           59646 2021-12-10 10:33 system_ext@priv-app@Settings@Settings.apk@classes.vdex

Don’t judge a file by its extension: classes.dex here is actually an ELF-format OAT file. .art is its own thing too. ART is implemented in C++, and Java primitives have C++ counterparts:

  • java.lang.Class ↔ art::mirror::Class
  • java.lang.String ↔ art::mirror::String
  • java.lang.reflect.Method ↔ art::mirror::Method
  • …

Creating a Java object creates the corresponding C++ object and calls its constructor; the VM manages references to those C++ objects. To skip re-initializing these common classes on every boot, Android dumps them to an .art file — essentially a memory dump of a set of common C++ objects.

OAT, VDEX, and ART are all internal Android formats with no compatibility guarantee. They shift between versions in ways that are only visible in code, so you don’t need to memorize their layouts — just know what each is for.

File loading

In a prior article I traced app launch up to ActivityThread, where Application is created and Activity.onCreate fires. I skipped one question: how does the app’s code (DEX/OAT) actually get loaded into the process?

In Java the class loader is ClassLoader; Android’s BaseDexClassLoader extends it and implements the DEX loading. Its subclasses:

  • DexClassLoader — loads classes from a .jar or .apk.
  • PathClassLoader — initializes a class loader from local files.
  • InMemoryDexClassLoader — initializes a class loader from memory.

ClassLoader

Taking PathClassLoader as the example, its constructor chains to the parent’s constructor:

method file
new PathClassLoader …
new BaseDexClassLoader libcore/dalvik/src/main/java/dalvik/system/BaseDexClassLoader.java
new DexPathList libcore/dalvik/src/main/java/dalvik/system/DexPathList.java
DexPathList.makeDexElements …
DexPathList.loadDexFile …
new DexFile libcore/dalvik/src/main/java/dalvik/system/DexFile.java
DexFile.openDexFile …
DexFile.openDexFileNative …
DexFile_openDexFileNative art/runtime/native/dalvik_system_DexFile.cc
OatFileManager::OpenDexFilesFromOat art/runtime/oat_file_manager.cc

The real loading happens in OpenDexFilesFromOat:

std::vector<std::unique_ptr<const DexFile>> OatFileManager::OpenDexFilesFromOat() {
    std::vector<std::unique_ptr<const DexFile>> dex_files = OpenDexFilesFromOat_Impl(...);
    for (std::unique_ptr<const DexFile>& dex_file : dex_files) {
      if (!dex_file->DisableWrite()) {
        error_msgs->push_back("Failed to make dex file " + dex_file->GetLocation() + " read-only");
      }
    }
    return dex_files;
}

It loads a DexFile array, then marks the in-memory dex read-only — not yet enforced, but clearly where this is heading. The implementation:

std::vector<std::unique_ptr<const DexFile>> OatFileManager::OpenDexFilesFromOat_Impl() {
    // Extract dex file headers from `dex_mem_maps`.
    const std::vector<const DexFile::Header*> dex_headers = GetDexFileHeaders(dex_mem_maps);

    // Determine dex/vdex locations and the combined location checksum.
    std::string dex_location;
    std::string vdex_path;
    bool has_vdex = OatFileAssistant::AnonymousDexVdexLocation(dex_headers,
                                                             kRuntimeISA,
                                                             &dex_location,
                                                             &vdex_path);

    if (has_vdex && OS::FileExists(vdex_path.c_str())) {
        vdex_file = VdexFile::Open(vdex_path,
                                /* writable= */ false,
                                /* low_4gb= */ false,
                                /* unquicken= */ false,
                                &error_msg);
    }

    // Load dex files. Skip structural dex file verification if vdex was found
    // and dex checksums matched.
    std::vector<std::unique_ptr<const DexFile>> dex_files;
    for (size_t i = 0; i < dex_mem_maps.size(); ++i) {
        static constexpr bool kVerifyChecksum = true;
        const ArtDexFileLoader dex_file_loader;
        std::unique_ptr<const DexFile> dex_file(dex_file_loader.Open(
            DexFileLoader::GetMultiDexLocation(i, dex_location.c_str()),
            dex_headers[i]->checksum_,
            std::move(dex_mem_maps[i]),
            /* verify= */ (vdex_file == nullptr) && Runtime::Current()->IsVerificationEnabled(),
            kVerifyChecksum,
            &error_msg));
        if (dex_file != nullptr) {
            dex::tracking::RegisterDexFile(dex_file.get());  // Register for tracking.
            dex_files.push_back(std::move(dex_file));
        }
    }

    // Initialize an OatFile instance backed by the loaded vdex.
    std::unique_ptr<OatFile> oat_file(OatFile::OpenFromVdex(
        MakeNonOwningPointerVector(dex_files),
        std::move(vdex_file),
        dex_location));
    if (oat_file != nullptr) {
        VLOG(class_linker) << "Registering " << oat_file->GetLocation();
        *out_oat_file = RegisterOatFile(std::move(oat_file));
    }
    return dex_files;
}

Sequence: map the VDEX into memory, convert the in-memory (or on-disk) dex into DexFile structs, then associate the VDEX with the OAT file.

VdexFile

VDEX arrived in Android 8.0. It holds the original DEX info for the optimized code, while OAT holds the dex2oat-compiled native code. Rough layout (where D is the number of DEX files inside):

VdexFileHeader    fixed-length header
VdexSectionHeader[kNumberOfSections]

Checksum section
  VdexChecksum[D]

Optionally:
   DexSection
       DEX[0]                array of the input DEX files
       DEX[1]
       ...
       DEX[D-1]

VerifierDeps
   4-byte alignment
   uint32[D]                  DexFileDeps offsets for each dex file
   DexFileDeps[D][]           verification dependencies
     4-byte alignment
     uint32[class_def_size]     TypeAssignability offsets (kNotVerifiedMarker for a class that isn't verified)
     uint32                     Offset of end of AssignabilityType sets
     uint8[]                    AssignabilityType sets
     4-byte alignment
     uint32                     Number of strings
     uint32[]                   String data offsets for each string
     uint8[]                    String data

Full VdexFile structure: art/runtime/vdex_file.h

DexFile

dex_file_loader.Open resolves like this:

  • ArtDexFileLoader::Open
  • ArtDexFileLoader::OpenCommon
  • DexFileLoader::OpenCommon
    • magic == "dex\n" → new StandardDexFile()
    • magic == "cdex" → new CompactDexFile()

The first 4 bytes decide standard DEX vs. compact DEX (cdex — the internal format ART uses to shrink disk and memory). Both StandardDexFile and CompactDexFile derive from DexFile, and both constructors ultimately call the DexFile constructor.

Full DexFile structure: art/libdexfile/dex/dex_file.h

OatFile

After all DexFiles are initialized, OatFile::OpenFromVdex creates and registers the OAT file:

  • OatFile::OpenFromVdex
  • OatFileBackedByVdex::Open
  • new OatFileBackedByVdex
  • OatFileBase::OatFileBase
  • OatFile::OatFile

Unlike the old odex days, the OAT now takes ownership of the VDEX via oat_file->SetVdex, using it to implement part of the OAT interface — for example, getting the in-memory start address of the DEX:

const uint8_t* OatFile::DexBegin() const {
  return vdex_->Begin();
}

See: art/runtime/oat_file.h

Method invocation

I’m going out of timeline order here and starting with how a Java method call runs, because it’s what matters in practice; the runtime classes it touches get explained next.

On the web-server side, Java apps often run a RASP (Runtime Application Self-Protection) layer that monitors sensitive calls and alerts — hooking inside the JVM so both direct and reflective calls are caught. Android has the same kind of chain. The reflective case is the clearest to watch:

import java.lang.reflect.*;
public class Test {
    public static void main(String args[]) throws Exception {
        Class c = Class.forName("com.evilpan.DemoClass");
        Method m = c.getMethod("foo", null);
        m.invoke();
    }
}

Method.invoke is a native method, implemented in art/runtime/native/java_lang_reflect_Method.cc:

static jobject Method_invoke(JNIEnv* env, jobject javaMethod, jobject javaReceiver,
                             jobjectArray javaArgs) {
  ScopedFastNativeObjectAccess soa(env);
  return InvokeMethod<kRuntimePointerSize>(soa, javaMethod, javaReceiver, javaArgs);
}

InvokeMethod lives in art/runtime/reflection.cc:

template <PointerSize kPointerSize>
jobject InvokeMethod(const ScopedObjectAccessAlreadyRunnable& soa, jobject javaMethod,
                     jobject javaReceiver, jobject javaArgs, size_t num_frames) {
    ObjPtr<mirror::Executable> executable = soa.Decode<mirror::Executable>(javaMethod);
    const bool accessible = executable->IsAccessible();
    ArtMethod* m = executable->GetArtMethod();

    if (UNLIKELY(!declaring_class->IsVisiblyInitialized())) {
        Thread* self = soa.Self();
        Runtime::Current()->GetClassLinker()->EnsureInitialized(
            self, h_class,
            /*can_init_fields=*/ true,
            /*can_init_parents=*/ true)
    }

    if (!m->IsStatic()) {
        if (declaring_class->IsStringClass() && m->IsConstructor()) {
            m = WellKnownClasses::StringInitToStringFactory(m);
        } else {
            m = receiver->GetClass()->FindVirtualMethodForVirtualOrInterface(m, kPointerSize);
        }
    }

    if (!accessible && !VerifyAccess(/*...*/)) {
        ThrowIllegalAccessException(
        StringPrintf("Class %s cannot access %s method %s of class %s", ...));
    }

    InvokeMethodImpl(soa, m, np_method, receiver, objects, &shorty, &result);
}

Stripped of detail, the pre-call work is:

  1. If the method’s declaring class isn’t initialized, initialize it.
  2. Replace a String.<init> constructor call with its StringFactory factory.
  3. For a virtual call, resolve to the actual runtime method.
  4. Check access; throw on violation.
  5. Invoke.

Notably, a jobject method can be cast to an ArtMethod* — the struct that describes a specific method in ART. Then:

  • InvokeMethodImpl
  • InvokeWithArgArray
  • method->Invoke()

Down into ArtMethod::Invoke:

void ArtMethod::Invoke(Thread* self, uint32_t* args, uint32_t args_size, JValue* result,
                       const char* shorty) {
    Runtime* runtime = Runtime::Current();
    if (UNLIKELY(!runtime->IsStarted() ||
               (self->IsForceInterpreter() && !IsNative() && !IsProxyMethod() && IsInvokable()))) {
        art::interpreter::EnterInterpreterFromInvoke(...);
    } else {
        bool have_quick_code = GetEntryPointFromQuickCompiledCode() != nullptr;
        if (LIKELY(have_quick_code)) {
            if (!IsStatic()) {
                (*art_quick_invoke_stub)(this, args, args_size, self, result, shorty);
            } else {
                (*art_quick_invoke_static_stub)(this, args, args_size, self, result, shorty);
            }
        } else {
            LOG(INFO) << "Not invoking '" << PrettyMethod() << "' code=null";
        }
    }
    self->PopManagedStackFragment(fragment);
}

ART has two ways to run a Java method: interpret the bytecode (like Dalvik), or fast-mode — jump straight into the native code dex2oat produced.

Early ART split native code into Portable and Quick modes; Quick kept improving while Portable fell behind and was eventually dropped.

From the code: when the runtime isn’t started yet, or interpreter mode is forced, the VM interprets. You can force interpreter mode at startup with -Xint. But even then, some methods can’t be interpreted:

  1. Native methods — there’s only binary code, no bytecode.
  2. Uninvokable methods — access-flag violations or abstract methods.
  3. Proxy methods — ART has a separate native path for them.

Interpretation

The interpreter entry is art::interpreter::EnterInterpreterFromInvoke in art/runtime/interpreter/interpreter.cc:

void EnterInterpreterFromInvoke(Thread* self,
                                ArtMethod* method,
                                ObjPtr<mirror::Object> receiver,
                                uint32_t* args,
                                JValue* result,
                                bool stay_in_interpreter) {
    CodeItemDataAccessor accessor(method->DexInstructionData());
    if (accessor.HasCodeItem()) {
        num_regs =  accessor.RegistersSize();
        num_ins = accessor.InsSize();
    }
    // Initialize the stack frame ......
    if (LIKELY(!method->IsNative())) {
        JValue r = Execute(self, accessor, *shadow_frame, JValue(), stay_in_interpreter);
        if (result != nullptr) {
        *result = r;
        }
  }
}

The CodeItem is the method’s bytecode inside the DEX. The call chain:

method file
Execute art/runtime/interpreter/interpreter.cc
ExecuteSwitch …
ExecuteSwitchImpl art/runtime/interpreter/interpreter_switch_impl.h
ExecuteSwitchImplAsm …
ExecuteSwitchImplAsm art/runtime/arch/arm64/quick_entrypoints_arm64.S
ExecuteSwitchImplCpp art/runtime/interpreter/interpreter_switch_impl-inl.h

ExecuteSwitchImplAsm is raw assembly for speed. On ARM64:

//  Wrap ExecuteSwitchImpl in assembly method which specifies DEX PC for unwinding.
//  Argument 0: x0: The context pointer for ExecuteSwitchImpl.
//  Argument 1: x1: Pointer to the templated ExecuteSwitchImpl to call.
//  Argument 2: x2: The value of DEX PC (memory address of the methods bytecode).
ENTRY ExecuteSwitchImplAsm
    SAVE_TWO_REGS_INCREASE_FRAME x19, xLR, 16
    mov x19, x2                                   // x19 = DEX PC
    CFI_DEFINE_DEX_PC_WITH_OFFSET(0 /* x0 */, 19 /* x19 */, 0)
    blr x1                                        // Call the wrapped method.
    RESTORE_TWO_REGS_DECREASE_FRAME x19, xLR, 16
    ret
END ExecuteSwitchImplAsm

It’s really just calling the second argument held in x1. The call site:

template<bool do_access_check, bool transaction_active>
ALWAYS_INLINE JValue ExecuteSwitchImpl() {
    //...
    void* impl = reinterpret_cast<void*>(&ExecuteSwitchImplCpp<do_access_check, transaction_active>);
    const uint16_t* dex_pc = ctx.accessor.Insns();
    ExecuteSwitchImplAsm(&ctx, impl, dex_pc);
}

Which lands in ExecuteSwitchImplCpp — and here’s the classic interpreter loop:

template<bool do_access_check, bool transaction_active>
void ExecuteSwitchImplCpp(SwitchImplContext* ctx) {
    Thread* self = ctx->self;
    const CodeItemDataAccessor& accessor = ctx->accessor;
    ShadowFrame& shadow_frame = ctx->shadow_frame;
    self->VerifyStack();

    uint32_t dex_pc = shadow_frame.GetDexPC();
    const auto* const instrumentation = Runtime::Current()->GetInstrumentation();
    const uint16_t* const insns = accessor.Insns();
    const Instruction* next = Instruction::At(insns + dex_pc);

    while (true) {
        const Instruction* const inst = next;
        dex_pc = inst->GetDexPc(insns);
        shadow_frame.SetDexPC(dex_pc);
        TraceExecution(shadow_frame, inst, dex_pc);
        uint16_t inst_data = inst->Fetch16(0); // one instruction is 4 bytes

        if (InstructionHandler(...).Preamble()) {
            switch (inst->Opcode(inst_data)) {
                case xxx: ...;
                case yyy: ...;
                ...
            }
        }
    }
}

In Android 12 the opcode handlers are macro-expanded. Details vary by version, but the core loop hasn’t changed since Android 2.x — the bytecode definition barely has.

Fast execution

Back up to ArtMethod::Invoke: the non-interpreter path calls through art_quick_invoke_stub. A “stub” is a small piece of trampoline code that jumps to the real native body, written in assembly. ARM64 version in art/runtime/arch/arm64/quick_entrypoints_arm64.S:

.macro INVOKE_STUB_CALL_AND_RETURN
    REFRESH_MARKING_REGISTER
    REFRESH_SUSPEND_CHECK_REGISTER

    // load method-> METHOD_QUICK_CODE_OFFSET
    ldr x9, [x0, #ART_METHOD_QUICK_CODE_OFFSET_64]
    // Branch to method.
    blr x9
.endm

/*
 *  extern"C" void art_quick_invoke_stub(ArtMethod *method,   x0
 *                                       uint32_t  *args,     x1
 *                                       uint32_t argsize,    w2
 *                                       Thread *self,        x3
 *                                       JValue *result,      x4
 *                                       char   *shorty);     x5
 */
ENTRY art_quick_invoke_stub
    // ...
    INVOKE_STUB_CALL_AND_RETURN
END art_quick_invoke_static_stub

The save/restore boilerplate is elided. The core is a jump to the pointer at ArtMethod offset ART_METHOD_QUICK_CODE_OFFSET_64, which is:

ASM_DEFINE(ART_METHOD_QUICK_CODE_OFFSET_64,
           art::ArtMethod::EntryPointFromQuickCompiledCodeOffset(art::PointerSize::k64).Int32Value())

i.e. the address held in entry_point_from_quick_compiled_code_:

// art/runtime/art_method.h
static constexpr MemberOffset EntryPointFromQuickCompiledCodeOffset(PointerSize pointer_size) {
return MemberOffset(PtrSizedFieldsOffset(pointer_size) + OFFSETOF_MEMBER(
    PtrSizedFields, entry_point_from_quick_compiled_code_) / sizeof(void*)
        * static_cast<size_t>(pointer_size));
}

That pointer is the single entry point for all fast-mode execution. Where it points, and when it’s set, is the code-loading story in the next section.

One nuance: execution mode isn’t fixed for a whole run. Fast-mode code can call into a method that only interprets, and interpreted code can call JNI or fast-mode methods. Because each transition saves and restores context, the modes stay transparent to each other — a textbook modular design.

Code loading

ArtMethod::Invoke is how Java methods run. So when is the ArtMethod created, and why can a jmethod/jobject be cast to an ArtMethod*?

In Java, methods live on classes, so you analyze method init by analyzing class init. We know how a ClassLoader is built from OAT/VDEX/DEX, but at that point classes aren’t initialized yet. A quick demonstration:

public class Demo {
    static {
        Log.i("Demo", "static block called");
    }
    {
        Log.i("Demo", "IIB called");
    }
}

If Demo is never used, neither log fires. But Class.forName("Demo") triggers the static block. Tracing Class.forName:

@CallerSensitive
public static Class<?> forName(String className)
            throws ClassNotFoundException {
    Class<?> caller = Reflection.getCallerClass();
    // author's note: initialize = true
    return forName(className, true, ClassLoader.getClassLoader(caller));
}

It bottoms out in the native classForName in art/runtime/native/java_lang_Class.cc:

// "name" is in "binary name" format, e.g. "dalvik.system.Debug$1".
static jclass Class_classForName(JNIEnv* env, jclass, jstring javaName, jboolean initialize,
                                 jobject javaLoader) {
    ScopedFastNativeObjectAccess soa(env);
    ScopedUtfChars name(env, javaName);

    std::string descriptor(DotToDescriptor(name.c_str()));
    Handle<mirror::ClassLoader> class_loader(
      hs.NewHandle(soa.Decode<mirror::ClassLoader>(javaLoader)));
    ClassLinker* class_linker = Runtime::Current()->GetClassLinker();
    Handle<mirror::Class> c(
      hs.NewHandle(class_linker->FindClass(soa.Self(), descriptor.c_str(), class_loader)));

    if (initialize) {
        class_linker->EnsureInitialized(soa.Self(), c, true, true);
    }
    return soa.AddLocalReference<jclass>(c.Get());
}

It converts the Java-format class name to descriptor form, looks it up via the class loader (driven by class_linker), and — because forName passes initialize = true — runs EnsureInitialized after finding it.

FindClass

FindClass resolves a class by name, in art/runtime/class_linker.cc:

ObjPtr<mirror::Class> ClassLinker::FindClass(Thread* self,
                                             const char* descriptor,
                                             Handle<mirror::ClassLoader> class_loader)
    if (descriptor[1] == '\0')
        return FindPrimitiveClass(descriptor[0]);

    const size_t hash = ComputeModifiedUtf8Hash(descriptor);
    // Look up in already-loaded classes
    ObjPtr<mirror::Class> klass = LookupClass(self, descriptor, hash, class_loader.Get());
    if (klass != nullptr) {
        return EnsureResolved(self, descriptor, klass);
    }
    // Not loaded yet
    if (descriptor[0] != '[' && class_loader == nullptr) {
        // No class loader and not an array type; look it up in the boot classes
        ClassPathEntry pair = FindInClassPath(descriptor, hash, boot_class_path_);
        return DefineClass(self, descriptor, hash,
                           ScopedNullHandle<mirror::ClassLoader>(),
                           *pair.first, *pair.second);
    }

    ObjPtr<mirror::Class> result_ptr;
    bool descriptor_equals;
    ScopedObjectAccessUnchecked soa(self);
    // First search via the classLoader's parent
    bool known_hierarchy =
        FindClassInBaseDexClassLoader(soa, self, descriptor, hash, class_loader, &result_ptr);
    if (result_ptr != nullptr) {
        descriptor_equals = true;
    } else if (!self->IsExceptionPending()) {
        // If not found, search via the classLoader itself
        std::string class_name_string(descriptor + 1, descriptor_length - 2);
        std::replace(class_name_string.begin(), class_name_string.end(), '/', '.');
        ScopedLocalRef<jobject> class_loader_object(
            soa.Env(), soa.AddLocalReference<jobject>(class_loader.Get()));
        ScopedLocalRef<jobject> result(soa.Env(), nullptr);
        result.reset(soa.Env()->CallObjectMethod(class_loader_object.get(),
                                                 WellKnownClasses::java_lang_ClassLoader_loadClass,
                                                 class_name_object.get()));
    }

    // Insert the found class into the cache table
    ClassTable* const class_table = InsertClassTableForClassLoader(class_loader.Get());
    class_table->InsertWithHash(result_ptr, hash);

    return result_ptr;
}

First LookupClass checks already-loaded classes, stored in a ClassTable keyed by a hash of the descriptor. A hit returns immediately; a miss does the real load. That’s lazy loading — a never-used class costs nothing.

A null class loader means a system class, which doesn’t live in the app’s DEX but in the system image. Every app needs these, so zygote loads them and all children share. boot_class_path_ is populated in Runtime::Init from ART’s startup args.

App-class lookup is two-stage: search the parent class loader first, and only fall back to the specified loader — the “parent delegation” mechanism every Java tutorial mentions. Routing critical classes through the system loader first is exactly what stops an app from swapping out their implementations.

FindClassInBaseDexClassLoader in pseudocode:

Class ClassLinker::FindClassInBaseDexClassLoader(ClassLoader class_loader, size_t hash) {
    if (class_loader == java_lang_BootClassLoader) {
        return FindClassInBootClassLoaderClassPath(class_loader, hash);
    }
    if (class_loader == dalvik_system_PathClassLoader ||
        class_loader == dalvik_system_DexClassLoader ||
        class_loader == dalvik_system_InMemoryDexClassLoader) {
        // For regular path or dex class loader the search order is:
        //    - parent
        //    - shared libraries
        //    - class loader dex files
        FindClassInBaseDexClassLoader(class_loader->GetParent, hash) && return result;
        FindClassInSharedLibraries(...) && return result;
        FindClassInBaseDexClassLoaderClassPath(...) && return result;
        FindClassInSharedLibrariesAfter(...) && return result;
    }
    if (class_loader == dalvik_system_DelegateLastClassLoader) {
        // For delegate last, the search order is:
        //    - boot class path
        //    - shared libraries
        //    - class loader dex files
        //    - parent
        FindClassInBootClassLoaderClassPath(...) && return result;
        FindClassInBaseDexClassLoaderClassPath(...) && return result;
        FindClassInSharedLibrariesAfter(...) && return result;
        FindClassInBaseDexClassLoader(class_loader->GetParent, hash) && return result;
    }
    return null;
}

Different loader types get different search orders, recursing through parents until hitting java.lang.BootClassLoader (the base case).

FindClassInBootClassLoaderClassPath:

using ClassPathEntry = std::pair<const DexFile*, const dex::ClassDef*>;
bool ClassLinker::FindClassInBootClassLoaderClassPath(Thread* self,
                                                      const char* descriptor,
                                                      size_t hash,
                                                      /*out*/ ObjPtr<mirror::Class>* result) {
    ClassPathEntry pair = FindInClassPath(descriptor, hash, boot_class_path_);
    if (pair.second != nullptr) {
        ObjPtr<mirror::Class> klass = LookupClass(self, descriptor, hash, nullptr);
        if (klass != nullptr) {
            *result = EnsureResolved(self, descriptor, klass);
        } else {
            *result = DefineClass(self, ...);
        }
    }
    return true;

If the base loader misses, the search falls to the passed-in classLoader’s loadClass. For Class.forName that’s the caller’s loader. Printing it from a test APK:

dalvik.system.PathClassLoader[
DexPathList[[
zip file "/data/app/~~0FBqwacokhdG5rhF1RDZGg==/com.evilpan.test-fCJvsE74xP_SdvTlAfJDcA==/base.apk"
],
nativeLibraryDirectories=[/data/app/~~0FBqwacokhdG5rhF1RDZGg==/com.evilpan.test-fCJvsE74xP_SdvTlAfJDcA==/lib/arm64, /system/lib64, /system_ext/lib64]]]

A PathClassLoader — which doesn’t define loadClass, so the parent’s is used. The full path:

sequenceDiagram
%% loadClass
participant P as PathClassLoader
participant B as BaseDexClassLoader
participant C as ClassLoader
participant D as DexFile
P ->> C: loadClass
C ->> B: findClass
B ->> P: pathList.findClass
P ->> D: loadClassBinaryName
Note right of D: defineClass <br> defineClassNative

loadClass

Finally DexFile.defineClassNative, in art/runtime/native/dalvik_system_DexFile.cc:

static jclass DexFile_defineClassNative(JNIEnv* env,
                                        jclass,
                                        jstring javaName,
                                        jobject javaLoader,
                                        jobject cookie,
                                        jobject dexFile) {
    std::vector<const DexFile*> dex_files;
    ConvertJavaArrayToDexFiles(env, cookie, /*out*/ dex_files, /*out*/ oat_file);

    ScopedUtfChars class_name(env, javaName);
    const std::string descriptor(DotToDescriptor(class_name.c_str()));
    const size_t hash(ComputeModifiedUtf8Hash(descriptor.c_str()));
    for (auto& dex_file : dex_files) {
        const dex::ClassDef* dex_class_def = OatDexFile::FindClassDef(*dex_file, descriptor.c_str(), hash);
        // dex_class_def != nullptr
        ClassLinker* class_linker = Runtime::Current()->GetClassLinker();
        Handle<mirror::ClassLoader> class_loader(
          hs.NewHandle(soa.Decode<mirror::ClassLoader>(javaLoader)));
        ObjPtr<mirror::DexCache> dex_cache =
          class_linker->RegisterDexFile(*dex_file, class_loader.Get());
        // dex_cache != nullptr
        ObjPtr<mirror::Class> result = class_linker->DefineClass(soa.Self(),
                                                               descriptor.c_str(),
                                                               hash,
                                                               class_loader,
                                                               *dex_file,
                                                               *dex_class_def);
        class_linker->InsertDexFileInToClassLoader(soa.Decode<mirror::Object>(dexFile),
                                                 class_loader.Get());
    }
}

Whether via FindClassInBaseDexClassLoader or the loader’s loadClass, the end is the same: search the relevant DEX (OatDexFile) for the class’s ClassDef by name. If you know the DEX format you know that field — it holds the class definition. After locating it, ClassLinker finishes registration:

  • On first sight of a DEX file, create a DexCache and store it in the linker’s dex_caches_ hash table.
  • ClassLinker::DefineClass defines the class (next section).
  • Add the DEX to the loader’s ClassTable.

DefineClass

ObjPtr<mirror::Class> ClassLinker::DefineClass(Thread* self,
                                               const char* descriptor,
                                               size_t hash,
                                               Handle<mirror::ClassLoader> class_loader,
                                               const DexFile& dex_file,
                                               const dex::ClassDef& dex_class_def) {
    ScopedDefiningClass sdc(self);
    StackHandleScope<3> hs(self);
    auto klass = hs.NewHandle<mirror::Class>(nullptr);

    // Load the class from the dex file.
    if (UNLIKELY(!init_done_)) {
        // [1] finish up init of hand crafted class_roots_
    }

    ObjPtr<mirror::DexCache> dex_cache = RegisterDexFile(*new_dex_file, class_loader.Get());
    klass->SetDexCache(dex_cache);
    ObjPtr<mirror::Class> existing = InsertClass(descriptor, klass.Get(), hash);
    if (existing != nullptr) {
        // Another thread is linking this class; block and wait for it to finish
        return sdc.Finish(EnsureResolved(self, descriptor, existing));
    }
    LoadClass(self, *new_dex_file, *new_class_def, klass);
    // klass->IsLoaded
    LoadSuperAndInterfaces(klass, *new_dex_file))
    Runtime::Current()->GetRuntimeCallbacks()->ClassLoad(klass);
    // klass->IsResolved
    LinkClass(self, descriptor, klass, interfaces, &h_new_class)
    Runtime::Current()->GetRuntimeCallbacks()->ClassPrepare(klass, h_new_class);

    jit::Jit::NewTypeLoadedIfUsingJit(h_new_class.Get());
    return sdc.Finish(h_new_class);
}

init_done_ tracks the linker’s init state. Init either loads from the Image space or hand-crafts the internal classes:

  • Ljava/lang/Object;
  • Ljava/lang/Class;
  • Ljava/lang/String;
  • Ljava/lang/ref/Reference;
  • Ljava/lang/DexCache;
  • Ldalvik/system/ClassExt;

All defined directly in the art::runtime::mirror namespace (Object is mirror::Object, in art/runtime/mirror/object.h).

LoadClass

ClassLinker::LoadClass loads a class’s fields and methods from a DEX. Note it runs after the class is added to the ClassTable — an ART optimization, and a necessity since the class’s field roots are only reachable through the table:

void ClassLinker::LoadClass(Thread* self,
                            const DexFile& dex_file,
                            const dex::ClassDef& dex_class_def,
                            Handle<mirror::Class> klass) {
    ClassAccessor accessor(dex_file,
                         dex_class_def,
                         /* parse_hiddenapi_class_data= */ klass->IsBootStrapClassLoaded());
    Runtime* const runtime = Runtime::Current();
    accessor.VisitFieldsAndMethods(
        [&](const ClassAccessor::Field& field) {
            LoadField(field, klass, &sfields->At(num_sfields));
            ++num_sfields;
        },
        [&](const ClassAccessor::Field& field) {
            LoadField(field, klass, &ifields->At(num_ifields));
            ++num_ifields;
        },
        [&](const ClassAccessor::Method& method) {
            ArtMethod* art_method = klass->GetDirectMethodUnchecked(
                class_def_method_index,
                image_pointer_size_);
            LoadMethod(dex_file, method, klass, art_method);
            LinkCode(this, art_method, oat_class_ptr, class_def_method_index);
            ++class_def_method_index;
        },
        [&](const ClassAccessor::Method& method) {
            ArtMethod* art_method = klass->GetVirtualMethodUnchecked(
                class_def_method_index - accessor.NumDirectMethods(),
                image_pointer_size_);
            LoadMethod(dex_file, method, klass, art_method);
            LinkCode(this, art_method, oat_class_ptr, class_def_method_index);
            ++class_def_method_index;
        }
    );
    klass->SetSFieldsPtr(sfields);
    klass->SetIFieldsPtr(ifields);
}

C++11 lambdas walk the class’s members in four buckets:

  1. sfields — static fields.
  2. ifields — instance fields.
  3. direct methods — instance methods.
  4. virtual methods.

LoadField initializes an ArtField and ties it to the class; LoadMethod does the analogous thing for an ArtMethod, initializing it from the method’s CodeItem in the DEX. Methods get one extra step: LinkCode.

LinkCode

LinkCode links the code, as the name says:

static void LinkCode(ClassLinker* class_linker,
                     ArtMethod* method,
                     const OatFile::OatClass* oat_class,
                     uint32_t class_def_method_index) {
    Runtime* const runtime = Runtime::Current();
    const void* quick_code = nullptr;
    if (oat_class != nullptr) {
         // Every kind of method should at least get an invoke stub from the oat_method.
         // non-abstract methods also get their code pointers.
         const OatFile::OatMethod oat_method = oat_class->GetOatMethod(class_def_method_index);
         quick_code = oat_method.GetQuickCode();
    }
    runtime->GetInstrumentation()->InitializeMethodsCode(method, quick_code);

    if (method->IsNative()) {
    // Set up the dlsym lookup stub. Do not go through `UnregisterNative()`
    // as the extra processing for @CriticalNative is not needed yet.
        method->SetEntryPointFromJni(
            method->IsCriticalNative() ? GetJniDlsymLookupCriticalStub() : GetJniDlsymLookupStub());
  }
}

quick_code points to the value at code_offset_ in the OatMethod — the OAT-optimized native code. InitializeMethodsCode is on the Instrumentation class (art/runtime/instrumentation.cc), a familiar friend from the app-launch walkthrough: a chokepoint for key calls that lets you plug in traceable behavior. Its guts:

void Instrumentation::InitializeMethodsCode(ArtMethod* method, const void* aot_code) {
    // Use instrumentation entrypoints if instrumentation is installed.
    if (UNLIKELY(EntryExitStubsInstalled())) {
        if (!method->IsNative() && InterpretOnly()) {
            UpdateEntryPoints(method, GetQuickToInterpreterBridge());
        } else {
            UpdateEntryPoints(method, GetQuickInstrumentationEntryPoint());
        }
        return;
    }
    if (UNLIKELY(IsForcedInterpretOnly())) {
        UpdateEntryPoints(
            method, method->IsNative() ? GetQuickGenericJniStub() : GetQuickToInterpreterBridge());
        return;
    }
    // Use the provided AOT code if possible.
    if (CanUseAotCode(method, aot_code)) {
        UpdateEntryPoints(method, aot_code);
        return;
    }
    // Use default entrypoints.
    UpdateEntryPoints(
      method, method->IsNative() ? GetQuickGenericJniStub() : GetQuickToInterpreterBridge());
}

The first branch is the tracing hook: if trace stubs are installed, methods get entry points according to their kind. Otherwise the normal rules:

  • Forced-interpreter runtime:
    • Native method → entry art_quick_generic_jni_trampoline, to jump into JNI code.
    • Java method → entry art_quick_to_interpreter_bridge, to bounce into the interpreter.
  • AOT code available → use the AOT code directly.
  • No AOT code → fall back to interpretation.

UpdateEntryPoints sets the ArtMethod entry point:

static void UpdateEntryPoints(ArtMethod* method, const void* quick_code)
    REQUIRES_SHARED(Locks::mutator_lock_) {
    if (kIsDebugBuild) {
        ...
    }
    // If the method is from a boot image, don't dirty it if the entrypoint
    // doesn't change.
    if (method->GetEntryPointFromQuickCompiledCode() != quick_code) {
        method->SetEntryPointFromQuickCompiledCode(quick_code);
    }
}

Which calls:

void SetEntryPointFromQuickCompiledCode(const void* entry_point_from_quick_compiled_code)
      REQUIRES_SHARED(Locks::mutator_lock_) {
    SetEntryPointFromQuickCompiledCodePtrSize(entry_point_from_quick_compiled_code,
                                              kRuntimePointerSize);
  }

Recall the method-invocation section: in fast mode, ArtMethod::Invoke jumps to entry_point_from_quick_compiled_code. This is where that field gets written. That closes the last gap in the invocation story.

Class initialization

We’ve loaded the class — methods and fields included. Back in classForName, after loading there’s one more EnsureInitialized call, which runs ClassLinker::InitializeClass: static-field initialization plus the class’s <clinit>. That’s why Demo’s static block fired at the top of this section.

Initialization follows the Java language spec exactly — JLS 12.4.2 “Detailed Initialization Procedure”.

Practical applications

None of this is academic — ART is load-bearing in the day-to-day of Android security. Here’s where it pays off.

Hot patching & hooking

Hot patching changes an app’s behavior without touching its original code — e.g. swapping out a method implementation for a hot update. It was all the rage years ago, and faded as ART replaced Dalvik and fragmentation piled up. But the use case never died: hooking in security research is hot patching’s direct descendant.

The question it poses: how do you hijack a Java method at runtime — ideally firing your own callback before the method and before its return, so you can observe and mutate arguments and results?

From the invocation and code-loading analysis, every Java method in ART runs through ArtMethod::Invoke, which either interprets (with JIT help) or jumps to GetEntryPointFromQuickCompiledCode’s native code — and the entry is always entry_point_from_quick_compiled_code. That gives a few hijack strategies:

  1. Replace the C++ function ArtMethod::Invoke with your own, adding hijack logic.
  2. Modify the target method’s flags so every call takes the quick path, then point entry_point_from_quick_compiled_code at your implementation.
  3. Same idea, but instead of changing the pointer, patch the stub code itself.
  4. …

The ideas are simple; the engineering is not. You have to keep the stack balanced around the call — the core robustness problem of any inline-hook framework. And on newer Android, calls to system-class methods are compiled straight into assembly jumps that bypass the ArtMethod lookup entirely, so strategies 1 and 2 can’t reach them. And so on.

Take Frida, the tool everyone reaches for. Its Java method hooking lives in frida-java-bridge, key code in lib/android.js:

class ArtMethodMangler {
    replace (impl, isInstanceMethod, argTypes, vm, api) {
        this.originalMethod = fetchArtMethod(this.methodId, vm);
        const originalFlags = this.originalMethod.accessFlags;
        if ((originalFlags & kAccXposedHookedMethod) !== 0 && xposedIsSupported()) {
            // Detect Xposed; if already hooked by Xposed, re-fetch the original function ...
        }
        const replacementMethodId = cloneArtMethod(hookedMethodId, vm);
        patchArtMethod(replacementMethodId, {
    jniCode: impl,
    accessFlags: ((originalFlags & ~(kAccCriticalNative | kAccFastNative | kAccNterpEntryPointFastPathFlag)) | kAccNative) >>> 0,
    quickCode: api.artClassLinker.quickGenericJniTrampoline,
    interpreterCode: api.artInterpreterToCompiledCodeBridge
}, vm);

    // Modify flags so the interpreter takes the branch we want
    let hookedMethodRemovedFlags = kAccFastInterpreterToInterpreterInvoke | kAccSingleImplementation | kAccNterpEntryPointFastPathFlag;
    if ((originalFlags & kAccNative) === 0) {
      hookedMethodRemovedFlags |= kAccSkipAccessChecks;
    }

    patchArtMethod(hookedMethodId, {
      accessFlags: (originalFlags & ~(hookedMethodRemovedFlags)) >>> 0
    }, vm);

    // Replace the Nterp interpreter entry with art_quick_to_interpreter_bridge so the code jumps to the quick entry
    const quickCode = this.originalMethod.quickCode;
    const { artNterpEntryPoint } = api;
    if (artNterpEntryPoint !== undefined && quickCode.equals(artNterpEntryPoint)) {
      patchArtMethod(hookedMethodId, {
        quickCode: api.artQuickToInterpreterBridge
      }, vm);
    }

    // Enable the hijack
    if (!isArtQuickEntrypoint(quickCode)) {
        const interceptor = new ArtQuickCodeInterceptor(quickCode);
        interceptor.activate(vm);

        this.interceptor = interceptor;
    }

    // Record replaced methods in a hash table for later restoration
    artController.replacedMethods.set(hookedMethodId, replacementMethodId);
    notifyArtMethodHooked(hookedMethodId, vm);
    }
}

Nterp is ART’s improved interpreter, the successor to Dalvik’s mterp. Skip its internals; the hijack itself is interceptor.activate(vm). The interceptor is constructed with quickCode — the target ArtMethod’s fast-execution entry — and activate does:

activate (vm) {
    this._createTrampoline();

    const { trampoline, quickCode, redirectSize } = this;

    const writeTrampoline = artQuickCodeReplacementTrampolineWriters[Process.arch];
    const prologueLength = writeTrampoline(trampoline, quickCode, redirectSize, vm);
    this.overwrittenPrologueLength = prologueLength;

    this.overwrittenPrologue = Memory.dup(this.quickCodeAddress, prologueLength);

    const writePrologue = artQuickCodePrologueWriters[Process.arch];
    writePrologue(quickCode, trampoline, redirectSize);
}

Frida uses strategy 3 — patching the stub code itself. This is dynamic callee-side rewriting, and its advantage is that it works even for OAT-optimized system-class methods. I’m only scratching the surface; the real implementation has far more subtlety and is worth reading in full.

Hardening

Anyone who’s done Android reversing knows Dalvik bytecode is easy to read, and modern decompilers can get you strikingly close to original Java source. Companies that want to hide secrets in code find that unacceptable.

So security engineers wrap the bytecode in a protective shell and decrypt it at runtime — 加壳 in Chinese, packing in English. From the analysis above you can brainstorm the schemes (all of which exist in the wild):

  1. Encrypt the whole DEX, decrypt and load it when the shell starts.
  2. Don’t drop the file to disk — decrypt and load in memory.
  3. Extract the bytecode from the DEX and restore it at runtime.
  4. Replace each method’s bytecode with decryption code, decrypting at runtime.
  5. …

By decryption granularity, these split into whole-file shells and extraction shells (“抽取壳”). Whole-file encryption is old hat from the PC era. Extraction shells are a whole zoo: some restore code during class init, some right before a method runs.

The shell is basically a hot-patch framework that hijacks every function and restores the real bytecode before execution. Class-level schemes ride on the code-loading flow and decrypt inside <clinit>, because the Java spec guarantees it’s the first thing that runs for a class.

Because the bytecode is encrypted, dex2oat can’t optimize it at install time, so shelled code only runs interpreted at runtime (with some JIT help) — throwing away most of ART’s optimization work. And packing leans on ART’s internal symbols and offsets, which shift per version, so a mismatch means constant crashes in the field.

That’s why many top-tier vendors don’t shell their apps. For the code that really needs protecting, they push it to JNI in C/C++ and lean on LLVM’s mature obfuscation instead.

Unpacking

Because hardening is a commercial service, most malware and illicit apps use commercial shells too — a real obstacle for the good guys, which is why unpacking is a standing need.

Early on, you’d unpack by hand: analyze, debug, restore. Then people realized ART’s execution model offered a more general approach.

Take a modern extraction shell. Whatever its extraction/restore scheme, it must fall back to interpretation (at least pre-JIT), because the encrypted code was never AOT-compiled at install. And to preserve semantics, the encrypted method must be correctly decrypted before it actually runs.

So: breakpoint before an ArtMethod call, then use method->GetDexFile() to get the dex’s in-memory address and dump it. If a dex region was maliciously modified, use method->GetCodeItem() to grab the method’s decrypted bytecode and reconstruct by hand.

Dumping the entire dex requires the app to call every method of every class at runtime — not realistic. But open-source projects like DexHunter and FART modified the Android source to actively fabricate method calls, triggering the shell’s decryption and achieving full recovery.

Hardening schemes are too varied for one universal unpacker; you usually need small per-shell adaptations. Still, the unpacker has the structural edge: it only targets one environment and ignores performance, while the packer needs deep ART knowledge for stability and must cover every environment. A classic case of asymmetric offense and defense.

Method tracing

Repeated rounds of offense/defense have shown shelling is only auxiliary protection, so mobile-security vendors moved to attacking the bytecode directly: LLVM-style control-flow and data-flow obfuscation to scramble execution order and insert dead instructions, or bulk-extracting bytecode to the JNI layer wrapped in binary-level hardening — often called Java2C, transpiling Java to C and compiling it to resist analysis.

Here traditional unpacking weakens: even if you recover bytecode, the flow is scrambled; and for Java2C you’re stuck reconstructing JNI calls in the binary.

But step back — what’s reversing for? If the goal is to recover execution flow and profile behavior for forensics, dynamic tracing gets you there. We already covered hooking one method; now hook all Java methods for runtime tracing.

For example, insert a hook on method entry and exit that emits the call events and their context — process, thread ID, method name, arguments — and the receiver builds a tree-shaped call graph:

com.evilpan.Foo.onCreate
├── com.evilpan.Foo.getContacts
│   ├── Context.getContentResolver
│   ├── ContentResolver.query
│   ├── Cursor.getColumnIndex
│   ├── Cursor.getString
│   ├── ...
│   └── Cursor.close
└── com.evilpan.Foo.upload
    ├── URL.<init>
    ├── URL.openConnection
    ├── HttpURLConnection.getOutputStream
    ├── BufferWriter.write
    └── ...

With front-end filtering, that reconstructs a program’s behavior to a large degree. How to trace every Java method? entry_point_from_quick_compiled_code_ is the obvious lever, but Frida-style hijacking per-method means a lot of work — flipping access_flags, rewriting the interpreter flow, accounting for JIT. Rolling your own is fine, but ART ships a built-in backdoor: that one line we saw in LinkCode:

runtime->GetInstrumentation()->InitializeMethodsCode(method, quick_code);

Instrumentation::InitializeMethodsCode first checks whether tracing stubs are registered, and if so swaps in the method entry point directly:

// art/runtime/instrumentation.cc
void Instrumentation::InitializeMethodsCode(ArtMethod* method, const void* aot_code)
    REQUIRES_SHARED(Locks::mutator_lock_) {
  // Use instrumentation entrypoints if instrumentation is installed.
  if (UNLIKELY(EntryExitStubsInstalled())) {
    if (!method->IsNative() && InterpretOnly()) {
      UpdateEntryPoints(method, GetQuickToInterpreterBridge());
    } else {
      UpdateEntryPoints(method, GetQuickInstrumentationEntryPoint());
    }
    return;
  }
  // ...

For already-initialized ArtMethods you can also call Instrumentation::InstallStubsForMethod to install tracing on a specific method. Instrumentation is under-documented publicly; it needs source-level study.

The obvious challenges, before anyone asks:

  1. Runtime overhead.
  2. How to start/stop — interrupt-driven control.
  3. How to emit events — a dedicated queue thread, and IPC for multiple processes.
  4. Filtering — e.g. stop tracing once you’re inside system methods.
  5. Recognizing loops — the receiver only sees a sequence of repeated events.
  6. …

That’s a topic for a future writeup; it’s still exploratory.

Closing

The point of this article was the Android 12 ART implementation — how Java methods are initialized and executed. From that understanding I listed the scenarios that come up in practice: hot patching, dynamic injection, hardening, and unpacking. It’s tempting to be a pure tool-user and only care that the tool works, but knowing the machinery underneath is what lets you adapt as the offense/defense race keeps escalating.

References

  • Android Packer — facing the challenges, building solutions (slides)
  • DexDefender: A DEX Protection Scheme to Withstand MemoryDump Attack Based on Android Platform
  • ArtHook: Callee-side Method Hook Injection on the New Android Runtime ART
  • epic — Dynamic Java method AOP hook for Android
  • frida-java-bridge

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch