Colin XiaoReverse Engineering & Security Research

Software Protection, Explained: From Application Packing to Algorithm Protection

About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

Software Protection, Explained: From Application Packing to Algorithm Protection

Fair warning: this is a survey, not a deep dive.

Table of contents

As computing and the internet have exploded, software has long ceased to be the exclusive tool of “professionals” and become deeply bound to everyday life. How to properly protect software — keep it running safely on users’ and customers’ devices, serve the public well, and keep the business stable — has always been an important problem.

From a software-protection standpoint, protection can be broadly divided into application protection, code protection, and algorithm protection. This article walks through all three.

Application protection

Application packing: compression packers, encryption packers, extraction packers, protection packers.

Application protection (also called application packing) is a measure for protecting a program’s copyright information, key resources, and key implementations. “Packing” is much like how plants and animals protect their “seed” — some shells are innate (an eggshell, for instance), others are acquired (feathers, or ancient armor).

Application protection has a long history. Decompilation techniques go back to the DOS era, and application protection was already taking shape then. The shell’s original purpose was protecting software, but later it branched into several directions — compression shells, encryption shells, extraction shells, protection shells, and virtual-execution shells:

  • Compression shell: focused on reducing binary size. The core idea is compressing the program. A pure compression shell doesn’t employ protection or debugger detection.
  • Encryption shell: focused on encrypting/encoding the program, decrypting it into real code at runtime. Beyond protecting the program itself, encryption shells are also used for license registration, usage-count and time limits, and trojan evasion. In practice, encryption shells are often combined with compression shells, encoding multiple times for higher protection.
  • Extraction shell: focused on extracting code. The corresponding code/instructions are extracted up front and patched back in at runtime. Before extraction the code is usually normalized, and the back-filled code may use lazy or hot loading.
  • Protection shell: focused on protecting the shell itself (note — the shell, not the app). It uses compression, encryption, encoding, and transformation, and also detects debuggers, analysis tools, and app fingerprints — including multi-process protection, memory checks, anti-debugging, and system-injection defense.
  • Instruction shell: focused on replacing the original code/instructions and building a mapping between real instructions and custom instructions. The shell uses its own interpreter (a “virtual execution engine”) to interpret custom instructions into real code. Combined with compression, encryption, encoding, obfuscation, and transformation, a single original instruction can balloon into thousands.

How protection works

Beyond protecting copyright information, key resources, and key implementations, packing also hides the program’s real Original Entry Point (OEP) — or substitutes a fake OEP. For protections that hide the OEP, you must find the real OEP before you can unpack.

Packing and unpacking are inverse operations (analogous to encryption and decryption). The measures differ, but the flow is roughly three steps:

  1. Extract information from the unpacked application.
  2. Compress, encrypt, extract, or virtualize that extracted information (headers, data, mappings). Along the way, “auxiliary” operations happen — header erasure, data protection (encoding transformations / expression splitting), hiding the mapping, changing the loading method, app signing.
  3. At runtime, run the shell’s logic first (the shell gains execution). Before the real app runs, the shell’s unpacking logic executes; only after it finishes does the actual program run.

This achieves protection without interfering with real logic. The loading of the code (or extracted code) is also fair game — lazy loading, or “in-memory” loading that never touches a code file on disk.

Unpacking approaches

Generally, once you’ve recovered the extracted information and the unpacking logic, the reverse engineer “simulates” the shell’s unpacking logic to complete the unpack (step 3 above). The simulation differs by shell type — for a virtual-execution shell you need to locate the shell’s interpreter and the custom-bytecode mapping to restore the original.

“Simulation” is a loose description (really a mindset — packing/unpacking and encryption/decryption are much the same in flow). Concrete unpacking can be done via dump, hooking, or active invocation. That’s case-specific, so I won’t belabor actual unpacking here.

btw: to know how to attack, you must know how to defend. To handle packing and unpacking, you likely need to understand a program’s runtime (forward tracing combined with reverse analysis). For example, Java-layer packing/unpacking on Android inevitably means talking about the classloader and parent delegation; Native-layer packing/unpacking means ELF structure and the loading mechanism. Encryption shells and virtual shells are the same story.

Code protection

  • Functional equivalence: obfuscated code must produce exactly the same results as the original.
  • Structural irreversibility: use information-entropy growth to prevent recovering the original logic via reverse engineering.
  • Performance neutrality: raise analysis difficulty without introducing significant performance cost.

If application protection puts a “big lock” on the front door, code protection puts many “small locks” on the source code that makes up the application. Because code is finer-grained and more scattered, the available protection measures are more numerous.

  • Static protection: layout obfuscation, numeric obfuscation, control-flow obfuscation, preventive obfuscation
  • Dynamic protection: VMP, self-modifying code
  • Other protection: polymorphic protection, code metamorphism, code packing

In practice, code protection can rarely provide complete, permanent protection — it mostly just extends the time and effort an analyst must invest. It also consumes more compute and storage, so there’s a trade-off between performance cost and the obstacle you present to reverse engineers.

From a structural view, an application is made of code (logic code and control code) and information data. So protection splits into code protection and data protection.

  • Code protection: beyond whole-app protection, you can protect code itself. Protection is divided into static obfuscation (hindering static analysis) and dynamic protection (preventing dynamic debugging). Strictly, static obfuscation mainly targets tools, while dynamic protection targets the human analyst (raising their cost).
  • Data protection: data can be protected by authenticating/authorizing access in layers, by desensitizing sensitive data (phone numbers, passwords, ID numbers), by compressing/encoding/encrypting in transit, or even by watermarking.
  • Source-to-binary pipeline: from a build perspective, protection can happen at coding time, compile time, or post-build instruction obfuscation.

Static obfuscation

Static analysis examines a program’s static features and functional modules without running it. Its advantage is sketching the program’s outline — control flow and data structures.

Static obfuscation is an equivalence-preserving transformation that makes code hard to read, analyze, and understand. “Hard to read/analyze” is the goal; “equivalence” ensures the obfuscated code behaves identically to the original.

Code-obfuscation taxonomy usually follows Collberg’s theory: layout obfuscation, data obfuscation, control obfuscation, and preventive obfuscation (anti-deobfuscation).

Layout obfuscation

Layout obfuscation removes or obscures auxiliary textual information unrelated to execution — comments, debug info, naming, formatting — making code harder to read. It can be applied with more computation added; on its own it isn’t strong, but it’s nearly always used in practice.

Layout obfuscation also includes processing identifiers — constant names, variable names, function names — to raise comprehension difficulty. Two concrete forms:

  • Dead-code removal: strip comments, uncalled code/data, indentation and newlines (a.k.a. code minification; often paired with a formatting check so it can’t be re-formatted).
  • Identifier renaming: rename file names, constant names, variable names, class names, function/method names, and parameter names to strip code semantics.

Data obfuscation

Obfuscating algorithms.

Data is the basic element of code (note: this “data” is the code’s own data, not application data) and an important basis for semantic analysis. Data obfuscation usually introduces more computation, but protects better than layout obfuscation.

Data obfuscation is a bigger topic than layout obfuscation — more data types, and each type offers more protection measures. Let’s discuss the basic types separately.

Various data types: char, string, number (int, long), byte, etc.

String obfuscation

Strings are the most common constant data (I’ll lump char, string, and text together as “string”). They often carry important information — class names, method names, variable names, exceptions, errors, even key text. String obfuscation protects this “business-semantic” information by encoding/encrypting it.

  • Encoding obfuscation: encode key strings and decode at runtime to fight static analysis — e.g. splitting a string into single chars and converting each to a number via ASCII.
  • Encryption protection: same idea, but encrypt the key strings and decrypt at runtime.

Numeric obfuscation

Numeric obfuscation protects numbers in code via arithmetic splitting, base conversion, and formula decomposition. Numbers may carry less sensitive business semantics than strings, but they’re everywhere — loop indices, array subscripts. Obfuscating them raises overall protection.

  • Arithmetic splitting: treat the number as a “result” (normally used directly), and split that result so it must be computed first. E.g. if the source says 1, the obfuscated version can be 3+7-9+2-1 (as long as the computed value matches).
  • Base conversion: convert decimal to binary, octal, hex, base-32, etc. — similar in spirit to string encoding obfuscation, but needs no runtime conversion (the machine reads it directly).

Boolean obfuscation

Booleans are just 0/1 (and variants) at the low level, so they can use both numeric-obfuscation tricks and type-conversion tricks.

MBA (Mixed Boolean-Arithmetic) expressions are one of the most powerful tools in data obfuscation. They mix arithmetic (add, sub, mul) with boolean operations (and, or, xor) to build functionally-identical but extremely complex expressions. A simple x can be replaced with something like (x ^ 0x1234) + (x & 0x1234), dramatically raising reverse-engineering difficulty.

Control-flow obfuscation

Code blocks are arranged in logical order with related code grouped together. Without changing functionality, you can split and recombine code to break that conventional logic and blur the relationships. Control obfuscation changes or complicates the control flow. Common techniques: opaque predicates, redundant code insertion, and control-flow flattening.

  • Opaque predicates: expressions proven true by rigorous logical reasoning, but whose truth is not obvious at a glance — hence “opaque.” Their result is known but superficially unclear.
  • Redundant/dead code: code with no call relationship, or code never executed. Inserting it has no effect but raises reading difficulty. Opaque predicates can be used to insert it.
  • Control-flow flattening: transform branches and loops into a single dispatcher structure, complicating the original control flow.
  • Bogus control flow: add never-executed blocks or edges to complicate the control flow.

Function obfuscation

Function obfuscation is the most core, complex area of code obfuscation. Its goal is clear: prevent the analyst from understanding what a function “does” and “how it’s called.”

  • Call-relation obfuscation: hide the call relationships between functions so the analyst can’t reconstruct the program architecture from the call graph.
    • Indirect calls: turn a direct call (call func()) into an indirect call whose target is computed at runtime, so static tools can’t build an accurate call graph.
    • Multi-function mixing: more aggressive — shuffle the code blocks of several functions together into one giant function. From a reverse-engineering view the program looks like a single big function.
  • Code splitting & restructuring: change a function’s physical structure to raise comprehension difficulty.
    • Function outlining: extract one or more blocks of a function into a new function and insert a call in place. This adds call depth and fragments the logic.
    • Function splitting: split one complete function into several small functions to defeat similarity-based detection (e.g. function currying).

Preventive obfuscation

Preventive obfuscation’s goal isn’t to make code harder to read — it’s to make existing deobfuscation techniques harder, or to detect flaws in existing deobfuscators and design obfuscations against their weaknesses. Examples: junk instructions, debugger detection.

  • Header erasure: erase headers and sever link relationships to construct a “corrupted” file, defeating analysis tools.
  • Junk instructions: insert useless, seemingly random or harmless code while preserving correct execution, to confuse disassemblers/decompilers.
  • Instruction integrity check: a mechanism to detect unauthorized modification of obfuscated code (against deobfuscation/disassembly).
  • Debugger detection: detect debugging tools (console detection / infinite-debugger loops in browsers; Xposed/Frida/Magisk detection on Android), and runtime environment (root, hook, Unidbg/Unicorn, inline-hook detection on Android).

Dynamic obfuscation

Dynamic obfuscation modifies code at runtime so it’s hard for static tools to analyze. The goal is to prevent analysts from discovering code structure and logic at rest.

Dynamic obfuscation mainly counters dynamic analysis. Its implementations include self-modifying code (SMC), code virtualization (VM), and instruction virtualization (VMP).

VM: code virtual machines

Compile the source, modify the bytecode, then run a standard interpreter over it at runtime — e.g. the Dart VM.

VMP: code virtualization protection

Protection via a custom instruction set and interpreter. Its hallmark is the custom instruction set (a “virtual instruction set”). Since the real code only exists after VMP runs dynamically, it can’t be directly, effectively debugged.

Some practitioners treat VM protection and VMP as the same thing, some don’t. I don’t. The difference is whether you use a virtual machine or a custom instruction set — e.g. Dart VM, Java VM (ART). I haven’t found an “authoritative” statement, so take this with caution.

SMC: self-modifying code

SMC is a special technique that modifies its own code at runtime, so the program’s actual behavior differs from the disassembly. The pre-modification code segment may even be non-legal instructions that a disassembler can’t recognize, raising reverse-engineering difficulty.

It exploits the stored-program property of the Von Neumann architecture: instructions and data share one memory space, so instructions can be treated as data and read/modified by other instructions. The program writes to its code segment at runtime and executes the written data as instructions — self-modification. SMC effectively resists static analysis, and since code only appears in plaintext when needed, it also partially resists dynamic analysis.

Other obfuscation

Code obfuscation, encryption, and encoding protect code effectively, but they have a problem: if the protection never changes, it’s only a matter of time before it’s cracked.

Polymorphic protection

Polymorphic protection’s hallmark is randomness and variability. It mutates the code itself to produce many different forms with identical function, possibly even using different implementation methods.

Code metamorphism

Code metamorphism can be seen as an evolution of polymorphic protection. It goes further: on each application, it transforms not just the decryption code but the code body itself, so that instances of the same code are completely different.

Algorithm protection

Algorithm protection is also called cryptographic-algorithm protection. Software protection usually involves encoding, encryption, and hashing. The standard cryptographic algorithms all have recognizable signatures — base64’s alphabet table, MD5’s magic constants. Once you spot the key point, dynamic debugging takes you straight to it. At that point, all the application protection and code protection above is rendered moot.

So you can protect the algorithm itself to prevent a direct crypto-side entry. The three most common approaches: algorithm modification, white-boxing, and TEE.

Algorithm modification

Algorithm modification tweaks or reimplements a standard algorithm to hide its signature. Common modification points (any one or a combination):

  • Input pre-processing
  • Changing initial variables (including alphabet tables, keys)
  • Modifying the computation process
  • Post-processing the return value

You can also chain multiple algorithms together.

White-boxing

In traditional encryption, the algorithm and the key are fully independent — same algorithm, different key, different result. White-box crypto tightly binds the algorithm and the key: from them it generates an encryption table and a decryption table, then encrypts/decrypts purely by table lookup, no longer depending on the original algorithm or key.

TEE

TEE is an isolated processing environment with compute and storage that provides security and integrity protection. The core idea: allocate a hardware-isolated memory region for sensitive data; all sensitive computation happens inside it; nothing else in the hardware can access that region except through authorized interfaces. This enables private computation of sensitive data, achieving algorithm protection.

Summary

This article walked through software protection across three dimensions:

  • Application protection: covered the protection principle and the unpacking approach — giving attackers a path to unpack and defenders a target to harden against.
  • Code protection: covered static and dynamic obfuscation, plus several related obfuscation techniques.
  • Algorithm protection: covered algorithm modification, white-boxing, and TEE.

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch