Obfuscation Pass Analysis — Flattening
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
Obfuscation Pass Analysis — Flattening
Overview
Control-flow flattening recombines the original program logic into a more complex form — most visibly by turning if/else statements into a switch. The switch contains multiple branches whose execution order is randomized, though the actual program logic is unaffected. Around the switch, one or more while loops are then wrapped.
Here’s a small test program. main calls a custom add function; add checks whether num1 equals 100 (returning 0 if so), then adds num1 and num2 into num3 and returns it:
int add(int num1, int num2){
if (num1 == 100) {
return 0;
}
int num3 = num1 + num2;
return num3;
}
int main(){
int num1 = 10;
int num2 = 20;
int num3 = add(num1,num2);
return 0;
}Compiling the code and decompiling the binary in IDA shows a very simple, easily-analyzed call graph:
Source analysis
obfuscator-llvm has a code-flattening pass named flattening. Let’s peel back, layer by layer, how it flattens code.
First, debug with opt. In Flattening.cpp, line 39 registers the pass under the name flattening; line 45 calls toObfuscate to check whether a function has the fla flag (commented out here for debugging convenience). If flagged, it calls flatten — the function we’ll focus on. Flattening.cpp:
namespace {
struct Flattening : public FunctionPass {
static char ID; // Pass identification, replacement for typeid
bool flag;
Flattening() : FunctionPass(ID) {}
Flattening(bool flag) : FunctionPass(ID) { this->flag = flag; }
bool runOnFunction(Function &F);
bool flatten(Function *f);
};
}
char Flattening::ID = 0;
static RegisterPass<Flattening> X("flattening", "Call graph flattening");
Pass *llvm::createFlattening(bool flag) { return new Flattening(flag); }
bool Flattening::runOnFunction(Function &F) {
Function *tmp = &F;
// Do we obfuscate
if (toObfuscate(flag, tmp, "fla")) {
if (flatten(tmp)) {
++Flattened;
}
}
return false;
}Set a breakpoint at flatten’s entry and run po f->dump() in lldb to print the add function’s intermediate code, annotated line by line:
; Function Attrs: noinline nounwind ssp uwtable
define i32 @add(i32 %num1, i32 %num2) #0 {
entry: ; 1st BasicBlock, named entry
%retval = alloca i32, align 4 ; 4 bytes for the return value
%num1.addr = alloca i32, align 4 ; 4 bytes for num1.addr
%num2.addr = alloca i32, align 4 ; 4 bytes for num2.addr
%num3 = alloca i32, align 4 ; 4 bytes for num3
store i32 %num1, i32* %num1.addr, align 4 ; store num1 into num1.addr
store i32 %num2, i32* %num2.addr, align 4 ; store num2 into num2.addr
%0 = load i32, i32* %num1.addr, align 4 ; load num1.addr into 0
%cmp = icmp ne i32 %0, 100 ; compare 0 != 100
br i1 %cmp, label %if.then, label %if.end ; branch to if.then if true
if.then: ; 2nd BasicBlock, named if.then ; preds = %entry
store i32 0, i32* %retval, align 4
br label %return ; branch to return
if.end: ; 3rd BasicBlock, named if.end ; preds = %entry
%1 = load i32, i32* %num1.addr, align 4 ; load num1.addr into 1
%2 = load i32, i32* %num2.addr, align 4 ; load num2.addr into 2
%add = add nsw i32 %1, %2 ; add 1 + 2 into add
store i32 %add, i32* %num3, align 4 ; store add into num3
%3 = load i32, i32* %num3, align 4 ; load num3 into 3
store i32 %3, i32* %retval, align 4 ; store 3 into retval
br label %return ; branch to return
return: ; 4th BasicBlock, named return ; preds = %if.end, %if.then
%4 = load i32, i32* %retval, align 4
ret i32 %4
}There are four BasicBlocks: entry, if.then, if.end, and return. Line 63 is the first instruction flatten executes; lines 63–64 generate a random number used later:
62 // Generate a random number
63 char scrambling_key[16];
64 llvm::cryptoutils->get_bytes(scrambling_key, 16);Lines 68–69 convert switch statements into if statements:
67 // Convert switch statements
68 FunctionPass *lower = createLowerSwitchPass();
69 lower->runOnFunction(*f);Lines 72–80 save the original BasicBlocks into the origBB container:
71 // Save all original BasicBlocks
72 for (Function::iterator i = f->begin(); i != f->end(); ++i) {
73 BasicBlock *tmp = &*i;
74 origBB.push_back(tmp);
75
76 BasicBlock *bb = &*i;
77 if (isa<InvokeInst>(bb->getTerminator())) {
78 return false;
79 }
80 }Line 88 erases the first BasicBlock from origBB, since it needs special handling:
87 // Erase the first BasicBlock
88 origBB.erase(origBB.begin());
89
90 // Get the pointer to the first BasicBlock
91 Function::iteratortmp = f->begin(); //++tmp;
92 BasicBlock *insert = &*tmp;
93Lines 95–98 check whether the first BasicBlock has a conditional statement; if so, grab it into br:
94 // If the first BasicBlock has a conditional statement
95 BranchInst *br = NULL;
96 if (isa<BranchInst>(insert->getTerminator())) {
97 br = cast<BranchInst>(insert->getTerminator());
98 }Which gives:
br i1 %cmp, label %if.then, label %if.endLines 100–107 fetch the second-to-last line of the first BasicBlock:
100 if ((br != NULL && br->isConditional()) ||
101 insert->getTerminator()->getNumSuccessors() > 1) {
102 BasicBlock::iterator i = insert->end();
103 --i;
104
105 if (insert->size() > 1) {
106 --i;
107 }
108Which gives:
%cmp = icmp ne i32 %0, 100Line 109 splits the BasicBlock; line 110 puts the split-off part at the front of origBB (recall line 88 erased the first entry):
109 BasicBlock *tmpBB = insert->splitBasicBlock(i, "first");
110 origBB.insert(origBB.begin(), tmpBB);
111 }
112After the split, tmp contains:
first: ; preds = %entry
%cmp = icmp ne i32 %0, 100
br i1 %cmp, label %if.then, label %if.endAnd insert contains:
entry:
%retval = alloca i32, align 4
%num1.addr = alloca i32, align 4
%num2.addr = alloca i32, align 4
%num3 = alloca i32, align 4
store i32 %num1, i32* %num1.addr, align 4
store i32 %num2, i32* %num2.addr, align 4
%0 = load i32, i32* %num1.addr, align 4
br label %firstLine 114 removes the branch (br label %first):
113 // Remove the branch
114 insert->getTerminator()->eraseFromParent();Lines 117–122 create the switchVar used by the switch, initialized with the scrambling_key generated earlier:
116 // Create switchVar and initialize it
117 switchVar = new AllocaInst(Type::getInt32Ty(f->getContext()), 0, "switchVar", insert);
118
119 new StoreInst(
120 ConstantInt::get(Type::getInt32Ty(f->getContext()),
121 llvm::cryptoutils->scramble32(0, scrambling_key)),
122 switchVar, insert);Printing insert again shows two new lines at the end:
%switchVar = alloca i32
store i32 1207049111, i32* %switchVarLine 125 creates the empty loopEntry and loopEnd blocks; line 128 creates a load putting switchvar into loopEntry:
124 // Create the main loop
125 loopEntry = BasicBlock::Create(f->getContext(), "loopEntry", f, insert);
126 loopEnd = BasicBlock::Create(f->getContext(), "loopEnd", f, insert);
127
128 load = new LoadInst(switchVar, "switchVar", loopEntry);
129
130 // Move the first BasicBlock to the top
131 insert->moveBefore(loopEntry);
132 BranchInst::Create(loopEntry, insert);
133
134 // Branch from loopEnd to loopEntry
135 BranchInst::Create(loopEntry, loopEnd);
136
137 BasicBlock *swDefault = BasicBlock::Create(f->getContext(), "switchDefault", f, loopEnd);
138
139 BranchInst::Create(loopEnd, swDefault);
140
141 // Create the switch itself and set its condition
142 switchI = SwitchInst::Create(&*f->begin(), swDefault, 0, loopEntry);
143 switchI->setCondition(load);
144
145 // Remove the branch in the first BasicBlock and create a jump to the while loop
146 f->begin()->getTerminator()->eraseFromParent();
147
148 BranchInst::Create(loopEntry, &*f->begin()); // Add br label %loopEntryLines 151–164 fill each origBB BasicBlock as a switch case:
150 // Put all BasicBlocks into the switch
151 for (vector<BasicBlock *>::iterator b = origBB.begin(); b != origBB.end();
152 ++b) {
153 BasicBlock *i = *b;
154 ConstantInt *numCase = NULL;
155
156 // Move the BasicBlock into the switch (visually only; no logic change)
157 i->moveBefore(loopEnd);
158
159 // Add a case; switchVar is a random number
160 numCase = cast<ConstantInt>(ConstantInt::get(
161 switchI->getCondition()->getType(),
162 llvm::cryptoutils->scramble32(switchI->getNumCases(), scrambling_key)));
163 switchI->addCase(numCase, i); // Add the switch case
164 }After all that, the code looks like:
(lldb) po f->dump()
; Function Attrs: noinline nounwind ssp uwtable
define i32 @add(i32 %num1, i32 %num2) #0 {
entry:
%retval = alloca i32, align 4
%num1.addr = alloca i32, align 4
%num2.addr = alloca i32, align 4
%num3 = alloca i32, align 4
store i32 %num1, i32* %num1.addr, align 4
store i32 %num2, i32* %num2.addr, align 4
%0 = load i32, i32* %num1.addr, align 4
%switchVar = alloca i32
store i32 1207049111, i32* %switchVar
br label %loopEntry
loopEntry: ; preds = %entry,%loopEnd
%switchVar1 = load i32, i32* %switchVar
switch i32 %switchVar1, label %switchDefault [
i32 1207049111, label %first
i32 -677357051, label %if.then
i32 -1251090459, label %if.end
i32 1194405227, label %return
]
switchDefault: ; preds = %loopEntry
br label %loopEnd
first: ; preds = %loopEntry
%cmp = icmp ne i32 %0, 100
br i1 %cmp, label %if.then, label %if.end
if.then: ; preds = %loopEntry,%first
store i32 0, i32* %retval, align 4
br label %return
if.end: ; preds = %loopEntry, %first
%1 = load i32, i32* %num1.addr, align 4
%2 = load i32, i32* %num2.addr, align 4
%add = add nsw i32 %1, %2
store i32 %add,i32* %num3, align 4
%3 = load i32, i32* %num3, align 4
store i32 %3, i32* %retval, align 4
br label %return
return: ; preds = %loopEntry, %if.end, %if.then
%4 = load i32, i32* %retval, align 4
ret i32 %4
loopEnd: ; preds = %switchDefault
br label %loopEntry
}Flattening is now visible: loopEntry and loopEnd form a while loop containing a switch with four cases — first, if.then, if.end, return. If switchVar is 1207049111 it jumps to first; -677357051 → if.then; -125090459 → if.end; 1194405227 → return.
The intermediate code already has the switch, but it doesn’t update switchVar on branch, so the real program logic wouldn’t execute correctly. Lines 166–237 update switchVar. A block with a single successor is an unconditional jump, so switchVar is set directly to the successor’s case:
166 // Recompute switchVar
167 for (vector<BasicBlock *>::iterator b = origBB.begin(); b != origBB.end();
168 ++b) {
169 BasicBlock *i = *b;
170 ConstantInt *numCase = NULL;
171
172 // Ret BasicBlock is a return block; no need to update caseVar
173 if (i->getTerminator()->getNumSuccessors() == 0) {
174 continue;
175 }
176
177 // Unconditional branch
178 if (i->getTerminator()->getNumSuccessors() == 1) {
179 // Get successor and delete terminator
180 BasicBlock *succ = i->getTerminator()->getSuccessor(0);
181 i->getTerminator()->eraseFromParent();
182
183 // Get the next case
184 numCase = switchI->findCaseDest(succ);
185
186 // If next case == default case (switchDefault)
187 if (numCase == NULL) {
188 numCase = cast<ConstantInt>(
189 ConstantInt::get(switchI->getCondition()->getType(),
190 llvm::cryptoutils->scramble32(
191 switchI->getNumCases() - 1, scrambling_key)));
192 }
193
194 // Update switchVar and jump to the loop tail
195 new StoreInst(numCase, load->getPointerOperand(), i); // store numCase into %switchVar
196 BranchInst::Create(loopEnd, i); // insert br label %loopEnd
197 continue;
198 }
199
200 // Conditional branch — two successors
201 if (i->getTerminator()->getNumSuccessors() == 2) {
202 // Get the next cases
203 ConstantInt *numCaseTrue =
204 switchI->findCaseDest(i->getTerminator()->getSuccessor(0));
205 ConstantInt *numCaseFalse =
206 switchI->findCaseDest(i->getTerminator()->getSuccessor(1));
207
208 // Check next case against default case (switchDefault)
209 if (numCaseTrue == NULL) {
210 numCaseTrue = cast<ConstantInt>(numCaseTrue = cast<ConstantInt>(
211 ConstantInt::get(switchI->getCondition()->getType(),
212 llvm::cryptoutils->scramble32(
213 switchI->getNumCases() - 1, scrambling_key)));
214 }
215
216 if (numCaseFalse == NULL) {
217 numCaseFalse = cast<ConstantInt>(
218 ConstantInt::get(switchI->getCondition()->getType(),
219 llvm::cryptoutils->scramble32(
220 switchI->getNumCases() - 1, scrambling_key)));
221 }
222
223 // Create a SelectInst %1 = select i1 %cmp, i32 117441206, i32 -880348549
224 BranchInst *br = cast<BranchInst>(i->getTerminator());
225 SelectInst *sel =
226 SelectInst::Create(br->getCondition(), numCaseTrue, numCaseFalse, "",
227 i->getTerminator());
228
229 // Erase the terminator
230 i->getTerminator()->eraseFromParent();
231
232 // Update switchVar and jump to the loop tail
233 new StoreInst(sel, load->getPointerOperand(), i);
234 BranchInst::Create(loopEnd, i);
235 continue;
236 }
237 }
238
239 fixStack(f);Finally, here’s the complete intermediate code:
(lldb) po f->dump()
; Function Attrs: noinline nounwind ssp uwtable
define i32 @add(i32 %num1, i32 %num2) #0 {
entry:
%.reg2mem = alloca i32
%retval = alloca i32, align 4
%num1.addr = alloca i32, align 4
%num2.addr = alloca i32, align 4
%num3 = alloca i32, align 4
store i32 %num1, i32* %num1.addr, align 4
store i32 %num2,i32* %num2.addr, align 4
%0 = load i32, i32* %num1.addr, align 4
store i32 %0, i32* %.reg2mem
%switchVar = alloca i32
store i32 1207049111, i32* %switchVar
br label %loopEntry
loopEntry: ; preds = %entry, %loopEnd
%switchVar1 = load i32, i32* %switchVar
switch i32 %switchVar1, label %switchDefault [
i32 1207049111, label %first
i32 -677357051, label %if.then
i32 -1251090459, label %if.end
i32 1194405227, label %return
]
switchDefault: ; preds = %loopEntry
br label %loopEnd
first: ; preds = %loopEntry
%.reload = load volatile i32, i32* %.reg2mem
%cmp = icmp ne i32 %.reload, 100
%1 = select i1 %cmp, i32 -677357051, i32 -1251090459
store i32 %1, i32* %switchVar
br label %loopEnd
if.then: ; preds = %loopEntry
store i32 0, i32* %retval, align 4
store i32 1194405227, i32* %switchVar
br label %loopEnd
if.end: ; preds = %loopEntry
%2 = load i32, i32* %num1.addr, align 4
%3 = load i32, i32* %num2.addr, align 4
%add = add nsw i32 %2, %3
store i32 %add, i32* %num3, align 4
%4 = load i32, i32* %num3, align 4
store i32 %4, i32* %retval, align 4
store i32 1194405227, i32* %switchVar
br label %loopEnd
return: ; preds = %loopEntry
%5 = load i32, i32* %retval, align 4
ret i32 %5
loopEnd: ; preds = %if.end,%if.then, %first, %switchDefault
br label %loopEntry
}Now first, if.then, and if.end all update switchVar, achieving the obfuscation without changing the real program logic. That completes the full flattening process.
Summary
This post analyzed the Flattening Pass in the original OLLVM, walking through the steps with a simple example. The repo’s LLVM version is old, so readers should check newer ports for the details — the overall idea is the same, but many specifics differ. For instance, OLLVM-17 improves code structure, readability, context management, RNG, code brevity, and debug markers; Hikari uses DataLayout for layout info and adds exception-handling checks and debug output. Feel free to debug those if you want the finer points.
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch