Writing and Defeating OLLVM Control-Flow Flattening
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
This article covers both sides of OLLVM control-flow flattening: how to write the pass, and how to defeat it — including a hardened variant and an unfamiliar one found in the wild. Understanding the pass that produces the obfuscation is the fastest way to understand how to reverse it.
Writing the flattening pass
OLLVM rewrites the jump relationships of the original code blocks: it splits the function into basic blocks and hands control to a central dispatcher block.
A basic block is a block of code ending in a terminator instruction. Every basic block jumps to the return block, and the return block goes back to the dispatcher block for the next jump.
The full pass:
#include <llvm/IR/Function.h>
#include <llvm/Pass.h>
#include "llvm/IR/LegacyPassManager.h"
#include "llvm/Transforms/IPO/PassManagerBuilder.h"
#include "llvm/Support/raw_ostream.h"
#include "llvm/IR/Instructions.h"
#include "llvm/Transforms/Utils.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Transforms/Utils/Local.h"
#include <SplitBasicBlock.h>
#include <Utils.h>
#include <vector>
#include <cstdlib>
#include <ctime>
using namespace llvm;
using std::vector;
namespace{
class Flattening : public FunctionPass{
public:
static char ID;
Flattening() : FunctionPass(ID){
srand(time(0));
}
void flatten(Function &F);
bool runOnFunction(Function &F);
};
}
bool Flattening :: runOnFunction(Function &F)
{
INIT_CONTEXT(F);
// split into smaller blocks to increase obfuscation
FunctionPass *pass = createSplitBasicBlockPass();
pass->runOnFunction(F);
flatten(F);
return true;
}
void Flattening :: flatten(Function &F)
{
if(F.size() <= 1)
{
return;
}
// collect the basic blocks
vector<BasicBlock*> origBB;
for(BasicBlock &BB : F)
{
origBB.push_back(&BB);
}
// remove the entry block
origBB.erase(origBB.begin());
// get the entry block
BasicBlock &entryBB = F.getEntryBlock();
// if the entry block's terminator is a branch, put it back into the vector
if(BranchInst *br = dyn_cast<BranchInst>(entryBB.getTerminator()))
{
if(br->isConditional())
{
BasicBlock *newBB = entryBB.splitBasicBlock(br,"newBB");
origBB.insert(origBB.begin(),newBB);
}
}
// create the dispatcher block
BasicBlock *dispatchBB = BasicBlock :: Create(*CONTEXT,"dispatchBB",&F,&entryBB);
// create the return block
BasicBlock *returnBB = BasicBlock :: Create(*CONTEXT,"returnBB",&F,&entryBB);
// move the entry block to the front
entryBB.moveBefore(dispatchBB);
// remove the entry block's jump
entryBB.getTerminator()->eraseFromParent();
// entry block jumps to the dispatcher block
BranchInst *brDispatchBB = BranchInst::Create(dispatchBB,&entryBB);
int randNumCase = rand();
// create a switch variable before the jump
AllocaInst *swVarPtr = new AllocaInst(TYPE_I32,0,"swVar.ptr",brDispatchBB);
// store the random number into the pointer
new StoreInst(CONST_I32(randNumCase),swVarPtr,brDispatchBB);
// load the random number
LoadInst *swVar = new LoadInst(TYPE_I32,swVarPtr,"swVar",false,dispatchBB);
// create the default block
BasicBlock *swDefault = BasicBlock :: Create(*CONTEXT,"swDefault",&F,returnBB);
// default block jumps to the return block
BranchInst::Create(returnBB,swDefault);
// return block jumps to the dispatcher block
BranchInst::Create(dispatchBB, returnBB);
// create the switch instruction
SwitchInst *swInst = SwitchInst::Create(swVar,swDefault,0,dispatchBB);
// add a case for every basic block
for(BasicBlock *BB : origBB)
{
BB->moveBefore(returnBB);
swInst->addCase(CONST_I32(randNumCase),BB);
randNumCase = rand();
}
for(BasicBlock *BB : origBB)
{
if(BB->getTerminator()->getNumSuccessors() == 0)
{
continue;
}
// exactly one successor
else if(BB->getTerminator()->getNumSuccessors() == 1)
{
BasicBlock *sucBB = BB->getTerminator()->getSuccessor(0);
// find the case that targets the successor
ConstantInt *numCase = swInst->findCaseDest(sucBB);
// remove the tail jump
BB->getTerminator()->eraseFromParent();
new StoreInst(numCase,swVarPtr,BB);
// jump back to the return block
BranchInst::Create(returnBB,BB);
}
// two successors
else if(BB->getTerminator()->getNumSuccessors() == 2)
{
ConstantInt *numCaseTrue = swInst->findCaseDest(BB->getTerminator()->getSuccessor(0));
ConstantInt *numCaseFalse = swInst->findCaseDest(BB->getTerminator()->getSuccessor(1));
BranchInst *br = cast<BranchInst>(BB->getTerminator());
// select the case corresponding to the condition
SelectInst *sel = SelectInst::Create(br->getCondition(),KeyTrue,keyFalse,"",BB->getTerminator());
// remove the terminator
BB->getTerminator()->eraseFromParent();
// store the case value
new StoreInst(sel,swVarPtr,BB);
// jump back to the return block
BranchInst::Create(returnBB,BB);
}
}
// fix PHI instructions and escaped variables
fixStack(F);
}
char Flattening::ID = 0;
static RegisterPass<Flattening> X("fla", "Flatten basic blocks");Defeating the standard flattening
The switch-case structure controls which block runs next, but the case values written this way are exposed directly in the code, so the jump relationships are easy to recover.
Let’s defeat it by hand:
In the basic block at loc_4011DF, the switch variable is assigned, then control returns to the return block. Here 0x5CAFE12C is the next case value the dispatcher will look for.
Find the dispatcher that compares against 0x5CAFE12C and you can follow it to discover the next block is loc_4011F2.
From there you can recover the jump relationships and remove the obfuscation.
Knowing the principle, an IDC script can do this in bulk:
// work from the basic blocks
#include <idc.idc>
static NopCode(Addr, Length)
{
auto i;
for (i = 0; i < Length; i++)
{
PatchByte(Addr + i, 0x90);
}
}
static findCaseVal(targetByte,cmovAddr)
{
while(1)
{
auto currentByte = Byte(cmovAddr);
auto caseVal = Dword(cmovAddr + 1);
if(currentByte == targetByte)
{
msg("find caseVal : %X\n",caseVal);
return caseVal;
}
cmovAddr = prev_head(cmovAddr,cmovAddr - 8);
}
}
static findCaseFunc(start_addr,end_addr,caseVal)
{
msg("find case : %X\n",caseVal);
while(start_addr != BADADDR && start_addr < end_addr)
{
auto op = Byte(start_addr);
auto checkCase = Dword(start_addr + 1);
if(op == 0x2D && checkCase == caseVal)
{
auto jz_addr = start_addr + 5;
auto offest = Dword(jz_addr + 2);
auto caseFunc = offest + jz_addr + 6;
return caseFunc;
}
start_addr = next_head(start_addr, end_addr);
}
}
static main()
{
auto seg;
for (seg = get_first_seg(); seg != BADADDR; seg = get_next_seg(seg))
{
auto seg_name = get_segm_name(seg);
//msg(seg_name + "\n");
if (seg_name == ".text")
{
auto start_addr = 0x401130;
auto current_addr = 0x401130;
auto end_addr = 0x40128D;
while (current_addr != BADADDR && current_addr < end_addr)
{
auto setValPtr = Word(current_addr + 1);
auto jmpCode = Byte(current_addr + 7);
auto jmpCodeBranch = Byte(current_addr + 3);
// jmp
if(setValPtr == 0xD845 && jmpCode == 0xE9)
{
auto caseVal;
auto caseFunc;
auto returnAddr;
auto jmpOffset;
auto patchByte;
// jmp
caseVal = Dword(current_addr + 3);
msg("caseVal : %X\n",caseVal);
caseFunc = findCaseFunc(start_addr,end_addr,caseVal);
returnAddr = current_addr + 7;
msg("case func : %X\n",caseFunc);
msg("return addr : %X\n",returnAddr);
jmpOffset = caseFunc - returnAddr;
if(jmpOffset >= -126 && jmpOffset <= 129)
{
patchByte = jmpOffset - 2;
PatchByte(returnAddr,0xEB);
PatchByte(returnAddr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
NopCode(returnAddr + 2,3);
}
else
{
patchByte = jmpOffset - 5;
PatchDword(returnAddr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
}
}
// branch
if(setValPtr == 0xD845 && jmpCodeBranch == 0xE9)
{
msg("branch jmp addr : %X\n",current_addr);
auto cmovAddr = prev_head(current_addr,start_addr);
auto cmovOp = print_insn_mnem(cmovAddr);
auto op1 = print_operand(cmovAddr,0);
auto op2 = print_operand(cmovAddr,1);
auto brByte;
auto brShortByte;
auto regByte;
auto caseFunc_op1;
auto caseFunc_op2;
auto brOffest;
auto jmpAddr;
if(cmovOp == "cmovb")
{
brShortByte = 0x72;
brByte = 0x820F;
}
// else comv...
if(op1 == "eax")
{
regByte = 0xB8;
caseVal = findCaseVal(regByte,cmovAddr);
msg("caseVal : %X\n",caseVal);
caseFunc_op1 = findCaseFunc(start_addr,end_addr,caseVal);
msg("case func : %X\n",caseFunc_op1);
}
// else op1...
if(op2 == "ecx")
{
regByte = 0xB9;
caseVal = findCaseVal(regByte,cmovAddr);
msg("caseVal : %X\n",caseVal);
caseFunc_op2 = findCaseFunc(start_addr,end_addr,caseVal);
msg("case func : %X\n",caseFunc_op2);
}
// else op2...
returnAddr = current_addr + 3;
brOffest = caseFunc_op2 - cmovAddr;
msg("br offest : %X\n",brOffest);
NopCode(cmovAddr,returnAddr + 5 - cmovAddr);
msg("nop addr : %X len : %X\n",cmovAddr,returnAddr + 5 - cmovAddr);
// short br
if(brOffest > -127 && brOffest < 130)
{
PatchByte(cmovAddr,brShortByte);
patchByte = brOffest - 2;
PatchByte(cmovAddr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
}
else
{
PatchWord(cmovAddr,brByte);
patchByte = brOffest - 5;
PatchByte(cmovAddr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
}
jmpAddr = next_head(cmovAddr,end_addr);
jmpOffset = caseFunc_op1 - jmpAddr;
msg("jmp offest : %X\n",jmpOffset);
// short jmp
if(jmpOffset >= -126 && jmpOffset <= 129)
{
patchByte = jmpOffset - 2;
PatchByte(jmpAddr,0xEB);
PatchByte(jmpAddr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
}
else
{
patchByte = jmpOffset - 5;
PatchByte(jmpAddr,0xE9);
PatchDword(jmpAddr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
}
}
current_addr = next_head(current_addr, end_addr);
}
}
}
}The script works from the basic blocks. From the pass we know the last two instructions of each basic block assign swValPtr and jump to the return block, and each basic block has exactly one jump instruction — those features make basic blocks easy to locate and process.
The result — before de-obfuscation:
__int64 __fastcall enc(unsigned __int8 *a1)
{
__int64 result; // rax
int v2; // eax
int swValPtr; // [rsp+4h] [rbp-28h]
int v4; // [rsp+8h] [rbp-24h]
unsigned __int64 v5; // [rsp+14h] [rbp-18h]
unsigned __int8 *v6; // [rsp+1Ch] [rbp-10h]
unsigned __int8 v7; // [rsp+27h] [rbp-5h]
int v8; // [rsp+28h] [rbp-4h]
swValPtr = 0x5DE91606;
while ( 1 )
{
while ( 1 )
{
while ( 1 )
{
while ( swValPtr == 0x17070602 )
{
v4 = v8;
swValPtr = 0x5CAFE12C;
}
if ( swValPtr != 0x26ADAA22 )
break;
v2 = 955080127;
if ( v5 < 8 )
v2 = 0x5EA2D108;
swValPtr = v2;
}
if ( swValPtr != 0x32A4BB49 )
break;
*v6 = v7 ^ 0xA;
swValPtr = 1094857714;
}
result = (unsigned int)(swValPtr - 0x38ED5DBF);
if ( swValPtr == 0x38ED5DBF )
break;
switch ( swValPtr )
{
case 0x414233F2:
v8 = v4 + 1;
swValPtr = 386336258;
break;
case 0x5CAFE12C:
v5 = v4;
swValPtr = 648915490;
break;
case 0x5DE91606:
v4 = 0;
swValPtr = 0x5CAFE12C;
break;
default:
v6 = &a1[v4];
v7 = *v6;
swValPtr = 0x32A4BB49;
break;
}
}
return result;
}After running the IDC script:
__int64 __fastcall enc(unsigned __int8 *a1)
{
__int64 result; // rax
int i; // [rsp+8h] [rbp-24h]
for ( i = 0; ; ++i )
{
result = 955080127;
if ( (unsigned __int64)i >= 8 )
break;
a1[i] ^= 0xAu;
}
return result;
}Cleanly removed.
Hardening: mutating the case values
The problem with the standard flattening is that case values are exposed directly in the assembly, so tools like d810 clean it in one shot. To make it harder, we can mutate how the case values are generated.
for(BasicBlock *BB : origBB)
{
if(BB->getTerminator()->getNumSuccessors() == 0)
{
continue;
}
// exactly one successor
else if(BB->getTerminator()->getNumSuccessors() == 1)
{
BasicBlock *sucBB = BB->getTerminator()->getSuccessor(0);
ConstantInt *numCase = swInst->findCaseDest(sucBB);
// new: mutate the case
ConstantInt *nowCase = swInst->findCaseDest(BB);
// current case value
int nowCaseVal = nowCase->getZExtValue();
// successor case value
int numCaseVal = numCase->getZExtValue();
// compute the XOR key
int xorKey = nowCaseVal ^ numCaseVal;
LoadInst *currentSwVar = new LoadInst(TYPE_I32,swVarPtr,"curVar",false,BB->getTerminator());
ConstantInt *keyStone = CONST_I32(xorKey);
// add the XOR operation
BinaryOperator *nextSwVar = BinaryOperator::CreateXor(currentSwVar, keyStone, "nextSwVar", BB->getTerminator());
// remove the tail jump
BB->getTerminator()->eraseFromParent();
new StoreInst(nextSwVar,swVarPtr,BB);
// jump back to the return block
BranchInst::Create(returnBB,BB);
}
// two successors
else if(BB->getTerminator()->getNumSuccessors() == 2)
{
ConstantInt *numCaseTrue = swInst->findCaseDest(BB->getTerminator()->getSuccessor(0));
ConstantInt *numCaseFalse = swInst->findCaseDest(BB->getTerminator()->getSuccessor(1));
BranchInst *br = cast<BranchInst>(BB->getTerminator());
// new: mutate the case
ConstantInt *nowCase = swInst->findCaseDest(BB);
int nowCaseVal = nowCase->getZExtValue();
int trueCaseVal = numCaseTrue->getZExtValue();
int falseCaseVal = numCaseFalse->getZExtValue();
int keyTrueVal = nowCaseVal ^ trueCaseVal;
int keyFalseVal = nowCaseVal ^ falseCaseVal;
ConstantInt *KeyTrue = CONST_I32(keyTrueVal);
ConstantInt *keyFalse = CONST_I32(keyFalseVal);
// select the case corresponding to the condition
SelectInst *sel = SelectInst::Create(br->getCondition(),KeyTrue,keyFalse,"",BB->getTerminator());
LoadInst *currentSwVar = new LoadInst(TYPE_I32,swVarPtr,"curVal",BB->getTerminator());
BinaryOperator *nextSwVar = BinaryOperator::CreateXor(currentSwVar,sel,"newSwVar",BB->getTerminator());
// remove the terminator
BB->getTerminator()->eraseFromParent();
// store the case value
new StoreInst(nextSwVar,swVarPtr,BB);
// jump back to the return block
BranchInst::Create(returnBB,BB);
}
}Now each basic block XORs the switch variable instead of storing a literal, which defeats d810:
Defeating the XOR-mutated variant
Here loc_4011F6 has case value 0x772811A7; XORing it with 0x15FBEF3 gives 0x7677AF54, which is the next case value, leading to successor block loc_40120E.
For a two-successor block, the condition is cmovb, which should become jb + jmp.
The IDC script to remove it:
// work from the dispatcher block
#include <idc.idc>
static NopCode(Addr, Length)
{
auto i;
for (i = 0; i < Length; i++)
{
PatchByte(Addr + i, 0x90);
}
}
static findCaseVal(targetByte,cmov_addr)
{
while(1)
{
auto currentByte = Byte(cmov_addr);
auto caseVal = Dword(cmov_addr + 1);
if(currentByte == targetByte)
{
msg("find caseVal : %X\n",caseVal);
return caseVal;
}
cmov_addr = prev_head(cmov_addr,cmov_addr - 8);
}
}
static findCaseFunc(start_addr,end_addr,caseVal)
{
msg("find case : %X\n",caseVal);
while(start_addr != BADADDR && start_addr < end_addr)
{
auto op = Byte(start_addr);
auto checkCase = Dword(start_addr + 1);
if(op == 0x2D && checkCase == caseVal)
{
auto jz_addr = start_addr + 5;
auto offest = Dword(jz_addr + 2);
auto caseFunc = offest + jz_addr + 6;
return caseFunc;
}
start_addr = next_head(start_addr, end_addr);
}
}
static findJmpAddr(start_addr,end_addr)
{
auto i;
for(i = start_addr; i < end_addr; i = next_head(i,end_addr))
{
auto jmp_byte = Byte(i);
if(jmp_byte == 0xE9)
{
return i;
}
}
}
static checkType(start_addr,end_addr)
{
auto i;
for(i = start_addr; i < end_addr; i = next_head(i,end_addr))
{
auto xor_byte = Byte(i);
auto movSwValPtr_byte = Byte(i + 5);
auto movSwValPtr_branch = Byte(i + 3);
if(xor_byte == 0x35 && movSwValPtr_byte == 0x89)
{
return 1;
}
else if(xor_byte == 0x33 && movSwValPtr_branch == 0x89)
{
return 2;
}
}
return 0;
}
static findXorData(start_addr)
{
while(1)
{
auto xor_op = Byte(start_addr);
auto mov_op = Byte(start_addr + 5);
if(xor_op == 0x35 && mov_op == 0x89)
{
auto data = Dword(start_addr + 1);
break;
}
start_addr = next_head(start_addr, start_addr + 8);
}
return data;
}
static main()
{
auto seg;
for (seg = get_first_seg(); seg != BADADDR; seg = get_next_seg(seg))
{
auto seg_name = get_segm_name(seg);
//msg(seg_name + "\n");
if (seg_name == ".text")
{
auto start_addr = 0x401130;
auto current_addr = 0x401130;
auto end_addr = 0x4012A5;
while (current_addr != BADADDR && current_addr < end_addr)
{
auto sub_code = Byte(current_addr);
auto jz_addr = current_addr + 5;
auto jz_code = Word(jz_addr);
auto jz_offest = Dword(jz_addr + 2);
auto check_dword = Dword(current_addr + 1);
if(sub_code == 0x2D && jz_code == 0x840F)
{
auto basic_block_addr = jz_addr + jz_offest + 6;
msg("check dword : %X\n",check_dword);
msg("basic block addr : %X\n",basic_block_addr);
auto type = checkType(basic_block_addr,end_addr);
msg("type : %X\n",type);
// jmp
if(type == 1)
{
auto xor_data = findXorData(basic_block_addr);
msg("xor data : %X\n",xor_data);
auto swVar = xor_data ^ check_dword;
auto caseFunc = findCaseFunc(start_addr,end_addr,swVar);
msg("case func : %X\n",caseFunc);
auto jmp_addr = findJmpAddr(basic_block_addr,end_addr);
auto offest = caseFunc - jmp_addr;
auto patchByte;
if(offest >= -126 && offest <= 129)
{
patchByte = offest - 2;
PatchByte(jmp_addr,0xEB);
PatchByte(jmp_addr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
NopCode(jmp_addr + 2,3);
}
else
{
patchByte = offest - 5;
PatchDword(jmp_addr + 1,patchByte);
}
NopCode(jmp_addr - 8,8);
}
// branch
if(type == 2)
{
jmp_addr = findJmpAddr(basic_block_addr,end_addr);
auto mov_addr = prev_head(jmp_addr,basic_block_addr);
auto xor_addr = prev_head(mov_addr,basic_block_addr);
auto cmov_addr = prev_head(xor_addr,basic_block_addr);
auto cmov_op = print_insn_mnem(cmov_addr);
auto op1 = print_operand(cmov_addr,0);
auto op2 = print_operand(cmov_addr,1);
auto br_byte;
auto br_short;
auto caseFunc_op1;
auto caseFunc_op2;
auto regByte;
if(cmov_op == "cmovb")
{
br_short = 0x72;
br_byte = 0x820F;
}
// other comv...
if(op1 == "eax")
{
regByte = 0xB8;
xor_data = findCaseVal(regByte,cmov_addr);
msg("xor data : %X\n",xor_data);
swVar = check_dword ^ xor_data;
caseFunc_op1 = findCaseFunc(start_addr,end_addr,swVar);
msg("case func : %X\n",caseFunc_op1);
}
// other op1...
if(op2 == "ecx")
{
regByte = 0xB9;
xor_data = findCaseVal(regByte,cmov_addr);
msg("xor data : %X\n",xor_data);
swVar = check_dword ^ xor_data;
caseFunc_op2 = findCaseFunc(start_addr,end_addr,swVar);
msg("case func : %X\n",caseFunc_op2);
}
// other op2...
auto brOffest = caseFunc_op2 - cmov_addr;
msg("br offest : %X\n",brOffest);
NopCode(cmov_addr,jmp_addr + 5 - cmov_addr);
msg("nop addr : %X len : %X\n",cmov_addr,jmp_addr + 5 - cmov_addr);
// short br
if(brOffest > -127 && brOffest < 130)
{
PatchByte(cmov_addr,br_short);
patchByte = brOffest - 2;
PatchByte(cmov_addr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
}
else
{
PatchWord(cmov_addr,br_byte);
patchByte = brOffest - 5;
PatchByte(cmov_addr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
}
auto jmpAddr = next_head(cmov_addr,end_addr);
auto jmpOffset = caseFunc_op1 - jmpAddr;
msg("jmp offest : %X\n",jmpOffset);
// short jmp
if(jmpOffset >= -126 && jmpOffset <= 129)
{
patchByte = jmpOffset - 2;
PatchByte(jmpAddr,0xEB);
PatchByte(jmpAddr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
}
else
{
patchByte = jmpOffset - 5;
PatchByte(jmpAddr,0xE9);
PatchDword(jmpAddr + 1,patchByte);
msg("patch byte : %X\n",patchByte);
}
}
}
current_addr = next_head(current_addr, end_addr);
}
}
}
}Because the XOR needs the block’s own case value, parsing now starts from the dispatcher block.
Result — before:
__int64 __fastcall enc(unsigned __int8 *a1)
{
__int64 result; // rax
int v2; // eax
int swValPtr; // [rsp+4h] [rbp-28h]
int v4; // [rsp+8h] [rbp-24h]
unsigned __int64 v5; // [rsp+14h] [rbp-18h]
unsigned __int8 *v6; // [rsp+1Ch] [rbp-10h]
unsigned __int8 v7; // [rsp+27h] [rbp-5h]
int v8; // [rsp+28h] [rbp-4h]
swValPtr = 1392335309;
while ( 1 )
{
while ( 1 )
{
while ( swValPtr == 80288048 )
{
v6 = &a1[v4];
v7 = *v6;
swValPtr = 1219708018;
}
if ( swValPtr != 188876390 )
break;
v8 = v4 + 1;
swValPtr = 1299578078;
}
result = (unsigned int)(swValPtr - 989136729);
if ( swValPtr == 989136729 )
break;
switch ( swValPtr )
{
case 1219708018:
*v6 = v7 ^ 0xA;
swValPtr = 188876390;
break;
case 1299578078:
v4 = v8;
swValPtr = 1999114663;
break;
case 1392335309:
v4 = 0;
swValPtr = 1999114663;
break;
case 1987555156:
v2 = 1283631117;
if ( v5 < 8 )
v2 = 1925101156;
swValPtr ^= v2;
break;
case 1999114663:
v5 = v4;
swValPtr = 1987555156;
break;
}
}
return result;
}After:
__int64 __fastcall enc(unsigned __int8 *a1)
{
__int64 result; // rax
int i; // [rsp+8h] [rbp-24h]
for ( i = 0; ; ++i )
{
result = 1283631117;
if ( (unsigned __int64)i >= 8 )
break;
a1[i] ^= 0xAu;
}
return result;
}The IDC approach is general, but the code above is tuned to the specific binary I was analyzing. To reuse it on different files, factor the common pieces into functions and adapt to each OLLVM mutation.
Case study: a CrackMe with an unfamiliar OLLVM
This CrackMe uses a modified OLLVM. Let’s walk through analyzing and defeating an unfamiliar flattening from scratch, then write a one-click IDC script.
Looking at the CFG, it differs from what we learned earlier: there are only entry, dispatcher, and basic blocks — no return block.
Every basic block jumps straight back to the main dispatcher, and there’s no “store the case value into a pointer” operation. So how does the dispatcher decide the next block?
Look at the main dispatcher:
The value compared against the case values is register eax, computed by a formula where only ecx is unknown. And in the basic blocks, each block assigns and transforms ecx — which is exactly why there’s no return block: each basic block computes ecx and passes it to the main dispatcher, which then computes the case value.
Let’s compute the case value of loc_140001F5C by hand: 5DE42860. Now look for the sub-dispatcher entry that matches it:
No sub-dispatcher compares against 5DE42860 directly. Observe the sub-dispatcher: except for the bottom layer (jz/jnz equality comparisons), all comparisons are magnitude comparisons. The sub-dispatcher doesn’t spell out every case value; it decides via a series of greater/less-than comparisons. Tracing 5DE42860 down the comparison tree, it should eventually land at loc_140001DF2.
OLLVM scrambles the original block relationships and centralizes them in the dispatcher, so de-obfuscating means restoring those relationships. We’ve already recovered loc_140001F5C -> loc_140001DF2. Repeat for every dispatcher block.
The IDC script:
#include <idc.idc>
static NopCode(Addr, Length)
{
auto i;
for (i = 0; i < Length; i++)
{
PatchByte(Addr + i, 0x90);
}
}
static calCaseVal(key_data)
{
auto mul_data = 0x3650F7CD;
auto xor_data = 0x268916D7;
auto caseVal = (((mul_data * key_data) & 0xFFFFFFFF) ^ xor_data) & 0xFFFFFFFF;
return caseVal;
}
static checkJmp(jmp_name)
{
if(jmp_name == "jg" | jmp_name == "jz" | jmp_name == "jnz" | jmp_name == "jle" | jmp_name == "jmp")
{
return 1;
}
return 0;
}
static getRegData(current_addr,start_addr,reg_name)
{
auto xor_reg_name;
auto add_data = 0;
auto sub_data = 0;
auto xor_data = 0;
while(current_addr != BADADDR && current_addr > start_addr)
{
auto insn_name = print_insn_mnem(current_addr);
auto reg = print_operand(current_addr,0);
auto op2 = print_operand(current_addr,1);
auto data = get_operand_value(current_addr,1) & 0xFFFFFFFF;
if(reg == reg_name)
{
if(insn_name == "mov")
{
auto reg_data = data;
break;
}
if(insn_name == "add")
{
add_data = data;
}
if(insn_name == "sub")
{
sub_data = data;
}
if(insn_name == "xor")
{
xor_reg_name = op2;
}
}
if(reg == xor_reg_name)
{
if(insn_name == "mov")
{
xor_data = data;
}
}
current_addr = prev_head(current_addr,start_addr);
}
auto key_data = ((reg_data + add_data - sub_data) ^ xor_data) & 0xFFFFFFFF;
return key_data;
}
extern ecx_data;
extern other_data;
static getKeyData(current_addr,start_addr,type)
{
if(type == 1)
{
return getRegData(current_addr,start_addr,"ecx");
}
if(type == 2)
{
ecx_data = getRegData(current_addr,start_addr,"ecx");
auto other_reg = getOtherReg(current_addr,start_addr);
other_data = getRegData(current_addr,start_addr,other_reg);
}
}
static getOtherReg(current_addr,start_addr)
{
current_addr = prev_head(current_addr,start_addr);
while(current_addr != BADADDR && current_addr > start_addr)
{
auto cmov_insn = print_insn_mnem(current_addr);
auto other_reg = print_operand(current_addr,1);
if(strstr(cmov_insn, "cmov", 0) != -1)
{
return other_reg;
}
if(checkJmp(cmov_insn))
{
break;
}
current_addr = prev_head(current_addr,start_addr);
}
return 1;
}
static getCmovType(current_addr,start_addr)
{
current_addr = prev_head(current_addr,start_addr);
while(current_addr != BADADDR && current_addr > start_addr)
{
auto cmov_insn = print_insn_mnem(current_addr);
if(strstr(cmov_insn, "cmov", 0) != -1)
{
if(cmov_insn == "cmovb")
{
return "jb";
}
if(cmov_insn == "cmovz")
{
return "jz";
}
if(cmov_insn == "cmovnz")
{
return "jnz";
}
}
if(checkJmp(cmov_insn))
{
break;
}
current_addr = prev_head(current_addr,start_addr);
}
return 1;
}
static checkType(current_addr,start_addr)
{
current_addr = prev_head(current_addr,start_addr);
while(current_addr != BADADDR && current_addr > start_addr)
{
auto cmov_insn = print_insn_mnem(current_addr);
if(strstr(cmov_insn, "cmov", 0) != -1)
{
return 2;
}
if(checkJmp(cmov_insn))
{
break;
}
current_addr = prev_head(current_addr,start_addr);
}
return 1;
}
static isBasicBlock(branch_addr)
{
auto branch_name = print_insn_mnem(branch_addr);
auto op1 = print_operand(branch_addr,0);
if(branch_name == "cmp" && op1 == "eax")
{
return 0;
}
return 1;
}
static findJmpAddr(jmp_basicblock_addr,caseVal,current_addr,end_addr)
{
while (current_addr != BADADDR && current_addr < end_addr)
{
auto op = Byte(current_addr);
auto cmpCase = Dword(current_addr + 1);
auto jmp_insn_addr = current_addr + 5;
auto jmp_name = print_insn_mnem(jmp_insn_addr);
auto next_addr = next_head(jmp_insn_addr,end_addr);
auto insn_len = next_addr - (jmp_insn_addr);
auto offset;
if(insn_len < 5)
{
offset = Byte(current_addr + 6);
}
else
{
offset = Dword(current_addr + 7);
}
if(offset > 0xFF000000)
{
offset = 0xFFFFFFFF00000000 | offset;
}
auto jmp_addr = next_addr + offset;
auto is_jmp = checkJmp(jmp_name);
auto greater_branch;
auto other_branch;
auto equal_branch;
auto branch_addr;
if(op == 0x3D && is_jmp)
{
if(jmp_name == "jg")
{
greater_branch = jmp_addr;
other_branch = next_addr;
if(caseVal > cmpCase)
{
branch_addr = greater_branch;
}
else
{
branch_addr = other_branch;
}
}
if(jmp_name == "jz")
{
equal_branch = jmp_addr;
other_branch = next_addr;
if(caseVal == cmpCase)
{
branch_addr = equal_branch;
}
else
{
branch_addr = other_branch;
}
}
if(jmp_name == "jnz")
{
equal_branch = next_addr;
other_branch = jmp_addr;
if(caseVal == cmpCase)
{
branch_addr = equal_branch;
}
else
{
branch_addr = other_branch;
}
}
if(jmp_name == "jle")
{
greater_branch = next_addr;
other_branch = jmp_addr;
if(caseVal > cmpCase)
{
branch_addr = greater_branch;
}
else
{
branch_addr = other_branch;
}
}
if(isBasicBlock(branch_addr))
{
break;
}
current_addr = prev_head(branch_addr,branch_addr - 10);
}
current_addr = next_head(current_addr, end_addr);
}
msg("%X -> %X\n",jmp_basicblock_addr,branch_addr);
return branch_addr;
}
static findMovAddr(current_addr,start_addr)
{
current_addr = prev_head(current_addr,start_addr);
while(current_addr != BADADDR && current_addr > start_addr)
{
auto reg_name = print_operand(current_addr,0);
auto op = print_insn_mnem(current_addr);
if(op == "mov" && reg_name == "ecx")
{
return current_addr;
}
if(checkJmp(op))
{
break;
}
current_addr = prev_head(current_addr,start_addr);
}
}
static PatchJmp(current_addr,next_insn_addr,start_addr,branch_addr_ecx,branch_addr_other,type,cmov_type)
{
auto mov_addr;
auto nop_len;
auto offset;
mov_addr = findMovAddr(current_addr,start_addr);
nop_len = next_insn_addr - mov_addr;
NopCode(mov_addr,nop_len);
if(type == 1)
{
PatchByte(mov_addr,0xE9);
offset = branch_addr_ecx - mov_addr - 5;
PatchDword(mov_addr + 1,offset);
}
if(type == 2)
{
auto cmov_byte;
if(cmov_type == "jz")
{
cmov_byte = 0x840F;
}
if(cmov_type == "jnz")
{
cmov_byte = 0x850F;
}
auto offest_other = branch_addr_other - mov_addr - 6;
PatchWord(mov_addr,cmov_byte);
PatchDword(mov_addr + 2,offest_other);
auto jmp_addr = mov_addr + 6;
PatchByte(jmp_addr,0xE9);
auto offest_ecx = branch_addr_ecx - jmp_addr - 5;
PatchDword(jmp_addr + 1,offest_ecx);
}
}
static main()
{
auto current_addr = 0x1400017D8;
auto start_addr = 0x1400017D8;
auto end_addr = 0x140001FAF;
while (current_addr != BADADDR && current_addr < end_addr)
{
auto jmp_returnBlock = Byte(current_addr);
auto next_insn_addr = next_head(current_addr,end_addr);
auto jmp_returnBlock_offset = Dword(current_addr + 1);
if(jmp_returnBlock_offset > 0xFF000000)
{
jmp_returnBlock_offset = 0xFFFFFFFF00000000 | jmp_returnBlock_offset;
}
auto swichBlock_addr = 0x1400019E0;
// basic block
if(jmp_returnBlock == 0xE9 && next_insn_addr + jmp_returnBlock_offset == swichBlock_addr)
{
msg("basicblock addr : %X\n",current_addr);
auto type = checkType(current_addr,start_addr);
auto key_data;
auto caseVal;
auto branch_addr;
// branch == 1
if(type == 1)
{
key_data = getKeyData(current_addr,start_addr,type);
caseVal = calCaseVal(key_data);
branch_addr = findJmpAddr(current_addr,caseVal,start_addr,end_addr);
PatchJmp(current_addr,next_insn_addr,start_addr,branch_addr,0,type,0);
}
// branch == 2
if(type == 2)
{
getKeyData(current_addr,start_addr,type);
key_data = ecx_data;
auto caseVal_ecx = calCaseVal(key_data);
auto caseVal_other = calCaseVal(other_data);
auto cmov_type = getCmovType(current_addr,start_addr);
msg(cmov_type + " ");
auto branch_addr_other = findJmpAddr(current_addr,caseVal_other,start_addr,end_addr);
msg("jmp ");
auto branch_addr_ecx = findJmpAddr(current_addr,caseVal_ecx,start_addr,end_addr);
PatchJmp(current_addr,next_insn_addr,start_addr,branch_addr_ecx,branch_addr_other,type,cmov_type);
}
}
current_addr = next_head(current_addr, end_addr);
}
}How the script works
Following the earlier approach, we start from the basic blocks. Each basic block ends by jumping to a fixed address — previously the return block, now the main dispatcher — which makes it easier to locate than anything else.
// get the first byte; check it's a jump (0xE9)
auto jmp_returnBlock = Byte(current_addr);
auto next_insn_addr = next_head(current_addr,end_addr);
// get the dword at +1; if the first byte is a jump, this is the offset
auto jmp_returnBlock_offset = Dword(current_addr + 1);
if(jmp_returnBlock_offset > 0xFF000000)
{
// handle negative offsets
jmp_returnBlock_offset = 0xFFFFFFFF00000000 | jmp_returnBlock_offset;
}
auto swichBlock_addr = 0x1400019E0;
// verify this jumps to the main dispatcher
if(jmp_returnBlock == 0xE9 && next_insn_addr + jmp_returnBlock_offset == swichBlock_addr)
{
// process the basic block
}Once a basic block is found, compute the corresponding ecx value. Basic blocks are handled by number of successors:
static getRegData(current_addr,start_addr,reg_name)
{
auto xor_reg_name;
auto add_data = 0;
auto sub_data = 0;
auto xor_data = 0;
while(current_addr != BADADDR && current_addr > start_addr)
{
auto insn_name = print_insn_mnem(current_addr);
// operand 1
auto reg = print_operand(current_addr,0);
// operand 2
auto op2 = print_operand(current_addr,1);
// operand 2's concrete value
auto data = get_operand_value(current_addr,1) & 0xFFFFFFFF;
// if operand 1 is the register we want
if(reg == reg_name)
{
if(insn_name == "mov")
{
auto reg_data = data;
break;
}
if(insn_name == "add")
{
add_data = data;
}
if(insn_name == "sub")
{
sub_data = data;
}
if(insn_name == "xor")
{
// the xor is not against an immediate but another register
xor_reg_name = op2;
}
}
// find the assignment to the xor register
if(reg == xor_reg_name)
{
if(insn_name == "mov")
{
xor_data = data;
}
}
current_addr = prev_head(current_addr,start_addr);
}
// compute the final value
auto key_data = ((reg_data + add_data - sub_data) ^ xor_data) & 0xFFFFFFFF;
return key_data;
}Once ecx is computed, a fixed formula yields the block’s case value.
Next, process the dispatcher to find the next block. The sub-dispatcher is easy to locate: every block is a compare-plus-jump, and the jumps come in only two flavors — greater / not-greater and equal / not-equal — since the only jump instructions are jg, jle, jz, jnz.
static findJmpAddr(jmp_basicblock_addr,caseVal,current_addr,end_addr)
{
while (current_addr != BADADDR && current_addr < end_addr)
{
auto op = Byte(current_addr);
auto cmpCase = Dword(current_addr + 1);
auto jmp_insn_addr = current_addr + 5;
auto jmp_name = print_insn_mnem(jmp_insn_addr);
auto next_addr = next_head(jmp_insn_addr,end_addr);
auto insn_len = next_addr - (jmp_insn_addr);
auto offset;
// handle short jumps
if(insn_len < 5)
{
offset = Byte(current_addr + 6);
}
else
{
offset = Dword(current_addr + 7);
}
// handle negative offsets
if(offset > 0xFF000000)
{
offset = 0xFFFFFFFF00000000 | offset;
}
auto jmp_addr = next_addr + offset;
auto is_jmp = checkJmp(jmp_name);
auto greater_branch;
auto other_branch;
auto equal_branch;
auto branch_addr;
if(op == 0x3D && is_jmp)
{
// greater-than jump
if(jmp_name == "jg")
{
greater_branch = jmp_addr;
other_branch = next_addr;
if(caseVal > cmpCase)
{
branch_addr = greater_branch;
}
else
{
branch_addr = other_branch;
}
}
// the rest are analogous
if(jmp_name == "jz")
{
equal_branch = jmp_addr;
other_branch = next_addr;
if(caseVal == cmpCase)
{
branch_addr = equal_branch;
}
else
{
branch_addr = other_branch;
}
}
if(jmp_name == "jnz")
{
equal_branch = next_addr;
other_branch = jmp_addr;
if(caseVal == cmpCase)
{
branch_addr = equal_branch;
}
else
{
branch_addr = other_branch;
}
}
if(jmp_name == "jle")
{
greater_branch = next_addr;
other_branch = jmp_addr;
if(caseVal > cmpCase)
{
branch_addr = greater_branch;
}
else
{
branch_addr = other_branch;
}
}
if(isBasicBlock(branch_addr))
{
break;
}
current_addr = prev_head(branch_addr,branch_addr - 10);
}
current_addr = next_head(current_addr, end_addr);
}
// print the recovered jump relationship
msg("%X -> %X\n",jmp_basicblock_addr,branch_addr);
return branch_addr;
}That recovers the key jump relationships. Running it:
basicblock addr : 140001A77
140001A77 -> 140001F94
basicblock addr : 140001AEE
jnz 140001AEE -> 140001F5C
jmp 140001AEE -> 140001DB9
basicblock addr : 140001BF9
jz 140001BF9 -> 140001A2F
jmp 140001BF9 -> 140001E7A
basicblock addr : 140001C4B
jz 140001C4B -> 140001E7A
jmp 140001C4B -> 140001AA1
basicblock addr : 140001D16
jnz 140001D16 -> 140001B16
jmp 140001D16 -> 140001E3C
basicblock addr : 140001D45
140001D45 -> 140001E5E
basicblock addr : 140001DB4
jz 140001DB4 -> 140001990
jmp 140001DB4 -> 140001D26
basicblock addr : 140001DED
140001DED -> 140001DF2
basicblock addr : 140001E37
jnz 140001E37 -> 140001E5E
jmp 140001E37 -> 140001C16
basicblock addr : 140001E59
140001E59 -> 140001B16
basicblock addr : 140001E75
140001E75 -> 140001990
basicblock addr : 140001F57
jnz 140001F57 -> 140001DB9
jmp 140001F57 -> 140001D55
basicblock addr : 140001F8F
140001F8F -> 140001DF2With those in hand, patch the computations into direct jumps.
Finally, remember to fix the entry block — the logic above works for basic blocks, but the entry block needs special handling.
The entry block jumps to the main dispatcher with a short jump (two bytes), but a normal jump needs five bytes. A reasonable compromise: NOP the main dispatcher and rewrite it as a jmp.
Before / after:
Clean.
Summary
When you meet an unfamiliar OLLVM, two observations reveal the removal method: (1) how the case values are computed, and (2) how the dispatcher compares. Scripting follows the same reasoning. The pass that produces the flattening always tells you exactly what to look for when you defeat it.
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch