Colin XiaoReverse Engineering & Security Research

Reversing the MSVC Compiler: IL Emission and the Warbird Obfuscation Interface

About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

This is a first-pass reverse engineering record of the MSVC compiler toolchain (Visual Studio 2022), motivated by wanting to understand the compiler’s IL format and — more interestingly — the Warbird obfuscation interfaces. The end goal is understanding how compile-time obfuscation works under the hood, which is useful whether you’re building your own obfuscation passes or analyzing binaries that have been through one.

Component map

DLL Role
c1.dll parses C source and emits IL
c1xx.dll parses C++ source and emits IL
c2.dll compiler backend: optimizes and generates actual assembly
linker.exe the linker
cl.exe the compiler command-line driver

c1.dll: the front end

main_compile

The call chain is cl.exe -> c1.dll -> ... -> main_compile. The reconstructed control flow:

// Second-stage initialization.
// First stage (init_main1) handles command-line args and environment variables.
// Second stage (init_main2) initializes the compiler's internal modules.
init_main2();

// ...

// Various token handling and optional features.
// ...

// /PH emits #pragma file_hash during preprocessing
if (EmitPragmaFileHash)
    SymbolTableManager_t::DumpFileHashes(PchS);

// ...

// Some special handling
if (!PchUFlag)
{
    Comment_type = 10;
    Comment_string = "/CLRNETCORE";
    op_pragma(0xB); // calls ILEmitter::EmitPragma to emit a pragma IL
    if (Has_com_entry)
    {
        Comment_type = 10;
        Comment_string = "/NOCOMENTRY";
        op_pragma(0xB); // same as above
    }
}

// ...

// Possibly inject headers or handle preprocessor directives
InjectArtifactsBeforeInputSource();
ApplyCmdLineWarningFlags();

// ...

// Some precompiled-header special handling
if (!PchUFlag)
{
    MetaDataEmitter::Initialize();
    metaEmitter = MetaDataEmitter::Instance;
}

// ...

if (!PchUFlag || WorkList_t::m_usingWorkList)
{
    StoilHeader();
    ProcessBEWarnings();
}

if ( PchUFile )
    EnsureFilenameIsEmitted();

// ...

// Parse the source code and convert it to IL
CallPrimaryParser();

// ...

// Emit symbol data into GlobalSymbolIL and LocalSymbolIL
SymbolTableManager_t::DumpGlobalSymbols(PchS);

// ...

StoilHeader();

// ...

// IL generation done; close the file handles
ILSink::closeFile(&ilsExp);
ILSink::closeFile(&LocalSymbolIL);
ILSink::closeFile(&GlobalSymbolIL);
ILSink::closeFile(&ilsInit);
if ( IncludeDebugDataBase )
    ILSink::closeFile(&ilsDB);

// cleanup
// ...

init_main2

// Initialize various modules
// ...

// %Temp% + "/_CL_" + hash + "ex"
OpenExpFile();

// ...

// Open the IL files in the temp folder: %Temp% + "/_CL_" + hash + section
ILSink::fopen(&LocalSymbolIL, Basename, "sy", 6);
ILSink::fopen(&GlobalSymbolIL, Basename, "gl", 6);
ILSink::fopen(&ilsInit, Basename, "in", 6);
if (IncludeDebugDataBase)
    ILSink::fopen(&ilsDB, Basename, "db", 6);

// ...

OpenExpFile

// ...

ILSink::fopen(&ilsExp, Basename, "ex", 6)

// ...

if (!PchUFlag)
{
    ILSink::chwrite(&ilsExp, 0x5B);
    ILSink::lowrite(&ilsExp, 0x58);
    status = ILSink::seek();
    IOGlobalVal = status;
    if (status)
    {
        // handle error
    }
    ILSink::chwrite(&ilsExp, 0);
}

// ...

ILSink

Called by ILEmitter and its components to actually write bytes.

seek — appears to only be used to seek the ilsExp read/write pointer to 0x57:

// ...

// Jump straight to 0x57
fseek(&ilsExp, 0x57, 0);

// ...

chwrite — writes a single byte into the buffer:

// Grow the buffer
if (this->buffer_start + this->buffer_size - this->buffer_end >= 2
    || (res = Buffer::grow(this->buffer_start, 2), ok = 0, res) )
{
    ok = 1;
}

// Check for failure
fail = (ok & this->good) == 0;
this->good &= ok;
if (!fail)
    *(uint8_t *)this->buffer_end++ = ch;

lowrite — writes a compressed integer (single byte if it fits in 7 bits, otherwise a 0x80 marker plus 4 bytes):

// Grow the buffer
if (this->buffer_start + this->buffer_size - this->buffer_end >= 2
    || (res = Buffer::grow(this->buffer_start, 2), ok = 0, res) )
{
    ok = 1;
}

// Check for failure
fail = (ok & this->good) == 0;
this->good &= ok;
if (!fail) {
    buf = this->buffer_end;

    // Decide between single-byte and multi-byte encoding
    if (val & 0xFFFFFF80 != 0) {

        // 0x80 1000 0000 multi-byte marker
        *(uint8_t *)buf = 0x80;

        // Write the 4-byte value
        *(uint32_t *)++this->buffer_end = val;
        this->buffer_end += 4;
    } else {

        // Write a single byte
        *(uint8_t *)buf = val;
        ++this->buffer_end;
    }
}

ILEmitter

The class that emits IL:

ILEmitter::EmitOPbranch(OPTYPE,Symbol_t *);
ILEmitter::EmitOPgoto(Symbol_t *);
ILEmitter::EmitOPlabel(Symbol_t *);
ILEmitter::EmitOPname(Symbol_t *);
ILEmitter::EmitOPnbranch(Symbol_t *);
ILEmitter::EmitOPpragma(uchar);
ILEmitter::EmitOPswexp(Symbol_t *);
ILEmitter::EmitOPswitch(Symbol_t *,Expr_t *);
ILEmitter::EmitPrFilename(Filename *);
ILEmitter::EmitPragma(PRAGMA_TYPE,MessageNumber);
ILEmitter::EmitPragmaComment(PragmaComment_t::Kind_t,char const *);
ILEmitter::EmitWarningPragma(WarningState_t *,WarningState_t * const,std::vector<WarningNumber> const &);
ILEmitter::toil(s_tree *);
ILEmitter::toil_direct(s_tree *);

IL conversion flow (c1.dll)

For ilsExp and ilsInit:

source code -> PrimaryParser::Parse -> yacc parser -> ttoil (recursive)
    -> ILEmitterVisitor -> ILEmitter -> ILSink -> ilsExp & ilsInit

For LocalSymbolIL and GlobalSymbolIL:

source code -> PrimaryParser::Parse -> parse complete
    -> SymbolTableManager_t::DumpGlobalSymbols -> SymbolILWriters
    -> various Emitters -> ILSink -> LocalSymbolIL & GlobalSymbolIL

For ilsDB:

source code -> CVTSInit (start recording) -> PrimaryParser::Parse etc.
    -> save the records produced during parsing -> CVTSEnd (end recording) -> ilsDB

OBJ sections and IL file meanings

sy -> LocalSymbolIL  // local symbol info
gl -> GlobalSymbolIL // global symbol info
in -> ilsInit        // variable initialization data
db -> ilsDB          // debug database
ex -> ilsExp         // IL produced by ttoil and ILEmitterVisitor

// fg is not produced by c1.dll
fg -> command-line arguments

c2.dll: the backend

il_file

Opens the IL generated by c1:

wcscpy_s(file_name, 0x104, basename);
wcscat_s(file_name, 0x104, section);

// ...

utc_wfopen_s(&fs, file_name, L"rb")

CilToObj

Writes an IL file into the OBJ:

// secPrefix = ".cil$"
// secName = "ex" or "gl" or ...
// concatenate the section name
sectionName = pstrconcatdup(secPrefix, secName);
key = newkey();
seg = newseg(sectionName, "ILFILE", key, 4, 1, 0);
seginit(seg);

// ...

coff_setsection(seg);

// ...

// read the IL file data
int realSize = fread(buf, 1, ilSize, fs);

// ...

// write the data into the section
IfilePutNbytes(buf, realSize, 0);

// ...

LtcgCilToObj

Converts the IL files produced by c1 into OBJ (LTCG path):

// ...

ilmap_open(Basename, L"gl", ilMap);
read_gloil_header(ilMap);

// ...

ilmap_close(ilMap);

// ...

obj_start();
coff_directive("-compiler:", *flags, 0, 0);
CilWriteFlags(res, flags);

// ...

gl = il_file(Basename, L"gl");
CilToObj(gl, "gl", 0);
utc_fclose(gl);
in = il_file(Basename, L"in");
CilToObj(in, "in", 0);
utc_fclose(in);
CoffWriteChar(15);

// ...

ex = il_file(Basename, L"ex");
CilToObj(ex, "ex", 0);
utc_fclose(ex);
sy = il_file(Basename, L"sy");
CilToObj(sy, "sy", 0);
utc_fclose(sy);

// ...

md = il_file(Basename, L"md");
CilToObj(md, "md", 0);
utc_fclose(md);

// ...

db = il_file(Basename, L"db");
CilToObj(db, "db", 0);
utc_fclose(db);
CoffWriteChar(15);

// ...

CilToObj(pch, "pch");
utc_fclose(pch);
ModReleasePchs();

// ...

Warbird interface analysis

Warbird exposes two entry points: CreateObfuscator2 (link-time obfuscation) and GetObfuscator (compile-time obfuscation). This is the interface Microsoft’s own obfuscator uses — reverse-engineering it is the cleanest way to understand what compile/link-time transformations are actually available.

CreateObfuscator2

Initializes the obfuscator inside the linker, for link-time obfuscation.

// pObfuscatorBase is provided by the linker
// Its base class contains Warbird::CWarbirdLinkerTransformations,
// the interface the linker hands to Warbird for link-time obfuscation.

// Warbird::CWarbirdLinkerTransformations
Warbird::CWarbirdLinkerTransformations::Log(Warbird::LogLevel,ushort const *);
Warbird::CWarbirdLinkerTransformations::GetImageBase(void);
Warbird::CWarbirdLinkerTransformations::SetSectionFlags(char const *,ulong,ulong);
Warbird::CWarbirdLinkerTransformations::GetSectionLocation(char const *,ulong &,ulong &,ulong &,long);
Warbird::CWarbirdLinkerTransformations::GetRelocStartRva(ulong,ushort &,ushort &);
Warbird::CWarbirdLinkerTransformations::GetSymbolRelocsCount(char const *,void *,ulong &,long);
Warbird::CWarbirdLinkerTransformations::GetSymbolLocation(char const *,void *,Warbird::tagWARBIRD_LINK_SYMBOL &,long);
Warbird::CWarbirdLinkerTransformations::ReadBytes(ulong,uchar *,ulong);
Warbird::CWarbirdLinkerTransformations::WriteBytes(ulong,uchar *,ulong);
Warbird::CWarbirdLinkerTransformations::SetSymbolValue(char const *,void *,uchar *,ulong,int);
Warbird::CWarbirdLinkerTransformations::AddSymbolToGuardTable(char const *,void *);
Warbird::CWarbirdLinkerTransformations::ReplaceEntryFunction(char const *,void *,ulong *);
Warbird::CWarbirdLinkerTransformations::GetImageType(Warbird::IMAGE_TYPE *);
Warbird::CWarbirdLinkerTransformations::AddSymboltoTCEGraph(char const *,long,void *);
Warbird::CWarbirdLinkerTransformations::AddSymboltoTCEGraph(char const *,long,char const *,long,void *,void *);
Warbird::CWarbirdLinkerTransformations::GetBinaryFilename(void);
Warbird::CWarbirdLinkerTransformations::CreateSegment(char const *,ulong,ulong,ulong);
Warbird::CWarbirdLinkerTransformations::GetContributionLocation(void *,Warbird::tagWARBIRD_LINK_SYMBOL &);
Warbird::CWarbirdLinkerTransformations::GetRelocations(ulong,ulong,ulong,Warbird::tagWARBIRD_PRIVATE_RELOCATION_ITEM *,ulong &);


int CreateObfuscator2(
    const wchar_t *szWbrdCfgFilename,      // Warbird config file path
    const wchar_t *szWbrdSchemaFilename,   // Warbird schema file path
    const wchar_t *szWbrdLogFilename,      // Warbird log file path
    const wchar_t *szWbrdSummaryFilename,  // Warbird summary file path
    unsigned int nWbrdFlags,               // Warbird flags
    int tArchType,                         // Warbird platform type
    Warbird::CObfuscator **pObfuscatorBase // Obfuscator base class, provided by the linker
);

// After calling CreateObfuscator2, the linker uses pObfuscatorBase through
// CWarbirdClient to drive the Obfuscator it created.

// The linker's CWarbirdClient (slightly different from the one in c2.dll)
Warbird::CWarbirdClient::TransformFunction(Warbird::Phase);
Warbird::CWarbirdClient::CleanPrivateSymbolTable(void);
Warbird::CWarbirdClient::IsEnabled(void);
Warbird::CWarbirdClient::GetInstance(void);
Warbird::CWarbirdClient::UnInitialize(void);
Warbird::CWarbirdClient::Initialize(IMAGE *,ushort const *,ushort const *,ushort const *,int,int);
Warbird::CWarbirdClient::CleanupConSymbolTable(void);
Warbird::CWarbirdClient::FinishProcessNonLtcgModule(void);
Warbird::CWarbirdClient::IsExcludedSymbol(char const *);
Warbird::CWarbirdClient::RegisterSymbol(char const *,int,uchar,CON *,MOD *,ulong,int);

GetObfuscator

Initializes the obfuscator inside c2.dll, for compile-time obfuscation.

// pObfuscatorBase is provided by c2.dll
// Its base class contains Warbird::CC2CodeTransformations,
// the interface c2.dll hands to Warbird for compile-time obfuscation.

// Warbird::CC2CodeTransformations
Warbird::CC2CodeTransformations::GetFunctionInstructionCount(void *);
Warbird::CC2CodeTransformations::GetFunctionBlockCount(void *);
Warbird::CC2CodeTransformations::Log(Warbird::LogLevel,ushort const *);
Warbird::CC2CodeTransformations::FindSymbolReference(char const *,Warbird::tagWARBIRD_SYMBOL &);
Warbird::CC2CodeTransformations::GetSymbolInformation(ulong,Warbird::tagWARBIRD_SYMBOL *);
Warbird::CC2CodeTransformations::GetSymbolSectionID(void *,long *);
Warbird::CC2CodeTransformations::GetSymbolTypeID(char const *,void *);
Warbird::CC2CodeTransformations::ExpandSymbolDataSize(char const *,void *,ulong);
Warbird::CC2CodeTransformations::SetSymbolAttributes(char const *,void *,unsigned __int64);
Warbird::CC2CodeTransformations::SetFunctionAttributes(char const *,void *,ulong,ulong);
Warbird::CC2CodeTransformations::SetFunctionProtectionLevel(char const *,void *,ulong);
Warbird::CC2CodeTransformations::GetFunctionProtectionLevel(char const *,void *);
Warbird::CC2CodeTransformations::GetFunctionBaseSegmentName(void *,char const * *);
Warbird::CC2CodeTransformations::CreateGlobal(char const *,char const *,Warbird::SymbolType,ulong,ulong,ulong,int,ulong *,void * *);
Warbird::CC2CodeTransformations::CreateBaseSegment(char const *,ulong,ulong,Warbird::tagWARBIRD_SYMBOL &);
Warbird::CC2CodeTransformations::CreateSegment(void *,char const *,ulong,ulong,int,void * *);
Warbird::CC2CodeTransformations::MoveSymbolToCodeSegment(void *);
Warbird::CC2CodeTransformations::AssignSegmentForSymbol(void *,void *);
Warbird::CC2CodeTransformations::SetImageBaseSymbol(void *);
Warbird::CC2CodeTransformations::AllocateSegmentForGlobalVariable(void *,Warbird::SymbolType,char const *,ulong,ulong);
Warbird::CC2CodeTransformations::DuplicateSymbolForProxy(void *,char const *,char const *,ulong,ulong,ulong *);
Warbird::CC2CodeTransformations::CheckLiveSymbol(char const *,int *);
Warbird::CC2CodeTransformations::GetFunctionAttributes(void *);
Warbird::CC2CodeTransformations::DoesAssemblyReferenceExternalSymbols(void *);
Warbird::CC2CodeTransformations::ContainsInstruction(char const *,void *,ulong);
Warbird::CC2CodeTransformations::GetFunctionSize(char const *,void *);
Warbird::CC2CodeTransformations::GetFunctionStackSize(char const *,void *,int *);
Warbird::CC2CodeTransformations::GetImageBaseFixupOffset(char const *,void *);
Warbird::CC2CodeTransformations::CountSymbolReferences(char const *,void *,Warbird::tagSYMBOL_REFERENCE *,unsigned __int64);
Warbird::CC2CodeTransformations::GetFirstFunctionBlock(void *,Warbird::tagWARBIRD_BLOCK &);
Warbird::CC2CodeTransformations::GetNextFunctionBlock(void *,Warbird::tagWARBIRD_BLOCK &);
Warbird::CC2CodeTransformations::GetFirstFunctionInstruction(void *,ulong,ulong,Warbird::tagWARBIRD_INSTRUCTION &,void * *);
Warbird::CC2CodeTransformations::GetNextFunctionInstruction(void *,ulong,ulong,Warbird::tagWARBIRD_INSTRUCTION &,void * *);
Warbird::CC2CodeTransformations::FindPreviousSegmentTransitionRuntime(void *,void *,ulong *,ulong *,void * *);
Warbird::CC2CodeTransformations::RemoveUnusedSegmentTransitionInstructions(void *,ulong);
Warbird::CC2CodeTransformations::BuildFlowGraph(char const *,void *);
Warbird::CC2CodeTransformations::MakeLabelGlobal(void *,ulong *,ulong *);
Warbird::CC2CodeTransformations::ExpandSwitches(char const *,void *);
Warbird::CC2CodeTransformations::MakeSingleReturn(char const *,void *);
Warbird::CC2CodeTransformations::InsertFunctionCall(char const *,void *,Warbird::tagWARBIRD_PARAMETER *,ulong,int,void *);
Warbird::CC2CodeTransformations::MakeBlockSegmentStart(void *,char const *,ulong,ulong,ulong *,ulong *,ulong *);
Warbird::CC2CodeTransformations::ReplaceDirectCall(void *,void *,void *,Warbird::tagWARBIRD_PARAMETER *,ulong);
Warbird::CC2CodeTransformations::PopulateVMWrapper(void *,void *,void *);
Warbird::CC2CodeTransformations::PopulateVMWrapperProEpi(char const *,void *);
Warbird::CC2CodeTransformations::ReplaceConditionalBranch(void *,void *,void *,Warbird::tagWARBIRD_PARAMETER *,ulong);
Warbird::CC2CodeTransformations::ReplaceUnconditionalBranch(void *,void *,void *,Warbird::tagWARBIRD_PARAMETER *,ulong);
Warbird::CC2CodeTransformations::InsertFallThrough(void *,void *,void *,Warbird::tagWARBIRD_PARAMETER *,ulong);
Warbird::CC2CodeTransformations::PopulateProxyFunction(char const *,void *,void *);
Warbird::CC2CodeTransformations::MakeAbsolutePreLower(char const *,void *);
Warbird::CC2CodeTransformations::MakeAbsolutePostLower(char const *,void *,Warbird::Phase);
Warbird::CC2CodeTransformations::AddFunctionToICFExclude(char const *,void *);
Warbird::CC2CodeTransformations::AddSymbolToICFExclude(void *);

int GetObfuscator(
    Warbird::CObfuscator **pCObfuscatorBase // Obfuscator base class, provided by c2.dll
);

// After calling GetObfuscator, c2.dll uses pObfuscatorBase through CWarbirdClient
// to drive the Obfuscator it created.

// c2.dll's CWarbirdClient
Warbird::CWarbirdClient::TransformFunction(Warbird::Phase,tagFUNC *);;
Warbird::CWarbirdClient::GetInstance(void);
Warbird::CWarbirdClient::IsProtectedSymbol(Symbol *);
Warbird::CWarbirdClient::IsEnabled(void);
Warbird::CWarbirdClient::RegisterSymbols(void);
Warbird::CWarbirdClient::CanUseRegister(Tuple *,ulong);
Warbird::CWarbirdClient::ConvertToWideString(char const *);
Warbird::CWarbirdClient::GetSymbolFilename(Symbol *);
Warbird::CWarbirdClient::IsWarbirdRuntimeModule(tagMOD *);
Warbird::CWarbirdClient::RegisterEnclaveSymbols(void);
Warbird::CWarbirdClient::RegisterEnumerate(ushort const * const,ushort const * const,uint);
Warbird::CWarbirdClient::RegisterSymbol(Symbol *);
Warbird::CWarbirdClient::RegisterTypeField(Warbird::tagWARBIRD_TYPE_FIELD const &);
Warbird::CWarbirdClient::SetFunctionImageBaseOffset(tagFUNC *,Symbol *,ulong);

A few of the names worth noticing: PopulateVMWrapper / PopulateVMWrapperProEpi suggest Warbird has a virtual-machine wrapper transformation, and ReplaceDirectCall / ReplaceConditionalBranch / ReplaceUnconditionalBranch / InsertFallThrough are the standard control-flow-obfuscation primitives.

Reference resources

Related work found on GitHub:

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch