Page-Granular Dynamic Decryption and Lazy Re-encryption in a Windows Packer

About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

This builds on a previous post where I set up a basic packer framework. After that I added some simple static anti-debugging. This post records how I implemented page-granular dynamic decryption and lazy re-encryption.

Requirements and feasibility

Previously, the stub decrypted the entire .text section in one shot. That’s weak against dynamic analysis: once decrypted, the code sits in memory as plaintext and is trivially dumped.

To strengthen the packer, I need to decrypt .text in multiple passes, and re-encrypt less frequently used fragments so they can be decrypted again on demand. That way, complete plaintext code never exists in memory at once, and the protected program gets better coverage.

For dynamic decryption I need a mechanism that pauses execution when it hits encrypted code and hands control to the decryption routine.

The OS paging mechanism together with Windows VEH fits the bill. We tie the code’s state (encrypted or decrypted) to illegal memory access, and handle the resulting exceptions in a VEH handler to implement dynamic decryption and encryption.

VEH is used over SEH because it’s simpler and doesn’t depend on stored unwind info like pdata, which simplifies the stub design.

Page-granular dynamic decryption

Basic framework

The idea:

Split .text into pages and set every page to PAGE_NOACCESS
Register a VEH handler for illegal memory access
The stub runs the original entry point, which triggers an access violation and invokes the VEH callback
The VEH handler decrypts as needed, then returns EXCEPTION_CONTINUE_EXECUTION
The program keeps running; each further violation repeats step 4

That’s a very simple strategy. The core code:

// 安装VEH
PVOID hHandler = g_MyAddVectoredExceptionHandler(1, MyVEH);

// 用于触发异常
DWORD dwOld = 0;
g_MyVirtualProtect((PVOID)sg_ullTextStart, g_param.dwTextSize, PAGE_NOACCESS, &dwOld);

// 执行原始入口点
if (g_param.dwEP && g_ImageBase)
{
    typedef void (*FUNC)();
    // 必然引发访问异常，在VEH中动态解密
    FUNC ep = (FUNC)(g_param.dwEP + g_ImageBase);
    ep();
}

MyVEH must check that the exception is an access violation and that the faulting address lies inside .text:

LONG MyVEH(PEXCEPTION_POINTERS pExceptionInfo)
{
    if (pExceptionInfo->ExceptionRecord->ExceptionCode == EXCEPTION_ACCESS_VIOLATION) // 是否为访问异常
    {
        void *faultAddr = (void *)pExceptionInfo->ExceptionRecord->ExceptionAddress;

        // 发生异常的地址是否在text节区中
        if ((ULONGLONG)faultAddr >= sg_ullTextStart && (ULONGLONG)faultAddr < sg_ullTextEnd)
        {
            // 解密
            DecryptTextPage(faultAddr);
        }

        return EXCEPTION_CONTINUE_EXECUTION;
    }

    return EXCEPTION_CONTINUE_SEARCH;
}

DecryptTextPage does the decryption, then restores the page to readable/executable:

DWORD dwOld = 0;
BOOL res = g_MyVirtualProtect((PVOID)faultAddr, g_param.dwTextSize, PAGE_EXECUTE_READWRITE, &dwOld);
if (!res)
{
    g_MyExitProcess(1);
}

// 解密text
PBYTE ch = (PBYTE)sg_ullTextStart;
for (size_t i = 0; i < g_param.dwTextSize; i++)
{
    ch[i] ^= KEY;
}

res = g_MyVirtualProtect((PVOID)sg_ullTextStart, g_param.dwTextSize, PAGE_EXECUTE_READ, &dwOld);
if (!res)
{
    g_MyExitProcess(1);
}

The hard problem: instructions crossing a page boundary

If you pack a program with just the framework above, some programs run and some don’t.

Debugging the VEH handler shows the faults always happen near the end of a page. The reason: if an instruction straddles the end of one page and the start of the next, it gets split into two halves — an already-decrypted front half and an encrypted back half. If the front half happens to decode as a valid instruction, the CPU executes that wrong instruction before the next page’s decryption can be triggered.

That has a serious consequence: it corrupts the thread context. The CPU executes a wrong instruction, and even after the next page is decrypted, the bytes it reads back are no longer the start of the original instruction.

Cross-page instruction problem

Solving cross-page instructions

The core conflict is between the fixed page size and the variable length of amd64 instructions. When decrypting, we can’t just decrypt the faulting page — we also have to decrypt a few bytes at the head of the next page, so that even a straddling instruction won’t be split into a decrypted front and an encrypted back.

To do that we set the second page to readable, so the CPU doesn’t fetch garbage. But when the CPU tries to execute the straddling instruction, it still hits the access violation, and the faulting address still belongs to the first page. So the VEH repeatedly checks the first page, finds it already decrypted, returns, the program re-executes the straddling instruction, hits the violation again, and the VEH tries to decrypt the first page again — an infinite loop.

The infinite loop

We can design a mechanism: if a page enters the VEH handler repeatedly above a threshold, decrypt its next page and make it executable. That breaks the loop.

Putting the pieces together, I use a structure to track page state:

struct PAGE_INFO
{
    ULONGLONG ullPageBase;
    bool bHeadTag; // 头部15字节是否已解密
    bool bDecrypted; // 页是否解密

    // 当前页错误的次数，如果是3次以上，说明有指令跨页了，此时解密下一页并给与其执行权限
    WORD wFaultCount;
};

Each page has one structure object, managed in an array.

The updated decryption flow:

static void DecryptTextPage(PVOID faultAddr)
{
    // 计算异常地址所在的页起始地址
    ULONGLONG ullFaultPageStart = (ULONGLONG)faultAddr & ~(PAGE_SIZE - 1);
    ULONGLONG ullFaultPageEnd = (ULONGLONG)ullFaultPageStart + PAGE_SIZE - 1;
    ULONGLONG ullNextPageStart = ullFaultPageEnd + 1;
    ULONGLONG ullNextPageEnd = ullNextPageStart + PAGE_SIZE; // 注意处理最后一个页面的情况

    // 定位页对应的PAGE_INFO
    DWORD dwIndex = 0;
    bool finded = false;
    for (dwIndex; dwIndex < sg_dwTextPageNums; dwIndex++) // 这里的查找可以优化，因为ullPageBase是递增的
    {
        if (sg_pPageInfo[dwIndex].ullPageBase == ullFaultPageStart)
        {
            finded = true;
            break;
        }
    }

    sg_pPageInfo[dwIndex].wFaultCount++;
    // =======================这里说明有指令跨页了
    if (sg_pPageInfo[dwIndex].wFaultCount > 3)
    {
        DecryptTextPage((PVOID)sg_pPageInfo[dwIndex + 1].ullPageBase);
        sg_pPageInfo[dwIndex].wFaultCount = 0; // 记得清零，否则所有页都要不断进行扩展解密
        return;
    }

    // =======================更新活跃度，当前页和下一页都更新
    sg_pPageInfo[dwIndex].dwLastDecryptTime = g_MyGetTickCount();
    sg_pPageInfo[dwIndex + 1].dwLastDecryptTime = g_MyGetTickCount();

    // 修改页面权限
    DWORD dwOld = 0;
    BOOL res = g_MyVirtualProtect((PVOID)ullFaultPageStart, PAGE_SIZE, PAGE_EXECUTE_READWRITE, &dwOld);
    if (!res)
    {
        g_MyExitProcess(1);
    }

    // 开始解密流程
    if (!sg_pPageInfo[dwIndex].bDecrypted) // 当前页未解密
    {
        // 解密text
        PBYTE ch = (PBYTE)ullFaultPageStart;
        if (sg_pPageInfo[dwIndex].bHeadTag) // 当前头部已解密，跳过头部
        {
            ch += HEAD_SIZE; // 调整指针跳过头部
            for (size_t i = 0; i < PAGE_SIZE - HEAD_SIZE; i++)
            {
                ch[i] ^= KEY;
            }
        }
        else // 当前头部未解密，则解密当前整个页
        {
            for (size_t i = 0; i < PAGE_SIZE; i++)
            {
                ch[i] ^= KEY;
            }
            sg_pPageInfo[dwIndex].bHeadTag = true;
        }
        sg_pPageInfo[dwIndex].bDecrypted = true;
    }

    // 修改为正常权限
    res = g_MyVirtualProtect((PVOID)ullFaultPageStart, PAGE_SIZE, PAGE_EXECUTE_READ, &dwOld);
    if (!res)
    {
        g_MyExitProcess(1);
    }

    // 检查下一页面是否还在text中
    if (ullNextPageStart >= sg_ullTextStart && ullNextPageEnd <= sg_ullTextEnd)
    {
        res = g_MyVirtualProtect((PVOID)ullNextPageStart, PAGE_SIZE, PAGE_EXECUTE_READWRITE, &dwOld);
        if (!res)
        {
            g_MyExitProcess(1);
        }

        // 检查下一页面头部是否解密
        if (!sg_pPageInfo[dwIndex + 1].bHeadTag) // 下一页头部未解密，进行解密
        {
            PBYTE c = (PBYTE)ullNextPageStart;
            for (size_t i = 0; i < HEAD_SIZE; i++)
            {
                c[i] ^= KEY;
            }
            sg_pPageInfo[dwIndex + 1].bHeadTag = true; // 设置头部标记
        }

        res = g_MyVirtualProtect((PVOID)ullNextPageStart, PAGE_SIZE, PAGE_READONLY, &dwOld);
        if (!res)
        {
            g_MyExitProcess(1);
        }
    }
}

With that, the cross-page instruction problem is solved.

Lazy re-encryption

Once the page-granular dynamic decryption framework exists, lazy re-encryption is easy.

We can spawn a thread and use a timer kernel object to re-encrypt periodically or on schedule, guarding the decryption flow against races with a critical section or other synchronization.

There’s an even simpler approach: add a dwLastDecryptTime field to the page-info structure:

struct PAGE_INFO
{
    ULONGLONG ullPageBase;
    bool bHeadTag; // 头部15字节是否已解密
    bool bDecrypted;

    // 当前页错误的次数，如果是3次以上，说明有指令跨页了，此时解密下一页并给与其执行权限
    WORD wFaultCount;

    // 最后一次解密时间
    DWORD dwLastDecryptTime;
};

Before each encryption pass, scan the page-info array and encrypt the pages with the lowest activity:

// 遍历页面信息数组，如果页面的活跃度小于设定的阈值，则加密该页面
static void EncryptTextPage(PVOID pCurrentPage)
{
    // 以毫秒为单位
    const DWORD dwThreshold = 7000;
    DWORD dwCurentTime = g_MyGetTickCount();
    for (size_t i = 0; i < sg_dwTextPageNums; i++)
    {
        if (sg_pPageInfo[i].ullPageBase != (ULONGLONG)pCurrentPage)
        {
            if (dwCurentTime - sg_pPageInfo[i].dwLastDecryptTime <= dwThreshold)
            {
                // g_MyMessageBoxA(nullptr, AddressToString((PVOID)sg_pPageInfo[i].ullPageBase),
                //   "EncryptTextPage", MB_OK);
                PBYTE ch = (PBYTE)sg_pPageInfo[i].ullPageBase;
                if (sg_pPageInfo[i].bHeadTag) // 头部已解密，连同头部一起加密
                {
                    for (size_t i = 0; i < PAGE_SIZE; i++)
                    {
                        ch[i] ^= KEY;
                    }
                    sg_pPageInfo[i].bHeadTag = false;
                }
                else // 头部未解密，则头部不需要加密
                {
                    ch += HEAD_SIZE; // 调整指针跳过头部
                    for (size_t i = 0; i < PAGE_SIZE - HEAD_SIZE; i++)
                    {
                        ch[i] ^= KEY;
                    }
                }
                sg_pPageInfo[i].bDecrypted = false;

                DWORD dwOld = 0;
                g_MyVirtualProtect((PVOID)sg_pPageInfo[i].ullPageBase, PAGE_SIZE, PAGE_NOACCESS, &dwOld);
            }
        }
    }
}

This approach needs a sensible threshold.

Demo

After running the packed program for a while and inspecting .text, you can see pages that remain undecrypted:

Undecrypted pages in .text

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch