Tracing Windows Syscalls on 10/11: LSTAR, the SSDT, and Parameter-Count Determination

About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.

This research is published for educational and defensive purposes. The target sample is anonymized throughout.

Most syscall writeups still use 32-bit XP. This one walks the 64-bit Windows 10/11 path instead: how the system-call table is invoked (non-GUI syscalls specifically), how the parameter count is encoded, and how the kernel turns a syscall number into a ring-0 function. Since the SSDT became an unexported object on Windows 10, I also give signature bytes for locating it across 7/10/11.

The syscall instruction

In ring 3, every non-GUI Win32 API eventually reaches ntdll, which holds the ring-0 entry stubs. Each stub ends in a syscall instruction, with the syscall number held in r10:

The Intel manual describes syscall like this:

SYSCALL invokes an OS system-call handler at privilege level 0. It does so by loading RIP from the IA32_LSTAR MSR (after saving the address of the instruction following SYSCALL into RCX). (The WRMSR instruction ensures that the IA32_LSTAR MSR always contain a canonical address.)

So syscall uses the IA32_LSTAR MSR to pick the ring-0 entry point. The manual’s MSR chapter gives its number:

0xc0000082. We can read it in the kernel debugger and resolve the symbol:

0: kd> rdmsr 0xc0000082
msr[c0000082] = fffff807`2dc161c0
0: kd> ln fffff807`2dc161c0
Browse module
Set bu breakpoint

(fffff807`2dc161c0)   nt!KiSystemCall64Shadow   |  (fffff807`2dc17060)   nt!_guard_retpoline_icall_handler

KiSystemCall64Shadow

IA32_LSTAR points at nt!KiSystemCall64Shadow, an unexported function. For some reason ntoskrnl doesn’t load identically every time, so using a static file offset to find the function is unreliable. Instead, search for its signature bytes (disassemble with u or dump with dq, then search the bytes in IDA).

Setting a breakpoint directly at the function’s entry will blue-screen. But breaking at offset 0x3c stops cleanly (no idea why — it just works).

From there, the function eventually leads to KiSystemServiceStart (also located via signature in IDA). But first, note that the breakpoint we just set fires for every syscall number, which makes it hard to track a specific one.

Tracking a specific syscall number

Use the kernel debugger’s:

.process /i _eprocess

This attaches the debugger to a ring-3 process and freezes the whole system while debugging. To analyze a specific syscall number, write a small ring-3 program that calls that API, set a ring-3 breakpoint in the corresponding ntdll function, step to the syscall instruction, then set the breakpoint on KiSystemCall64Shadow and step — now you land on exactly the syscall number you want.

KiSystemServiceStart and KiSystemServiceRepeat

From KiSystemCall64Shadow, execution reaches KiSystemServiceStart, immediately followed by KiSystemServiceRepeat (these names are mine — IDA won’t auto-recognize them; you find them by signature and name them yourself):

The earlier code is mostly setup. From here on is the part that determines the syscall function and its parameter count. First, KiSystemServiceStart stores bit 12 of the service number, shifted left by 4, into edi; bits 1–11 go into eax.

Then KiSystemServiceRepeat loads two tables into r10/r11: KeServiceDescriptorTable and its shadow variant. On Windows 10+ the service-table structure is no longer exported, but we can reconstruct it by extrapolating from the 32-bit layout:

typedef struct _SERVICE_DESCRIPTOR_TABLE
{
    PVOID ServiceTableBase;        // SSDT start address
    PVOID ServiceCounterTableBase; // 8-byte counter of how many times each service was called
    DWORD NumberOfService;         // total number of service functions in the SSDT
    PUCHAR ParamTableBase;         // start of the per-service parameter-count array; each byte = arg count * 4
} _SERVICE_DESCRIPTOR_TABLE;

// KeServiceDescriptorTable and Shadow are composed of the above:
typedef struct _KSERVICE_TABLE_DESCRIPTOR
{
    _SERVICE_DESCRIPTOR_TABLE ntoskrnl;  // SSDT for ntoskrnl
    _SERVICE_DESCRIPTOR_TABLE win32kSys; // SSDT for win32k.sys (GUI)
    _SERVICE_DESCRIPTOR_TABLE notUsed1;
    _SERVICE_DESCRIPTOR_TABLE notUsed2;
} _KSERVICE_TABLE_DESCRIPTOR;

Both KeServiceDescriptorTable and KeServiceDescriptorTableShadow have symbols in WinDbg:

2: kd> dq nt!KeServiceDescriptorTable L8
fffff807`2e0018c0  fffff807`2d2c7b30 00000000`00000000
fffff807`2e0018d0  00000000`000001d9 fffff807`2d2c8298
fffff807`2e0018e0  00000000`00000000 00000000`00000000
fffff807`2e0018f0  00000000`00000000 00000000`00000000
2: kd> dq nt!KeServiceDescriptorTableShadow L8
fffff807`2defca40  fffff807`2d2c7b30 00000000`00000000
fffff807`2defca50  00000000`000001d9 fffff807`2d2c8298
fffff807`2defca60  fffffa8b`38289000 00000000`00000000
fffff807`2defca70  00000000`00000526 fffffa8b`3828a9c4

The first _SERVICE_DESCRIPTOR_TABLE is identical in both; the second differs — the shadow table’s second slot is populated with win32k.sys. Normal user processes use the shadow table.

Below KiSystemServiceRepeat a branch picks the table. It reads the ThreadFlag member at offset 0x78 of the KTHREAD: if the GuiThread bit is set it picks the shadow table; if the RestrictGuiThread bit is set it picks another table, KeServiceDescriptorTableFilter. Dig into the selection logic yourself if you’re curious.

Then the left branch uses the syscall number (in eax) to pick the function. ServiceTableBase doesn’t store real function pointers — it stores a signed offset from the table base, at most a DWORD. The arithmetic makes that clear:

movsxd  r11, dword ptr [r10+rax*4] ; one entry in tableBase — call it fn
mov     rax, r11
sar     r11, 4                      ; fn >> 4
add     r10, r11                    ; actual function address = tableBase + (fn >> 4)

So the called function is tableBase + tableBase[syscall_number] >> 4. Here’s what the table entries look like:

2: kd> dq nt!KeServiceDescriptorTableShadow L4
fffff807`2defca40  fffff807`2d2c7b30 00000000`00000000
fffff807`2defca50  00000000`000001d9 fffff807`2d2c8298
2: kd> dd fffff807`2d2c7b30
fffff807`2d2c7b30  027fd004 02495b00 05d33a02 080f2c00
fffff807`2d2c7b40  05761e00 033a1600 0601dd05 05839706
fffff807`2d2c7b50  06013d05 051c2001 0599b600 0522e500
fffff807`2d2c7b60  05166100 05828500 059a7c00 05763300
fffff807`2d2c7b70  0647cf01 05331301 05afb800 052cb802
fffff807`2d2c7b80  05533c00 05bc2b00 057df201 057d4902
fffff807`2d2c7b90  05329d02 05599501 05b05101 05b6cf05
fffff807`2d2c7ba0  058fdf00 06205e03 0572ee00 07fa3000

The reason for the >> 4 is that the low 4 bits encode the parameter count. Let’s see how those bits are used.

Determining the parameter count

Unlike 32-bit XP, which used a separate table for parameter counts, 64-bit Windows 10 encodes the count in the low 4 bits of each SSDT entry. The branch above has a jnz that separates ordinary syscalls from win32k.sys syscalls (edi holds bit 12 of the call number). For ordinary syscalls this jump is taken:

We already put the function-offset entry into rax, so after & 0xf only the low 4 bits remain in eax.

Fewer than 4 parameters (no stack arguments)

If eax == 0 — the function uses only register arguments or none — we reach KiSystemServiceCopyEnd:

Then this branch, where call rax indirectly invokes the syscall function:

With parameters

With parameters, execution takes the other branch, which uses the low 4 bits to jump to some offset above KiSystemServiceCopyEnd:

Look just above KiSystemServiceCopyEnd:

This is where stack arguments are handled. Tracing back, rsi and rdi hold ring-3 stack + 0x18 and ring-0 stack + 0x18 respectively (I’ve forgotten where they’re assigned, so no demo here). Combined with the jump:

lea     r11, KiSystemServiceCopyEnd
sub     r11, rax        ; rax = ssdtEntry[:4] << 3
jmp     r11             ; jmp KiSystemServiceCopyEnd - 8*ssdtEntry[:4]

If the low 4 bits are non-zero, execution jumps to KiSystemServiceCopyEnd - 8*ssdtEntry[:4]. The code above that address copies parameters to rdi in 8-byte units. So a syscall needing one stack argument jumps to KiSystemServiceCopyEnd - 8 and runs:

48 8B 46 08                mov     rax, [rsi+8]    ; rax = [r3_rsp+0x18+8] -> [r3_rsp+0x20]
48 89 47 08                mov     [rdi+8], rax    ; [r0_rsp+0x20] = rax <-> [r3_rsp+0x20]

Two stack arguments jump to KiSystemServiceCopyEnd - 0x10:

48 8B 46 10                mov     rax, [rsi+10h]  ; rax = [r3_rsp+0x18+0x10] -> [r3_rsp+0x28]
48 89 47 10                mov     [rdi+10h], rax  ; [r0_rsp+0x28] = rax <-> [r3_rsp+0x28]
48 8B 46 08                mov     rax, [rsi+8]    ; rax = [r3_rsp+0x18+8] -> [r3_rsp+0x20]
48 89 47 08                mov     [rdi+8], rax    ; [r0_rsp+0x20] = rax <-> [r3_rsp+0x20]

And so on — that’s how stack parameters are passed. The first four parameters live in registers, so they’re backed up and restored with the usual push/pop.

Signature bytes for the SSDT on 7/10/11

On the 64-bit builds of Windows 7, 10, and 11 there’s one universal signature: 8b f8 c1 ef 07 83 e7 20 25 ff 0f 00 00 4c 8d. Search nt for it: the byte starting after the first 4c 8d is the offset to KeServiceDescriptorTable relative to this address + 7; after the second 4c 8d is the offset to KeServiceDescriptorTableShadow relative to this address + 7.

The signature is just the first few instructions of KiSystemServiceStart. On x64 the function opens like this:

KiSystemServiceStart:
48 89 A3 90 00 00 00       mov     [rbx+90h], rsp   ; may vary across kthread layouts — not part of the signature
8B F8                      mov     edi, eax
C1 EF 07                   shr     edi, 7
83 E7 20                   and     edi, 20h
25 FF 0F 00 00             and     eax, 0FFFh
4C 8D 15 XX XX XX XX       lea     r10, KeServiceDescriptorTable
4C 8D 1D XX XX XX XX       lea     r11, KeServiceDescriptorTableShadow

The signature runs from mov edi, eax through the two lea instructions. Those leas load KeServiceDescriptorTable and KeServiceDescriptorTableShadow with hard-coded absolute addresses — so finding them and decoding the displacement gives you both table addresses directly.

Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).

Get in touch