Breaking VMProtect's Local License Lock: RSA, Big-Number Encryption, and a Working Keygen
About the author. I’m Colin Xiao — an independent reverse engineer, mostly working solo on contract jobs. I dig into compiled code of every kind, including but not limited to: Android and iOS apps, x86/x64/ARM native binaries, packers and obfuscators, DRM and license checks, network protocols and APIs, crypto and signature algorithms, malware, game binaries, and embedded/IoT firmware. I also handle large-scale data collection and vulnerability research. If you have any project in mind, I’d be glad to take it on — whatever it is, as long as it’s within my wheelhouse. Just reach out. You can reach me on X at @colinxiao_lab or on Telegram at https://t.me/ColinXiao666.
This research is published for educational and defensive purposes. The target sample is anonymized throughout.
Breaking VMProtect’s Local License Lock: RSA, Big-Number Encryption, and a Working Keygen
A note up front: the third-party network-verification product referenced in the original sample isn’t its own protector — it’s VMProtect under the hood. This article does not analyze or break that product’s verification layer; the target here is VMProtect’s local license lock only.
Introduction
VMProtect’s local license lock ships with VMProtect’s own VM protection. You lock protected code to a serial number and optionally layer on expiry or run-time limits, and you get a modest amount of anti-cracking for free. The locked code needs a valid serial to decrypt at all; patching it just means adding more protected markers to raise the attacker’s effort. It’s a simple, effective anti-unpacking and anti-cracking scheme.
But everything hinges on VMProtectSetSerialNumber. Once its flow is understood — and a keygen exists — no amount of extra virtualized markers helps. Two cracking approaches dominate: patch the RSA modulus and keygen yourself, or rewrite the decrypted result at the right moment. Let me walk through both.
Preparation
Tools: x64dbg, and VMProtect 3.5–3.8 for the test binaries. The sample is a 64-bit PE with a local license lock.
The SDK prototypes I used:
int VMP_API VMProtectSetSerialNumber(const char *serial);
void VMP_API VMProtectBeginUltraLockByKey(const char *);The test program just calls VMProtectSetSerialNumber, then protects another function with VMProtectBeginUltraLockByKey:
void Test_Lic()
{
VMProtectBeginUltraLockByKey("lock");
cout << "LockByKey" << endl;
VMProtectEnd();
}
void Test_VMP()
{
string serial;
VMProtectSerialNumberData data;
ifstream ifile("./test.key", ios::in | ios::binary);
if (ifile.is_open())
{
ifile >> serial; // reads a single line, so don't put a newline in the serial
cout << "SetSerial Status: " << VMProtectSetSerialNumber(serial.c_str()) << endl;
if (VMProtectGetSerialNumberData(&data, sizeof(data)))
{
cout << "State: " << data.nState << endl; // state
wcout << L"Username: " << data.wUserName << endl; // username
wcout << L"EMail: " << data.wEMail << endl; // email
cout << "Expire: " << (int)data.dtExpire.wYear << (int)data.dtExpire.bMonth << (int)data.dtExpire.bDay << endl; // expiry date
cout << "MaxBuild: " << (int)data.dtMaxBuild.wYear << (int)data.dtMaxBuild.bMonth << (int)data.dtMaxBuild.bDay << endl; // max build date
cout << "RunningTime: " << (int)data.bRunningTime << endl; // allowed runtime in minutes
cout << "UserDataLength: " << (int)data.nUserDataLength << endl; // length of custom user data
if (data.nUserDataLength)
{
cout << "UserData: " << (char*)data.bUserData << endl; // custom user data
}
}
}
Test_Lic();
system("pause");
}Compile it, pack with VMProtect 3.8, and set the key length to 4096. Packing, anti-debug, and anti-VM are beside the point here, so I skipped them and only protected the functions.
Then I generated a serial with the keygen VMP shipped in its 2.x releases: export the key pair and paste it into the keygen source. Strip the newline from the generated serial and write it to test.key. The keygen already tells us the serial is RSA, so cracking means either replacing the modulus or rewriting the decryption result — either way we need a valid serial to run and decrypt first. Knowing it’s RSA, the goal is to find the public key and modulus.
Key decryption and the big-number storage cipher
Attach x64dbg, find VMProtectSetSerialNumber, set a breakpoint, and the serial shows up immediately.
That spot is usually hidden behind virtualized watermarks, though. When it is, break on RtlEnterCriticalSection instead and watch the stack and registers for the key — below you can see the serial at rdx and rsp+78. The 32-bit path is the same idea but you’d watch ecx and ebp.
Step on, break on RtlAllocateHeap, and run. The first stop:
0x2AC is the serial’s length; this buffer will hold the base64-decoded serial. Run to retn, note the returned address, continue — and the decrypted result matches a plain base64 decode.
The second allocation is 0x202 bytes — 0x200 being the RSA-4096 data length — and it converts that base64 result into VMP’s internal big-number structure. Run to retn, note the address, continue. The data is there but encrypted; set a hardware write breakpoint before it’s written and trace through to recover the cipher.
I’ll skip the step-by-step trace — it’s grunt work — and go straight to the conclusion: VMP encrypts big numbers using the buffer’s memory address plus a random 20-byte salt. The salt lives on the stack; search the stack for the buffer address and the salt sits right after it.
This run’s salt was 6D 26 75 F5 3C D2 7D DA AE 9F 95 F3 79 60 1B 39 8B 66 3F 77 — random, so only valid for this run. The decrypted result (the algorithm is given further down):
00 01 69 71 63 7D 93 5F FF 4C E3 5D 49 AF 43 E2
0F 98 D9 32 67 61 41 14 99 93 01 0A 89 19 50 72
CB 15 E5 AC 3A B4 8A 55 3A 4B 71 08 F2 27 B2 62
4D 72 EB 28 4F 1E 67 DF A6 9E 8E CA FC 41 CA 97
D8 4C 4E 36 A5 39 42 00 1C F6 04 3C CD 8D 69 DB
5D 58 33 7B D2 D7 51 DB 67 5D B4 72 72 6A F8 3F
F1 DB 8D 87 64 F5 56 AA 61 F9 3C 73 26 6C 2D 15
A0 9D 3A B8 A5 CF 50 A2 80 47 75 5A 07 91 2B 4B
0E 29 02 26 D8 90 18 E9 5E C9 23 C2 F1 1A A6 88
4B 3D 8C 68 49 4F E1 1A 09 D3 84 27 3C 85 A7 CF
A1 08 A7 D9 76 63 C8 35 CD C5 87 E4 25 03 44 27
AA 1C 16 B7 79 B7 7D AF 7E 30 31 5E 67 00 43 02
C5 11 BB 93 F7 A6 2F DF B7 B3 65 38 32 64 68 56
B1 73 A8 BB B6 5C 99 02 47 4D A5 85 D4 D1 A7 A4
92 C0 73 5F 3A 78 2F CC 60 FD 2D C3 B7 8C 51 1F
07 DB DC 5F 44 DE CA FE 76 86 AA 1E 12 0B 15 BA
E6 66 10 A7 51 13 77 F4 76 27 AD 92 84 8C 2A 1E
00 C9 50 B3 74 0D DA AB 38 E5 A5 51 F6 8D A1 8F
D5 EB C4 DE 36 C8 A4 22 95 AA F5 2C AE FF 48 48
2A B1 78 37 3B 5F 3D FA F8 B8 7A 3A FD 5F B3 29
08 93 5F F6 25 05 EF CB 77 56 30 D3 70 11 C3 1C
27 76 5B 17 0F CB 5E 9D 76 5F 88 C4 7B 29 64 27
D5 27 5F 30 87 AC F3 F6 3E D7 BB 4C 82 2E 83 F8
12 92 65 19 94 7B 2E CA 2E 6D FA A3 68 97 13 BA
A3 6C 76 D8 5E 59 67 2D 72 E5 AF 5B E6 33 0F A1
1B F6 7B A2 6F 39 7D 38 D0 3A DE E6 B9 06 88 0C
39 BC 22 95 B6 51 D4 C9 B8 2B 81 7C 02 F1 60 58
7E 4A 20 F6 EA 01 4D DB 2C BB F5 25 25 2B A8 9F
00 77 FD 73 42 B4 26 14 57 8F C7 2F 46 5F 55 7C
6B E4 73 84 11 2D CC 0F B0 7D 65 30 3C E1 84 83
1E E5 91 A7 B4 DA 6A 4E 96 2E B8 97 35 50 A5 E4
F8 94 C4 43 32 9C 39 2B EF 28 AC CF 83 6C 0C 90
60 45The third RtlAllocateHeap allocates 0xC04 bytes for the public key (0x4; custom keys are allowed but rarely used), the modulus (0x200), the message (the base64 result, 0x200), and two buffers (0x400 × 2). The order is randomized per run. Run to retn, note the address, continue, then read and decrypt the data — this allocation is effectively the end of decryption. (64-bit allocations are 0x20; 32-bit are 0x10.)
Extract and decrypt, using the same salt as before but this buffer’s own address as the key. Tidied up:
msg:
60 45 0C 90 83 6C AC CF EF 28 39 2B 32 9C C4 43
F8 94 A5 E4 35 50 B8 97 96 2E 6A 4E B4 DA 91 A7
1E E5 84 83 3C E1 65 30 B0 7D CC 0F 11 2D 73 84
6B E4 55 7C 46 5F C7 2F 57 8F 26 14 42 B4 FD 73
00 77 A8 9F 25 2B F5 25 2C BB 4D DB EA 01 20 F6
7E 4A 60 58 02 F1 81 7C B8 2B D4 C9 B6 51 22 95
39 BC 88 0C B9 06 DE E6 D0 3A 7D 38 6F 39 7B A2
1B F6 0F A1 E6 33 AF 5B 72 E5 67 2D 5E 59 76 D8
A3 6C 13 BA 68 97 FA A3 2E 6D 2E CA 94 7B 65 19
12 92 83 F8 82 2E BB 4C 3E D7 F3 F6 87 AC 5F 30
D5 27 64 27 7B 29 88 C4 76 5F 5E 9D 0F CB 5B 17
27 76 C3 1C 70 11 30 D3 77 56 EF CB 25 05 5F F6
08 93 B3 29 FD 5F 7A 3A F8 B8 3D FA 3B 5F 78 37
2A B1 48 48 AE FF F5 2C 95 AA A4 22 36 C8 C4 DE
D5 EB A1 8F F6 8D A5 51 38 E5 DA AB 74 0D 50 B3
00 C9 2A 1E 84 8C AD 92 76 27 77 F4 51 13 10 A7
E6 66 15 BA 12 0B AA 1E 76 86 CA FE 44 DE DC 5F
07 DB 51 1F B7 8C 2D C3 60 FD 2F CC 3A 78 73 5F
92 C0 A7 A4 D4 D1 A5 85 47 4D 99 02 B6 5C A8 BB
B1 73 68 56 32 64 65 38 B7 B3 2F DF F7 A6 BB 93
C5 11 43 02 67 00 31 5E 7E 30 7D AF 79 B7 16 B7
AA 1C 44 27 25 03 87 E4 CD C5 C8 35 76 63 A7 D9
A1 08 A7 CF 3C 85 84 27 09 D3 E1 1A 49 4F 8C 68
4B 3D A6 88 F1 1A 23 C2 5E C9 18 E9 D8 90 02 26
0E 29 2B 4B 07 91 75 5A 80 47 50 A2 A5 CF 3A B8
A0 9D 2D 15 26 6C 3C 73 61 F9 56 AA 64 F5 8D 87
F1 DB F8 3F 72 6A B4 72 67 5D 51 DB D2 D7 33 7B
5D 58 69 DB CD 8D 04 3C 1C F6 42 00 A5 39 4E 36
D8 4C CA 97 FC 41 8E CA A6 9E 67 DF 4F 1E EB 28
4D 72 B2 62 F2 27 71 08 3A 4B 8A 55 3A B4 E5 AC
CB 15 50 72 89 19 01 0A 99 93 41 14 67 61 D9 32
0F 98 43 E2 49 AF E3 5D FF 4C 93 5F 63 7D 69 71
pub:
01 00 01 00
tmp1:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
// 0x200 zeros, some elided
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
02 00 2F E2 20 25 86 8C 52 9D 1C 60 F2 B1 E4 56
C3 30 01 00 02 01 4A 08 68 6F 20 6E 6F 44 03 65
6A 0C 68 6F 40 6E 6F 64 2E 65 6F 63 04 6D 00 10
00 00 00 00 00 00 00 00 00 00 00 00 00 00 07 00
A4 76 A3 A8 83 89 F8 00 33 FF 2F 31 DD 77 D2 3D
5B FD CF 77 08 A7 39 DC CB 08 E2 24 02 8A 56 B5
42 E2 23 E0 9E C4 F9 3B 67 8D ED 4F 70 9E CA 75
E6 5E 58 D2 A6 86 7E EC 33 C5 96 4E 34 58 6F A3
1A 1A B8 5C 53 B5 F9 9E 2C F7 2F AB CC 69 31 02
6A AF A2 D9 17 DE 5A BB 53 41 22 E8 C2 2B 41 3D
2F 71 17 45 34 9E 55 45 BC 61 8E 50 35 EE A0 3B
1D 40 4F 09 4C ED FC 1A 9F 8B C9 3E 6B CD 31 6C
52 59 49 CC 6D 79 C8 CF E5 39 FC 77 62 7B C7 1A
68 A3 D6 C4 28 6F AA FB BE F6 AA 77 5E 90 39 BB
84 05 2F 08 9A 4A A0 05 29 35 41 40 7C E0 6A BE
E7 2F 00 56 04 27 BC F7 0B 1A 31 A6 40 47 E5 DD
FE 6E 7B 71 CF 14 3F CE 3A 4F 73 2C 25 75 61 6C
6F FC EA E6 2B 5C 9E 4A 92 51 21 41 2C C7 58 28
AC D0 38 65 91 5F 1B 3B 02 44 FE 22 EE 92 15 88
04 6B 07 07 5D 59 D0 D3 98 BB 10 1E AC 70 46 91
2D 2C 3F 26 5A B8 BF FA 9C 93 A5 6C DE 95 87 9F
DD 9C 9C 2A CF 6C 66 18 93 3B 6B 79 44 9F 78 39
54 44 41 58 97 B0 C9 E4 D9 05 7C 39 72 61 4A 60
EB 76 C3 26 29 67 89 3E 2D 78 EF F4 E7 B8 CD DE
A9 A1 C0 06 AC DD 6D 73 3F A0 EA 9B 97 5A 05 9B
CF 23 EA 38 86 A2 76 13 45 DC B0 14 7F A3 35 67
E8 91 18 1D EE D5 ED C2 06 33 AF 8A B1 6A F2 CA
5D 11 D3 7F 77 78 FA 86 6D 99 15 40 69 4C B5 DA
80 A2 70 EC A7 38 A8 96 99 CD 5A DD 97 7E E3 87
20 F2 94 FE 80 46 FD AC 6C 9F 57 44 76 21 E7 68
94 A9 CB AD 16 A3 8A 53 1A C5 CF 56 14 89 BA 91
54 3D 96 9D 98 70 77 3B BB 28 06 D1 E9 97 79 5F
mod:
97 D7 00 1D 34 D4 36 1D 09 D8 21 F4 A1 35 AB 06
28 9F 9D 94 53 C1 A7 4E 00 1C 22 75 34 DF 3C B7
A9 17 31 97 63 B3 16 22 1F 0D 9F 80 2F 43 BE 39
84 62 13 5C 33 32 3F FC 6A A8 AC 0D 67 F2 F2 EE
4A A4 EA 83 04 29 28 7C D9 7D 2D B8 F5 BC AE 89
D2 84 70 22 EC 62 70 C4 E0 75 44 83 2A E9 2B B9
0B 72 C7 72 15 BB E1 C2 BF AE 27 65 40 BF 6E 7C
11 14 49 E6 1D A3 B8 90 E9 4B 55 A2 96 67 B6 E5
15 E0 55 BC 0D 55 F4 10 5F AF 6E BE A4 D8 24 5E
C3 57 8A 7E 72 2E CC 8B AB 6B C1 EF 40 8A 16 00
A2 54 52 BD C0 26 95 2B 5D 0C DF D2 4F C0 1D 30
11 D6 56 6B 52 08 CA DD 6C 38 57 F4 6C 16 3A 4C
6C BC 46 16 F5 39 90 C3 49 6B E6 B0 EB 2D 6B 75
09 0C FE 41 EC 60 CD 93 73 44 61 E7 C0 17 04 19
CC C6 2A F5 64 2B EE CA 37 03 11 9F 2A 9A C3 F7
DB 1B 3C 5A E6 80 B3 24 A6 C8 D6 44 92 AD 66 53
EA DB 65 7E 16 EA 3E 20 66 6B 4E B3 FC 9F F7 3D
1F 41 68 B0 53 3F 94 70 7C 53 25 DD 89 72 D2 D0
86 25 9F 5E BA 06 46 9A 5F 59 FA 51 FA 0D 28 5E
90 33 12 CB 9E 36 5D 31 F9 4F 9F 8E 63 17 C7 36
AA 2C 07 5A 2A FE 88 1B B7 43 55 CF FD 92 C5 C8
AA 3F 50 B1 EB 17 2A 18 89 0B 47 1E EC EA E9 0D
8D BD 1A 78 B8 98 5F B2 3F B6 29 6B 19 D2 6D 9F
10 98 C1 F2 AD C1 6D D4 C2 97 39 E8 E4 63 E6 00
FD 68 D7 46 87 28 0A 7C 2D D9 71 C9 54 F8 7B DB
8F 71 08 DF B2 A5 9C F4 FD 39 08 D7 6F DD B8 46
22 EB FC DC A3 A6 55 B4 3A 72 A0 E7 F3 D5 33 8C
FB D0 37 F8 10 27 F0 49 C0 43 80 46 E4 B0 AE A1
D1 BD A9 E7 57 5C 61 81 07 45 96 14 40 07 3D 49
E0 EB 84 35 10 8B 9D E1 53 7D CD F5 CA B1 E2 45
68 D8 2D 51 9A E0 74 DE 90 16 D5 F7 73 3C 0F C0
9C 70 9A 6B 15 D5 BE D0 B0 D7 B9 A4 65 51 49 16
tmp2:
// no use found, elided; the first 0x200 are all zeroFrom the msg decryption you can see the data is byte-swapped in two-byte pairs, so adjust for that.
The layout is scrambled, but pub is usually 0x10001 and doesn’t need extracting. tmp1 is the important one — the VMPSerialData, the decrypted serial — and it can be located by its 0x200 zeros plus the 00 02 marker. tmp2 has no apparent use; it also has 0x200 zeros but no 00 02, so skip it. msg can be excluded by plaintext search, and what’s left is the modulus. Extraction done, we end up with:
00 02 E2 2F 25 20 8C 86 9D 52 60 1C B1 F2 56 E4
30 C3 00 01 01 02 08 4A 6F 68 6E 20 44 6F 65 03
0C 6A 6F 68 6E 40 64 6F 65 2E 63 6F 6D 04 10 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 07
76 A4 A8 A3 89 83 00 F8 FF 33 31 2F 77 DD 3D D2
FD 5B 77 CF A7 08 DC 39 08 ... // the rest is useless paddingThe VMPSerialData structure
Cross-reference this against the keygen’s VMProtectGenerateSerialNumber. The extracted structure:
00 02 E2 2F 25 20 8C 86 9D 52 60 1C B1 F2 56 E4 30 C3 00 // leading 0002 and trailing 00 are fixed; the rest is padding of random length
01 01 // version marker; currently fixed at two 01 bytes
02 // username
08 4A 6F 68 6E 20 44 6F 65 // one-byte length + text
03 // email
0C 6A 6F 68 6E 40 64 6F 65 2E 63 6F 6D // one-byte length + text
04 // machine code
10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 // one-byte length + machine code
07 // ProductCode — the most important one; without it the serial-locked code can't be decrypted
76 A4 A8 A3 89 83 00 F8 // fixed 8-byte ProductCode
FF // CRC, and end of parsing
33 31 2F 77 DD 3D D2 FD 5B 77 CF A7 08 DC 39 08 // 20-byte SHA1 checksum to prevent tampering with this data
// end of parsing; the rest is useless random paddingThere are more fields (expiry, run-time limits, and so on) — see VMProtectGenerateSerialNumber for those. With the ProductCode in hand we can also generate our own serials: skip the optional limit fields, generate our own RSA key, replace the program’s modulus, and keygen away.
Big-number cipher version differences
VMP encrypts all big numbers in memory. The cipher varies slightly by version, but all versions use a random 20-byte salt plus the memory address; set a hardware write breakpoint and trace the VM to recover the algorithm.
3.5
//decrypt
uint16_t* salt_ = (uint16_t*)psalt;
for (size_t i = 0; i < BigNum.size() / 2; i++)
{
size_t offset = (addr + (addr >> 7)) % 16;
size_t salt = *((uint8_t*)(salt_)+offset) + 0x37 + (addr >> 4);
buffer[i] = (uint16_t)((buffer[i] ^ salt) + addr);
addr += 2;
}
//encrypt
uint16_t* salt_ = (uint16_t*)psalt;
for (size_t i = 0; i < BigNum.size() / 2; i++)
{
size_t offset = (addr + (addr >> 7)) % 16;
size_t salt = *((uint8_t*)(salt_)+offset) + 0x37 + (addr >> 4);
buffer[i] = (uint16_t)((buffer[i] - addr) ^ salt);
addr += 2;
}3.6 and 3.7 (identical)
//decrypt
uint16_t* salt_ = (uint16_t*)psalt;
for (size_t i = 0; i < BigNum.size() / 2; i++)
{
size_t offset = ((uint16_t)addr + ((uint16_t)(addr) >> 7)) % 10;
size_t salt = salt_[offset] + 0x73 + ((uint16_t)(addr) >> 4);
buffer[i] = (uint16_t)((buffer[i] ^ salt) + addr);
addr += 2;
}
//encrypt
uint16_t* salt_ = (uint16_t*)psalt;
for (size_t i = 0; i < BigNum.size() / 2; i++)
{
size_t offset = ((uint16_t)addr + ((uint16_t)(addr) >> 7)) % 10;
size_t salt = salt_[offset] + 0x73 + ((uint16_t)(addr) >> 4);
buffer[i] = (uint16_t)((buffer[i] - addr) ^ salt);
addr += 2;
}3.8
//decrypt
uint16_t* salt_ = (uint16_t*)psalt;
for (size_t i = 0; i < BigNum.size() / 2; i++)
{
size_t offset = ((uint16_t)addr + ((uint16_t)(addr) >> 7)) % 10;
size_t idx = offset ^ ((uint16_t)addr >> 5);
size_t salt = salt_[offset] + (_rotl(0xFACE001E, idx % 8) & 0xFFFF);
buffer[i] = (uint16_t)((buffer[i] ^ salt) + addr);
addr += 2;
}
//encrypt
uint16_t* salt_ = (uint16_t*)psalt;
for (size_t i = 0; i < BigNum.size() / 2; i++)
{
size_t offset = ((uint16_t)addr + ((uint16_t)(addr) >> 7)) % 10;
size_t idx = offset ^ ((uint16_t)addr >> 5);
size_t salt = salt_[offset] + (_rotl(0xFACE001E, idx % 8) & 0xFFFF);
buffer[i] = (uint16_t)((buffer[i] - addr) ^ salt);
addr += 2;
}Same idea across versions — just the constants and salt handling tweaked. With the algorithm, the big numbers come apart.
Feasibility of patching the decrypted result
The serial carries a CRC, so you can’t just flip a few VMPSerialData bytes to change bindings or expiry — you’d have to update the serial’s SHA1 too. But we have VMP’s own keygen; a small tweak — delete the RSA step and have it emit VMPSerialData directly — works. When calling VMProtectGenerateSerialNumber, everything except the required ProductCode can be left unset, and the regenerated VMPSerialData works forever.
The patch flow: pass in a forged serial of the same length, hook RtlAllocateHeap, and on the 0xC04 allocation note the address. On the 0x20 allocation, decrypt that big number — using the address recorded at the 0x202 allocation, which you search for on the stack to find the salt. Then patch the pre-generated VMPSerialData into tmp1. If you can’t tell tmp1 from tmp2, patch both: just search for a region starting with 0x200 zeros.
Feasibility of a keygen
On 3.5 and earlier it’s easy: we have the ProductCode, we can generate serials ourselves, and we don’t even need to modify the keygen — just generate our own RSA key, hook RtlAllocateHeap, and on the first 0x202 allocation (the modulus) decrypt it and swap in our own, then re-encrypt and write it back. From 3.6 on, VMP no longer allocates a dedicated buffer for the modulus; it allocates everything at once and scrambles the order, so you need a hardware write breakpoint to find the right patch window. That’s fiddly to write, so I’ll only sketch the approach here.
Writing a DLL helper to patch the lock
Rolling it together: inject a DLL that hooks RtlAllocateHeap and rewrites the RSA decryption result. The plan:
1. Hook RtlEnterCriticalSection. It's called constantly, so first check the return address belongs to the main module, then check registers/stack for the serial to confirm it's a VMProtectSetSerialNumber call. Record it as EnterRVA. (Find this by hand in the debugger.)
2. Once RtlEnterCriticalSection is called from EnterRVA, hook RtlAllocateHeap. On the 0x202 allocation, record the address; on the 0xC04 allocation, search for the 0x202 address to recover the salt, and try the built-in VMP big-number decryption algorithms — a successful decrypt also auto-detects the VMP version. Finally, on the 0x20 allocation, use the recorded salt to decrypt the 0xC04 buffer, copy out the tmp1 region, parse it, strip the limit fields (machine code, expiry, etc.), regenerate an unlimited VMPSerialData, save it to a local file, and exit.
3. Reopen the program, read the local file, forge the serial, re-find tmp1 as above, encrypt the VMPSerialData, and patch it in.
4. Keep fields that change often (like EnterRVA) in an ini file for easy tuning.Because generating VMPSerialData and patching conflict in flow — and to keep things lean — I split it into two DLLs: VMPGetKey.dll generates VMPSerialData and saves it to a file, and VMPKeyPatcher.dll reads that file and does the patching.
With the DLLs the workflow collapses to:
Manually locate the RtlEnterCriticalSection call made by VMProtectSetSerialNumber (the EnterRVA), write it to InjectConfig.ini and tune the settings → inject VMPGetKey.dll under normal execution to dump VMPSerialData.data (name configurable in InjectConfig.ini) → forge any serial of the same length and inject VMPKeyPatcher.dll to finish the crack.
Cracking a network-verification product’s license lock
The sample only tests the local lock — no network verification. I built it in E-language and added a license-lock marker.
After packing there’s a dllbox; it does nothing useful, so once the serial is recovered it can be killed.
A winspool.drv hijack patch can recover the VMProtectSetSerialNumber EnterRVA — or, if you can stomach the anti-debug, attach and debug directly: after clicking login and the message box pops, the address at TEB+0x100 on the main thread holds the serial.
Then break on RtlEnterCriticalSection to get the EnterRVA — the stack address minus 0x400000 is it — and write it to InjectConfig.ini.
The hijack patch also has an auto-finder: run it and it emits EnterLog.log with the EnterRVA. If multiple lines appear, the first is usually right; if auto-detection fails, find it by hand. Then determine the KeySize from the serial length — take the base64-decoded length × 8 and pick the nearest value.
Once recovered, the DLL injection can kill VMP’s dllbox. Since VMP requires the dllbox to load successfully to continue, write mov eax,1; retn 0xC; at its entry. Then the serial has to be set at TEB+0x100 — the drv patch exports VMPLic.key and does this itself; if you don’t want the drv, do it in the debugger. After killing the dllbox, the rest is loading the two DLLs. Set LoadMode to 0 in InjectConfig.ini and the drv patch handles it automatically, needing only the EnterRVA.
With the EnterRVA filled in and LoadMode set to 1, it loads VMPGetKey.dll to auto-export VMPKeyData.data (the drv patch fixes the serial). A file appearing means success.
Finally set LoadMode to 2 to load VMPKeyPatcher.dll and patch — again the drv patch handles forging the serial. The main window opens with every button clickable, including the locked ones.
And the VMProtect local license lock is broken.
Summary
As the introduction said, VMProtect’s license lock lives or dies on VMProtectSetSerialNumber. Every protected marker is encrypted, but once you decrypt the serial and pull the 8-byte ProductCode, you can forge serials or patch the decrypted result directly.
The sample was only for testing VMProtect’s local lock and doesn’t touch the network-verification layer — that part can simply be ret’d out. There’s no point analyzing it further; getting past the license lock is enough. (Friendly advice to the vendor: adopt a more effective protection scheme, such as another network-verification product’s PIC shield and its anti-dump algorithm — though even that PIC shield is something I haven’t cracked yet.)
The patch will be open-sourced later.
Colin Xiao — I take on contract reverse-engineering work, and writeups like this are how I show what I can do. If you need help with anything in this space — including but not limited to the areas I cover above — I’m available. No project is too big or too small — if it’s reverse engineering and I can do it, I’ll take it. DM me on X (@colinxiao_lab) or Telegram (@ColinXiao666).
Get in touch