runtime: AV when running MSBuild on bad invocation of VSD stub
Description
On validation build we are seeing a Virtual Stub Dispatch crash on an MSBuild assembly that has been crossgened. The crash stack is as follows:
# Child-SP RetAddr Call Site
00 00000012`047ff088 00007ffc`d51fd65f CLRStub[VSD_ResolveStub]@7ffc76aab223
01 00000012`047ff090 00007ffc`d5ed1d53 Microsoft_Build!Microsoft.Build.BackEnd.BufferedReadStream.Read+0x12f [/_/src/Shared/BufferedReadStream.cs @ 100]
02 00000012`047ff100 00007ffc`d5f777a1 System_Private_CoreLib!System.IO.Stream.<>c.<BeginReadInternal>b__40_0+0x43 [/_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs @ 246]
03 00000012`047ff150 00007ffc`d5e3684f System_Private_CoreLib+0x4277a1
04 00000012`047ff180 00007ffc`d5e1c985 System_Private_CoreLib!System.Threading.Tasks.Task.<>c.<.cctor>b__271_0+0x2f
05 00000012`047ff1b0 00007ffc`d5e31b48 System_Private_CoreLib!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop+0x35 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 268]
06 00000012`047ff200 00007ffc`d5e31a53 System_Private_CoreLib!System.Threading.Tasks.Task.ExecuteWithThreadLocal+0x98 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2331]
07 00000012`047ff2a0 00007ffc`d5e319fa System_Private_CoreLib!System.Threading.Tasks.Task.ExecuteEntryUnsafe+0x53 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2271]
08 00000012`047ff2e0 00007ffc`d5e253da System_Private_CoreLib!System.Threading.Tasks.Task.ExecuteFromThreadPool+0xa [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2256]
09 00000012`047ff310 00007ffc`d5e2cee5 System_Private_CoreLib!System.Threading.ThreadPoolWorkQueue.Dispatch+0x2ca
0a 00000012`047ff3a0 00007ffc`d5e1194f System_Private_CoreLib!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart+0x155 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 63]
0b 00000012`047ff4b0 00007ffc`d66c05c3 System_Private_CoreLib!System.Threading.Thread.StartCallback+0x3f [/_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 106]
0c 00000012`047ff4f0 00007ffc`d65e1c64 coreclr!CallDescrWorkerInternal+0x83
0d 00000012`047ff530 00007ffc`d66a38b3 coreclr!DispatchCallSimple+0x80 [D:\workspace\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 220]
0e 00000012`047ff5c0 00007ffc`d6645155 coreclr!ThreadNative::KickOffThread_Worker+0x63 [D:\workspace\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 158]
0f (Inline Function) --------`-------- coreclr!ManagedThreadBase_DispatchInner+0xd [D:\workspace\_work\1\s\src\coreclr\vm\threads.cpp @ 7312]
10 00000012`047ff620 00007ffc`d664505a coreclr!ManagedThreadBase_DispatchMiddle+0x85 [D:\workspace\_work\1\s\src\coreclr\vm\threads.cpp @ 7356]
11 00000012`047ff700 00007ffc`d6644e79 coreclr!ManagedThreadBase_DispatchOuter+0xae [D:\workspace\_work\1\s\src\coreclr\vm\threads.cpp @ 7515]
12 (Inline Function) --------`-------- coreclr!ManagedThreadBase_FullTransition+0x2d [D:\workspace\_work\1\s\src\coreclr\vm\threads.cpp @ 7560]
13 (Inline Function) --------`-------- coreclr!ManagedThreadBase::KickOff+0x2d [D:\workspace\_work\1\s\src\coreclr\vm\threads.cpp @ 7595]
14 00000012`047ff7a0 00007ffc`ffed7974 coreclr!ThreadNative::KickOffThread+0x79 [D:\workspace\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 230]
15 00000012`047ff800 00007ffd`021fa2f1 kernel32!BaseThreadInitThunk+0x14 [base\win32\client\thread.c @ 64]
16 00000012`047ff830 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [minkernel\ntdll\rtlstrt.c @ 1163]
The stub in question is this, with the faulting line highlighted:
CLRStub[VSD_ResolveStub]@7ffc76aab220:
00007ffc`76aab220 488b01 mov rax,qword ptr [rcx]
>> 00007ffc`76aab223 488b4060 mov rax,qword ptr [rax+60h]
00007ffc`76aab227 ff6018 jmp qword ptr [rax+18h]
00007ffc`76aab22a 2300 and eax,dword ptr [rax]
00007ffc`76aab22c 0000 add byte ptr [rax],al
00007ffc`76aab22e 0000 add byte ptr [rax],al
00007ffc`76aab230 488b01 mov rax,qword ptr [rcx]
00007ffc`76aab233 488b4060 mov rax,qword ptr [rax+60h]
The caller - Microsoft.Build.BackEnd.BufferedReadStream.Read(Byte[], Int32, Int32)
- is an prejitted method and around source line 100 is the issue. It does mov rcx,qword ptr [rsi+r15]
, where rsi+r15
is the address of the object. This means that the VSD stub received the MT as input and not the instance. If the instance was passed through RCX, I confirm the dispatch would’ve landed in the expected method.
disassembly
!u 00007ffc`d51fd65f
preJIT generated code
Microsoft.Build.BackEnd.BufferedReadStream.Read(Byte[], Int32, Int32)
ilAddr is 00007FFCD532E8B4 pImport is 000001FFFB4AACE0
Begin 00007FFCD51FD530, size 183
/_/src/Shared/BufferedReadStream.cs @ 66:
00007ffc`d51fd530 4157 push r15
00007ffc`d51fd532 4156 push r14
00007ffc`d51fd534 4154 push r12
00007ffc`d51fd536 57 push rdi
00007ffc`d51fd537 56 push rsi
00007ffc`d51fd538 55 push rbp
00007ffc`d51fd539 53 push rbx
00007ffc`d51fd53a 4883ec30 sub rsp,30h
00007ffc`d51fd53e 488bf1 mov rsi,rcx
00007ffc`d51fd541 488bda mov rbx,rdx
00007ffc`d51fd544 418be8 mov ebp,r8d
00007ffc`d51fd547 418bf9 mov edi,r9d
00007ffc`d51fd54a 81ff00040000 cmp edi,400h
00007ffc`d51fd550 7e72 jle Microsoft_Build!Microsoft.Build.BackEnd.BufferedReadStream.Read+0x94 (00007ffc`d51fd5c4)
/_/src/Shared/BufferedReadStream.cs @ 69:
00007ffc`d51fd552 4533f6 xor r14d,r14d
/_/src/Shared/BufferedReadStream.cs @ 70:
00007ffc`d51fd555 4c8b3dd4e93000 mov r15,qword ptr [Microsoft_Build!Microsoft.Build.Shared.ProjectErrorUtilities.VerifyThrowInvalidProject+0x224a10 (00007ffc`d550bf30)]
00007ffc`d51fd55c 468b643e10 mov r12d,dword ptr [rsi+r15+10h]
00007ffc`d51fd561 4585e4 test r12d,r12d
00007ffc`d51fd564 7e2c jle Microsoft_Build!Microsoft.Build.BackEnd.BufferedReadStream.Read+0x62 (00007ffc`d51fd592)
/_/src/Shared/BufferedReadStream.cs @ 72:
00007ffc`d51fd566 4489642420 mov dword ptr [rsp+20h],r12d
00007ffc`d51fd56b 4a8b4c3e08 mov rcx,qword ptr [rsi+r15+8]
00007ffc`d51fd570 428b543e14 mov edx,dword ptr [rsi+r15+14h]
00007ffc`d51fd575 4c8bc3 mov r8,rbx
00007ffc`d51fd578 448bcd mov r9d,ebp
00007ffc`d51fd57b ff15f7f72f00 call qword ptr [Microsoft_Build!Microsoft.Build.Shared.ProjectErrorUtilities.VerifyThrowInvalidProject+0x215858 (00007ffc`d54fcd78)] (System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32), mdToken: 000000000600016B)
/_/src/Shared/BufferedReadStream.cs @ 73:
00007ffc`d51fd581 468b743e10 mov r14d,dword ptr [rsi+r15+10h]
/_/src/Shared/BufferedReadStream.cs @ 74:
00007ffc`d51fd586 33c9 xor ecx,ecx
00007ffc`d51fd588 42894c3e14 mov dword ptr [rsi+r15+14h],ecx
/_/src/Shared/BufferedReadStream.cs @ 75:
00007ffc`d51fd58d 42894c3e10 mov dword ptr [rsi+r15+10h],ecx
/_/src/Shared/BufferedReadStream.cs @ 77:
00007ffc`d51fd592 4a8b0c3e mov rcx,qword ptr [rsi+r15]
00007ffc`d51fd596 448bcf mov r9d,edi
00007ffc`d51fd599 452bce sub r9d,r14d
00007ffc`d51fd59c 458d042e lea r8d,[r14+rbp]
00007ffc`d51fd5a0 488bd3 mov rdx,rbx
00007ffc`d51fd5a3 4c8d1d8e532f00 lea r11,[Microsoft_Build!Microsoft.Build.Shared.ProjectErrorUtilities.VerifyThrowInvalidProject+0x20b418 (00007ffc`d54f2938)]
00007ffc`d51fd5aa 3909 cmp dword ptr [rcx],ecx
00007ffc`d51fd5ac ff1586532f00 call qword ptr [Microsoft_Build!Microsoft.Build.Shared.ProjectErrorUtilities.VerifyThrowInvalidProject+0x20b418 (00007ffc`d54f2938)] (CLRStub[VSD_ResolveStub]@7ffc76aab220)
00007ffc`d51fd5b2 4103c6 add eax,r14d
00007ffc`d51fd5b5 4883c430 add rsp,30h
00007ffc`d51fd5b9 5b pop rbx
00007ffc`d51fd5ba 5d pop rbp
00007ffc`d51fd5bb 5e pop rsi
00007ffc`d51fd5bc 5f pop rdi
00007ffc`d51fd5bd 415c pop r12
00007ffc`d51fd5bf 415e pop r14
00007ffc`d51fd5c1 415f pop r15
00007ffc`d51fd5c3 c3 ret
/_/src/Shared/BufferedReadStream.cs @ 80:
00007ffc`d51fd5c4 4c8b3d65e93000 mov r15,qword ptr [Microsoft_Build!Microsoft.Build.Shared.ProjectErrorUtilities.VerifyThrowInvalidProject+0x224a10 (00007ffc`d550bf30)]
00007ffc`d51fd5cb 468b643e10 mov r12d,dword ptr [rsi+r15+10h]
00007ffc`d51fd5d0 443be7 cmp r12d,edi
00007ffc`d51fd5d3 7c35 jl Microsoft_Build!Microsoft.Build.BackEnd.BufferedReadStream.Read+0xda (00007ffc`d51fd60a)
/_/src/Shared/BufferedReadStream.cs @ 83:
00007ffc`d51fd5d5 897c2420 mov dword ptr [rsp+20h],edi
00007ffc`d51fd5d9 4a8b4c3e08 mov rcx,qword ptr [rsi+r15+8]
00007ffc`d51fd5de 428b543e14 mov edx,dword ptr [rsi+r15+14h]
00007ffc`d51fd5e3 4c8bc3 mov r8,rbx
00007ffc`d51fd5e6 448bcd mov r9d,ebp
00007ffc`d51fd5e9 ff1589f72f00 call qword ptr [Microsoft_Build!Microsoft.Build.Shared.ProjectErrorUtilities.VerifyThrowInvalidProject+0x215858 (00007ffc`d54fcd78)] (System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32), mdToken: 000000000600016B)
/_/src/Shared/BufferedReadStream.cs @ 84:
00007ffc`d51fd5ef 42017c3e14 add dword ptr [rsi+r15+14h],edi
/_/src/Shared/BufferedReadStream.cs @ 85:
00007ffc`d51fd5f4 42297c3e10 sub dword ptr [rsi+r15+10h],edi
/_/src/Shared/BufferedReadStream.cs @ 86:
00007ffc`d51fd5f9 8bc7 mov eax,edi
00007ffc`d51fd5fb 4883c430 add rsp,30h
00007ffc`d51fd5ff 5b pop rbx
00007ffc`d51fd600 5d pop rbp
00007ffc`d51fd601 5e pop rsi
00007ffc`d51fd602 5f pop rdi
00007ffc`d51fd603 415c pop r12
00007ffc`d51fd605 415e pop r14
00007ffc`d51fd607 415f pop r15
00007ffc`d51fd609 c3 ret
/_/src/Shared/BufferedReadStream.cs @ 91:
00007ffc`d51fd60a 4533f6 xor r14d,r14d
/_/src/Shared/BufferedReadStream.cs @ 92:
00007ffc`d51fd60d 4585e4 test r12d,r12d
00007ffc`d51fd610 7e2c jle Microsoft_Build!Microsoft.Build.BackEnd.BufferedReadStream.Read+0x10e (00007ffc`d51fd63e)
/_/src/Shared/BufferedReadStream.cs @ 94:
00007ffc`d51fd612 4489642420 mov dword ptr [rsp+20h],r12d
00007ffc`d51fd617 4a8b4c3e08 mov rcx,qword ptr [rsi+r15+8]
00007ffc`d51fd61c 428b543e14 mov edx,dword ptr [rsi+r15+14h]
00007ffc`d51fd621 4c8bc3 mov r8,rbx
00007ffc`d51fd624 448bcd mov r9d,ebp
00007ffc`d51fd627 ff154bf72f00 call qword ptr [Microsoft_Build!Microsoft.Build.Shared.ProjectErrorUtilities.VerifyThrowInvalidProject+0x215858 (00007ffc`d54fcd78)] (System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32), mdToken: 000000000600016B)
/_/src/Shared/BufferedReadStream.cs @ 95:
00007ffc`d51fd62d 468b743e10 mov r14d,dword ptr [rsi+r15+10h]
/_/src/Shared/BufferedReadStream.cs @ 96:
00007ffc`d51fd632 33c9 xor ecx,ecx
00007ffc`d51fd634 42894c3e14 mov dword ptr [rsi+r15+14h],ecx
/_/src/Shared/BufferedReadStream.cs @ 97:
00007ffc`d51fd639 42894c3e10 mov dword ptr [rsi+r15+10h],ecx
/_/src/Shared/BufferedReadStream.cs @ 100:
00007ffc`d51fd63e 4a8b0c3e mov rcx,qword ptr [rsi+r15]
00007ffc`d51fd642 4a8b543e08 mov rdx,qword ptr [rsi+r15+8]
00007ffc`d51fd647 4c8d1dea522f00 lea r11,[Microsoft_Build!Microsoft.Build.Shared.ProjectErrorUtilities.VerifyThrowInvalidProject+0x20b418 (00007ffc`d54f2938)]
00007ffc`d51fd64e 4533c0 xor r8d,r8d
00007ffc`d51fd651 41b900040000 mov r9d,400h
00007ffc`d51fd657 3909 cmp dword ptr [rcx],ecx
00007ffc`d51fd659 ff15d9522f00 call qword ptr [Microsoft_Build!Microsoft.Build.Shared.ProjectErrorUtilities.VerifyThrowInvalidProject+0x20b418 (00007ffc`d54f2938)] (CLRStub[VSD_ResolveStub]@7ffc76aab220)
>>> 00007ffc`d51fd65f 448be0 mov r12d,eax
/_/src/Shared/BufferedReadStream.cs @ 101:
00007ffc`d51fd662 33c9 xor ecx,ecx
00007ffc`d51fd664 42894c3e14 mov dword ptr [rsi+r15+14h],ecx
/_/src/Shared/BufferedReadStream.cs @ 102:
00007ffc`d51fd669 4689643e10 mov dword ptr [rsi+r15+10h],r12d
/_/src/Shared/BufferedReadStream.cs @ 106:
00007ffc`d51fd66e 438d0c26 lea ecx,[r14+r12]
00007ffc`d51fd672 3bcf cmp ecx,edi
00007ffc`d51fd674 7c05 jl Microsoft_Build!Microsoft.Build.BackEnd.BufferedReadStream.Read+0x14b (00007ffc`d51fd67b)
/_/src/Shared/BufferedReadStream.cs @ 108:
00007ffc`d51fd676 412bfe sub edi,r14d
00007ffc`d51fd679 eb03 jmp Microsoft_Build!Microsoft.Build.BackEnd.BufferedReadStream.Read+0x14e (00007ffc`d51fd67e)
/_/src/Shared/BufferedReadStream.cs @ 112:
00007ffc`d51fd67b 418bfc mov edi,r12d
/_/src/Shared/BufferedReadStream.cs @ 115:
00007ffc`d51fd67e 897c2420 mov dword ptr [rsp+20h],edi
00007ffc`d51fd682 4a8b4c3e08 mov rcx,qword ptr [rsi+r15+8]
00007ffc`d51fd687 458d0c2e lea r9d,[r14+rbp]
00007ffc`d51fd68b 4c8bc3 mov r8,rbx
00007ffc`d51fd68e 33d2 xor edx,edx
00007ffc`d51fd690 ff15e2f62f00 call qword ptr [Microsoft_Build!Microsoft.Build.Shared.ProjectErrorUtilities.VerifyThrowInvalidProject+0x215858 (00007ffc`d54fcd78)] (System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32), mdToken: 000000000600016B)
/_/src/Shared/BufferedReadStream.cs @ 116:
00007ffc`d51fd696 42017c3e14 add dword ptr [rsi+r15+14h],edi
/_/src/Shared/BufferedReadStream.cs @ 117:
00007ffc`d51fd69b 42297c3e10 sub dword ptr [rsi+r15+10h],edi
/_/src/Shared/BufferedReadStream.cs @ 119:
00007ffc`d51fd6a0 418d043e lea eax,[r14+rdi]
00007ffc`d51fd6a4 4883c430 add rsp,30h
00007ffc`d51fd6a8 5b pop rbx
00007ffc`d51fd6a9 5d pop rbp
00007ffc`d51fd6aa 5e pop rsi
00007ffc`d51fd6ab 5f pop rdi
00007ffc`d51fd6ac 415c pop r12
00007ffc`d51fd6ae 415e pop r14
00007ffc`d51fd6b0 415f pop r15
00007ffc`d51fd6b2 c3 ret
This is the MSBuild that’s included in the RC1 SDK 6.0.100-rc.1.21430.12. MSBuild Commit: 414393fc1ff0e808865a088c826122694fc4fe3f (method in question is here) Package: Microsoft.Build @ 17.0.0-preview-21427-02 Build: https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=5140185
I am still not sure how they prejit their assemblies.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (15 by maintainers)
From the dump, the object we’re working on is
Given the codegen, the offset in
r15
should have been0x10
to get the right behavior.If we look at all the jitted code indirs off of
rsi + r15
a similar picture emerges:We see fixed offsets of
0x0
,0x8
,0x10
,0x14
which imply the expected runtime offset should indeed be0x10
.But from exception context it looks like at runtime
r15
was zero… which is why we ended up loadingrcx
with the method table for theBufferedReadStream
object (also consistent with this:rdx
points at the semaphore at offset0x8
, rather than the buffer at offset0x18
).The jitted code fetches
r15
from a reloc. And contents of the reloc cell at0x00007ffcd550bf30
are indeed0x10
. Jitted code doesn’t modifyr15
in the method. So it appears thatr15
got corrupted somehow.Given the failing call site there’s one possible call the method could have made between setting
r15
and using in the code leading up to the VSD:But note that
r15
is used before and after that earlier call… so those post-call writes (if the call somehow causedr15
to be set to zero) would write zeros to[rsi + 0x1a]
and[rsi + 0x10]
respectively… but those fields have plausible values.So I don’t have any idea what might have caused
r15
to end up with the wrong value.