runtime: __pthread_mutex_lock: Assertion `mutex->data.owner == 0' failed. running DirectoryServices tests on Linux coreclr

Not much to go on –

Libraries Test Run release coreclr Linux x64 Debug (Fedora.30.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-30-helix-20200512010621-4f8cef7.

https://dev.azure.com/dnceng/public/_build/results?buildId=718537&view=logs&jobId=71afe768-e493-597c-228c-037834193aeb&j=71afe768-e493-597c-228c-037834193aeb&t=3e74a8b7-5e62-58f8-6a85-9e36ef39d1c1

    System.DirectoryServices.Protocols.Tests.DirectoryServicesProtocolsTests.TestAddUser [SKIP]
      Condition(s) not met: "IsLdapConfigurationExist"
dotnet: ../nptl/pthread_mutex_lock.c:79: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.
./RunTests.sh: line 161:  2035 Aborted                 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.DirectoryServices.Protocols.Tests.runtimeconfig.json --depsfile System.DirectoryServices.Protocols.Tests.deps.json xunit.console.dll System.DirectoryServices.Protocols.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
~/work/B45D097D/w/B56E0989/e
----- end Tue Jul 7 05:00:01 UTC 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
Looking around for any Linux dump...
... found no dump in /home/helixbot/work/B45D097D/w/B56E0989/e

@davidfowl where are we at with collecting dump files? 😃

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 52 (52 by maintainers)

Most upvoted comments

Got it to work! Close enough was good enough in this case. I went and found a daily build that was close to the hash of head for this PR. I downloaded it’s CoreCLRCrossDacArtifacts zip and extracted to a folder. At first it didn’t work because it had previously loaded the “wrong” windows/windows DAC and refused to reload the new windows/linux-x64 DAC. After restarting dotnet-dump I was in business:

C:\Debuggers>dotnet-dump analyze C:\Users\ericstj\Downloads\DS.P-dump\core.1000.24
Loading core dump: C:\Users\ericstj\Downloads\DS.P-dump\core.1000.24 ...
Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
Type 'quit' or 'exit' to exit the session.
> setclrpath C:\Users\ericstj\Downloads\DS.P-dump\CoreCLRCrossDacArtifacts\Linux.x64.Release\x64
Set load path for DAC/DBI to 'C:\Users\ericstj\Downloads\DS.P-dump\CoreCLRCrossDacArtifacts\Linux.x64.Release\x64'
> clrstack
OS Thread Id: 0x29 (13)
        Child SP               IP Call Site
00007F14ACBAB1C0 00007f14acedce4f [InlinedCallFrame: 00007f14acbab1c0] Interop.ldap_init(System.String, Int32)
00007F14ACBAB1C0 00007f14dead23e6 [InlinedCallFrame: 00007f14acbab1c0] Interop.ldap_init(System.String, Int32)
00007F14ACBAB1B0 00007F14DEAD23E6 ILStubClass.IL_STUB_PInvoke(System.String, Int32)
00007F14ACBAB2B0 00007F14DEAD206B System.DirectoryServices.Protocols.LdapConnection.InternalInitConnectionHandle(System.String)

I have to say, this is an awesome feature. I really look forward to being able to use this more.

ericstj on Sep 4, 2020

The Cross OS DAC gets built in CI, but there’s nothing that uses it so I never published it on non-official builds. It just seemed like added time for PR builds and even more asset upload. Given that we could use them for things like this, I guess it was short sighted and overly optimizing. I’ve opened #41852 so that it ends up in the artifacts for a pipeline under a container named CoreCLRCrossDacArtifacts:

It should work as-is on WinDBG if on the same directory. Recent previews of VS should work but there was a bug (RC1 doesn’t have the fix, RC2 should have it), so it will work on builds after today more or less. We do index the Cross DAC fully signed and all the symbols, but it won’t help as these are PR builds.

hoyosjs on Sep 4, 2020

“close enough” official build

Got it to work!

One thing I can imagine will be tricky here is the current workflow appears to require downloading and executing a binary produced by the PR build.

It is not accidental that close enough worked for the cross DAC. #32331 was implemented specifically to make this work more often (as well as for the single file longer term project.)

@noahfalk has suggested that the runtime data structures this depends on change rarely, They are in tricky places in the runtime. Few people touch them…

That said, I think we would guess “close enough” should work > 99% of the time.

I went and found a daily build that was close to the hash of head for this PR

The closest daily matching build would minimize the outages… It might be easier to use the Cross DAC’s matching the current SDK the runtime is building with.

We could probably add some testing to see if the SDK signed DAC will work with libcoreclr built by the PR.

If we start using this for PR dump debugging, we get the nice dogfooding testing to help make this more reliable.

sdmaclea on Sep 4, 2020

I can go to the test failure and click a link and it will essentially pull down everything and open it in the debugger for me with everything resolved.

I was thinking of the same. I love the idea. One thing I can imagine will be tricky here is the current workflow appears to require downloading and executing a binary produced by the PR build. That could be a bit sketchy as we don’t typically trust executing the code in PRs on our dev machines. I wonder if we could have a workflow that was more permissive and only ran trusted code 🤔

ericstj on Sep 4, 2020

Where we want to get to (our vision 😃 ) is that I can go to the test failure and click a link and it will essentially pull down everything and open it in the debugger for me with everything resolved. This is the experience for Microsoft devs looking at Watson bugs - after resolving symbols it even triggers !analyze for you. I would like this to work even if I’m viewing the test results on Windows, and it’s a Unix failure. So this is all promising.

danmoseley on Sep 4, 2020

runtime: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed. running DirectoryServices tests on Linux coreclr

About this issue

Most upvoted comments

runtime: __pthread_mutex_lock: Assertion `mutex->data.owner == 0' failed. running DirectoryServices tests on Linux coreclr