runtime: [wasm] `dotnet-runtime-perf` - `System.AccessViolationException: Attempted to read or write protected memory.`

This is failing on PERFTIGER198, which often hits odd errors, like inexplicable emcc/clang crashes. This machine might be in need of a memtest.

Build:

wasmaot.x64.micro.net7.0.Partition8
..

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]
   at Microsoft.CodeAnalysis.CSharp.InitializerRewriter.RewriteFieldInitializer(Microsoft.CodeAnalysis.CSharp.BoundFieldEqualsValue) [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]
   at System.Collections.Immutable.ImmutableArray.CreateRange[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Immutable.ImmutableArray`1<System.__Canon>, System.Func`2<System.__Canon,System.__Canon>) [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]
   at Microsoft.CodeAnalysis.CSharp.InitializerRewriter.RewriteConstructor(System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.CSharp.BoundInitializer>, Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol) [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileMethod(Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol, Int32, ProcessedFieldInitializers ByRef, Microsoft.CodeAnalysis.CSharp.SynthesizedSubmissionFields, Microsoft.CodeAnalysis.CSharp.TypeCompilationState) [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileNamedType(Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol) [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler+<>c__DisplayClass25_0.<CompileNamedTypeAsync>b__0() [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]
   at System.Threading.ThreadPoolWorkQueue.Dispatch() [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/home/helixbot/work/B07409D9/w/B42C095B/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net7.0]

cc @sblom @LoopedBard3 @DrewScoggins

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 22 (22 by maintainers)

Most upvoted comments

Re-removed the machine and getting guidance on more in-depth diagnostic help.

+1
LoopedBard3 on Nov 22, 2022

PERFTIGER198 is back - https://dev.azure.com/dnceng/internal/_build/results?buildId=2049158&view=results !

 "/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang++" -target wasm32-unknown-emscripten -DEMSCRIPTEN -D__EMSCRIPTEN_major__=3 -D__EMSCRIPTEN_minor__=1 -D__EMSCRIPTEN_tiny__=12 -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Cache.linux-x64/8.0.0-alpha.1.22558.2/tools/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -O2 -v -c /home/helixbot/work/AEA60976/p/performance/src/scenarios/blazorminappaot/app/obj/Release/net7.0/wasm/for-publish/System.Private.CoreLib.dll.bc -o /home/helixbot/work/AEA60976/t/tmpxzbXiT.tmp
clang version 15.0.0 (https://github.com/llvm/llvm-project d58cc0839ee5935e649dbbc18c898a8e4bcd6724)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin
 (in-process)
 "/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15" -cc1 -triple wasm32-unknown-emscripten -emit-obj --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name System.Private.CoreLib.dll.bc -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu generic -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/home/helixbot/work/AEA60976/p/performance/src/scenarios/blazorminappaot/app -resource-dir /home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/lib/clang/15.0.0 -O2 -Werror=implicit-function-declaration -fdebug-compilation-dir=/home/helixbot/work/AEA60976/p/performance/src/scenarios/blazorminappaot/app -ferror-limit 19 -fvisibility default -fgnuc-version=4.2.1 -vectorize-loops -vectorize-slp -iwithsysroot/include/SDL -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /home/helixbot/work/AEA60976/t/tmpxzbXiT.tmp -x ir /home/helixbot/work/AEA60976/p/performance/src/scenarios/blazorminappaot/app/obj/Release/net7.0/wasm/for-publish/System.Private.CoreLib.dll.bc
clang -cc1 version 15.0.0 based upon LLVM 15.0.0git default target x86_64-unknown-linux-gnu

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15 --driver-mode=g++ -target wasm32-unknown-emscripten -DEMSCRIPTEN -D__EMSCRIPTEN_major__=3 -D__EMSCRIPTEN_minor__=1 -D__EMSCRIPTEN_tiny__=12 -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Cache.linux-x64/8.0.0-alpha.1.22558.2/tools/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -O2 -v -c /home/helixbot/work/AEA60976/p/performance/src/scenarios/blazorminappaot/app/obj/Release/net7.0/wasm/for-publish/System.Private.CoreLib.dll.bc -o /home/helixbot/work/AEA60976/t/tmpxzbXiT.tmp
1.	Code generation
 #0 0x00005601aa0fd053 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x2da6053)
 #1 0x00005601aa0fa9de llvm::sys::RunSignalHandlers() (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x2da39de)
 #2 0x00005601aa0fc323 llvm::sys::CleanupOnSignal(unsigned long) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x2da5323)
 #3 0x00005601aa053c27 (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x2cfcc27)
 #4 0x00007f0c55e2d980 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12980)
 #5 0x00005601a9d60d61 llvm::MCAsmLayout::MCAsmLayout(llvm::MCAssembler&) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x2a09d61)
 #6 0x00005601a9d72c92 llvm::MCObjectStreamer::finishImpl() (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x2a1bc92)
 #7 0x00005601aafeec89 llvm::AsmPrinter::doFinalization(llvm::Module&) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x3c97c89)
 #8 0x00005601a9924aa1 llvm::FPPassManager::doFinalization(llvm::Module&) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x25cdaa1)
 #9 0x00005601a991cc51 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x25c5c51)
#10 0x00005601aa99c864 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, std::__2::unique_ptr<llvm::raw_pwrite_stream, std::__2::default_delete<llvm::raw_pwrite_stream>>) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x3645864)
#11 0x00005601aae11fe2 clang::CodeGenAction::ExecuteAction() (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x3abafe2)
#12 0x00005601aad33945 clang::FrontendAction::Execute() (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x39dc945)
#13 0x00005601aac94cd2 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x393dcd2)
#14 0x00005601aae0c8c3 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x3ab58c3)
#15 0x00005601a8cc067b cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x196967b)
#16 0x00005601a8cbe686 (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x1967686)
#17 0x00005601aaac67f2 (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x376f7f2)
#18 0x00005601aa0536c7 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x2cfc6c7)
#19 0x00005601aaac6448 clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optional<llvm::StringRef>>, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>*, bool*) const (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x376f448)
#20 0x00005601aaa81395 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x372a395)
#21 0x00005601aaaa20ec clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::__2::pair<int, clang::driver::Command const*>>&) (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x374b0ec)
#22 0x00005601a8cbdceb main (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x1966ceb)
#23 0x00007f0c54e6cbf7 __libc_start_main /build/glibc-S9d2JN/glibc-2.27/csu/../csu/libc-start.c:344:0
#24 0x00005601a8c3a79a _start (/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang-15+0x18e379a)

clang-15: error: clang frontend command failed with exit code 139 (use -v to see invocation)

clang version 15.0.0 (https://github.com/llvm/llvm-project d58cc0839ee5935e649dbbc18c898a8e4bcd6724)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin
clang-15: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
emcc: error: '/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.linux-x64/8.0.0-alpha.1.22558.2/tools/bin/clang++ -target wasm32-unknown-emscripten -DEMSCRIPTEN -D__EMSCRIPTEN_major__=3 -D__EMSCRIPTEN_minor__=1 -D__EMSCRIPTEN_tiny__=12 -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/home/helixbot/work/AEA60976/p/dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Cache.linux-x64/8.0.0-alpha.1.22558.2/tools/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -O2 -v -c /home/helixbot/work/AEA60976/p/performance/src/scenarios/blazorminappaot/app/obj/Release/net7.0/wasm/for-publish/System.Private.CoreLib.dll.bc -o /home/helixbot/work/AEA60976/t/tmpxzbXiT.tmp' failed (returned 139) [took 31.304s]
+1
radical on Nov 21, 2022

I do want to add that since the machine was taken out, I haven’t seen only one instance of the weird crashes from earlier.

+1
radical on Nov 18, 2022

PERFTIGER198 has been removed so we can run some tests on it to hopefully determine if the hardware is failing.

+1
LoopedBard3 on Oct 26, 2022

These are all on 198 specifically .

+1
radical on Oct 26, 2022
Read more comments on GitHub
← runtime: Wasm Browser work items failing with OpenQA.Selenium.WebDriverException
runtime: WASM: GL object no longer accessible in .net 7 (Skiasharp) →