msbuild: MSBUILD : corefx: error MSB4017: The build stopped unexpectedly because of an unexpected logger failure.

In corefx build https://dnceng.visualstudio.com/internal/_build/results?buildId=79263 failed with this issue. The connected issue is : https://github.com/dotnet/core-eng/issues/5071 The issue seems to be similar to

2019-01-24T06:33:05.0754128Z MSBUILD : error MSB4017: The build stopped unexpectedly because of an unexpected logger failure.
2019-01-24T06:33:05.0761011Z Microsoft.Build.Exceptions.InternalLoggerException: The build stopped unexpectedly because of an unexpected logger failure. ---> System.NullReferenceException: Object reference not set to an instance of an object.
2019-01-24T06:33:05.0765474Z    at Microsoft.Build.Logging.BuildEventArgsWriter.Write(ITaskItem item)
2019-01-24T06:33:05.0766136Z    at Microsoft.Build.Logging.BuildEventArgsWriter.WriteItems(IEnumerable items)
2019-01-24T06:33:05.0766272Z    at Microsoft.Build.Logging.BuildEventArgsWriter.Write(ProjectStartedEventArgs e)
2019-01-24T06:33:05.0773982Z    at Microsoft.Build.Logging.BuildEventArgsWriter.Write(BuildEventArgs e)
2019-01-24T06:33:05.0774216Z    at Microsoft.Build.Logging.BinaryLogger.Write(BuildEventArgs e)
2019-01-24T06:33:05.0774341Z    at Microsoft.Build.Logging.BinaryLogger.EventSource_AnyEventRaised(Object sender, BuildEventArgs e)
2019-01-24T06:33:05.0782331Z    at Microsoft.Build.Evaluation.ProjectCollection.ReusableLogger.AnyEventRaisedHandler(Object sender, BuildEventArgs e)
2019-01-24T06:33:05.0782512Z    at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseAnyEvent(Object sender, BuildEventArgs buildEvent)
2019-01-24T06:33:05.0782990Z    --- End of inner exception stack trace ---
2019-01-24T06:33:05.0801948Z    at Microsoft.Build.Exceptions.InternalLoggerException.Throw(Exception innerException, BuildEventArgs e, String messageResourceName, Boolean initializationException, String[] messageArgs)
2019-01-24T06:33:05.0802906Z    at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseAnyEvent(Object sender, BuildEventArgs buildEvent)
2019-01-24T06:33:05.0805951Z    at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseStatusEvent(Object sender, BuildStatusEventArgs buildEvent)
2019-01-24T06:33:05.0806894Z    at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseProjectStartedEvent(Object sender, ProjectStartedEventArgs buildEvent)
2019-01-24T06:33:05.0812100Z    at Microsoft.Build.BackEnd.Logging.EventSourceSink.Consume(BuildEventArgs buildEvent)
2019-01-24T06:33:05.0812253Z    at Microsoft.Build.BackEnd.Logging.EventSourceSink.Consume(BuildEventArgs buildEvent, Int32 sinkId)
2019-01-24T06:33:05.0812409Z    at Microsoft.Build.BackEnd.Logging.EventRedirectorToSink.Microsoft.Build.Framework.IEventRedirector.ForwardEvent(BuildEventArgs buildEvent)
2019-01-24T06:33:05.0812594Z    at Microsoft.Build.BackEnd.Logging.CentralForwardingLogger.EventSource_AnyEventRaised(Object sender, BuildEventArgs buildEvent)
2019-01-24T06:33:05.0817397Z    at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseAnyEvent(Object sender, BuildEventArgs buildEvent)

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 1
  • Comments: 20 (16 by maintainers)

Most upvoted comments

What I saw was caused by a DEVPATH that pointed to a bad file. Thanks, @KirillOsenkov

Hello,

We are having similar issue with MSBuild 17.1.0 (from VS 2022 17.1.2 installation) when using maximum CPU count, i.e. /m

MSBUILD : error MSB4017: The build stopped unexpectedly because of an unexpected logger failure. Microsoft.Build.Exceptions.InternalLoggerException: The build stopped unexpectedly because of an unexpected logger failure. —> System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Build.BackEnd.Logging.ParallelConsoleLogger.WriteMessageAligned(String message, Boolean prefixAlreadyWritten, Int32 prefixAdjustment) at Microsoft.Build.BackEnd.Logging.ParallelConsoleLogger.PrintTargetNamePerMessage(BuildMessageEventArgs e, Boolean lightenText) at Microsoft.Build.BackEnd.Logging.ParallelConsoleLogger.PrintMessage(BuildMessageEventArgs e, Boolean lightenText) at Microsoft.Build.BackEnd.Logging.ParallelConsoleLogger.MessageHandler(Object sender, BuildMessageEventArgs e) at Microsoft.Build.Logging.ConsoleLogger.MessageHandler(Object sender, BuildMessageEventArgs e) at ColorLogger.ColorLogger.MessageHandler(Object sender, BuildMessageEventArgs e) in C:\Users\bouchraREKHADDA\source\repos\ColorLogger\ColorLogger.cs:line 140 at Microsoft.Build.Evaluation.ProjectCollection.ReusableLogger.MessageRaisedHandler(Object sender, BuildMessageEventArgs e) at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseMessageEvent(Object sender, BuildMessageEventArgs buildEvent) — End of inner exception stack trace — at Microsoft.Build.Exceptions.InternalLoggerException.Throw(Exception innerException, BuildEventArgs e, String messageResourceName, Boolean initializationException, String[] messageArgs) at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseMessageEvent(Object sender, BuildMessageEventArgs buildEvent) at Microsoft.Build.BackEnd.Logging.EventSourceSink.Consume(BuildEventArgs buildEvent) at Microsoft.Build.BackEnd.Logging.EventSourceSink.Consume(BuildEventArgs buildEvent, Int32 sinkId) at Microsoft.Build.BackEnd.Logging.EventRedirectorToSink.Microsoft.Build.Framework.IEventRedirector.ForwardEvent(BuildEventArgs buildEvent) at Microsoft.Build.BackEnd.Logging.CentralForwardingLogger.EventSource_AnyEventRaised(Object sender, BuildEventArgs buildEvent) at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseAnyEvent(Object sender, BuildEventArgs buildEvent)

A full reproduction repository can be found here bouchraRekhadda/ColorLogger

There’s really not. The environment caused a mismatched system.dll to be loaded.

There are two implementations of CloneCustomMetadata() … ProjectItemInstance and TaskItem. I assume the logger gets the former:

            public IDictionary CloneCustomMetadata()
            {
                Dictionary<string, string> clonedMetadata = new Dictionary<string, string>(MSBuildNameIgnoreCaseComparer.Default);

                foreach (ProjectMetadataInstance metadatum in MetadataCollection)
                {
                    clonedMetadata[metadatum.Name] = metadatum.EvaluatedValue;
                }

                return clonedMetadata;
            }

clonedMetadata cannot be null. Also since MetadataCollection getter is not on the stack, and looks too big to inline I assume the NullReferenceException does not come from within it. And by inspection, it will never return null either. So either the dictionary contains a null metadatum - and from a quick look (I didn’t look carefully), it should not, because it uses key off each entry as its internal key - or something else is happening.

Implementation of CloneCustomMetadata() in TaskItem, if that’s relevant, is similar, again the source and target collection is checked for null.

Suggestion - some unprotected concurrent read and write to the backing dictionary here, either source or destination. That can often manifest as a NullReferenceException when accessing a dictionary. Since there’s definitely multithreading going on here, perhaps one thread didn’t properly “let go” of this item before it got to the logging thread?

If that’s a possibility, as well as code inspection, another possibility is swapping out the relevant Dictionary<K,V>'s with some custom dictionary that immediatley breaks into the debugger when a write occurs to it concurrent with another read (or write). That might possibly, in stress runs, cause this issue to happen locally, and make it possible to find the “other” callstack.