azure-functions-host: Host crashes caused by log contention

We are experiencing unexpected host restarts when running a function. When this happens, the diagnostic logs show the normal trace messages from the execution of the function but occasionally, before the invocation completes, the logs show that the host restarted, with no indication of what has caused it to crash.

As I said, the streaming logs do not provide any information, but in the event log messages in Kudu (eventlog.xml) we have observed that every time that this happens, it seems to have been caused by contention while writing to the host logs. Here’s an excerpt of the log showing the two events that we always see before each crash.

<Event>
    <System>
        <Provider Name="ASP.NET 4.0.30319.0"/>
        <EventID>1325</EventID>
        <Level>1</Level>
        <Task>0</Task>
        <Keywords>Keywords</Keywords>
        <TimeCreated SystemTime="2018-10-31T01:37:53Z"/>
        <EventRecordID>777728515</EventRecordID>
        <Channel>Application</Channel>
        <Computer>…</Computer>
        <Security/>
    </System>
    <EventData>
        <Data>
            An unhandled exception occurred and the process was terminated.

            Application ID: /LM/W3SVC/1134994283/ROOT

            Process ID: 1780

            Exception: System.IO.IOException

            Message: The process cannot access the file 'D:\home\LogFiles\Application\Functions\Host\2018-10-31T01-34-39Z-7f29d1a4c5.log' because it is being used by another process.

            StackTrace:    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
            at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
            at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
            at System.IO.FileInfo.Open(FileMode mode, FileAccess access, FileShare share)
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.AppendToFile(FileInfo fileInfo, String content) in C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileTraceWriter.cs:line 279
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.Flush(Boolean timerCallback) in C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileTraceWriter.cs:line 151
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.FlushCallback(Object state) in C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileTraceWriter.cs:line 91
            at System.Threading.TimerQueueTimer.CallCallbackInContext(Object state)
            at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
            at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
            at System.Threading.TimerQueueTimer.CallCallback()
            at System.Threading.TimerQueueTimer.Fire()
            at System.Threading.TimerQueue.FireQueuedTimerCompletion(Object state)
            at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
            at System.Threading.ThreadPoolWorkQueue.Dispatch()
            at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
        </Data>
    </EventData>
</Event>
<Event>
    <System>
        <Provider Name=".NET Runtime"/>
        <EventID>1026</EventID>
        <Level>1</Level>
        <Task>0</Task>
        <Keywords>Keywords</Keywords>
        <TimeCreated SystemTime="2018-10-31T01:37:54Z"/>
        <EventRecordID>777729218</EventRecordID>
        <Channel>Application</Channel>
        <Computer>…</Computer>
        <Security/>
    </System>
    <EventData>
        <Data>
            Application: w3wp.exe
            Framework Version: v4.0.30319
            Description: The process was terminated due to an unhandled exception.
            Exception Info: System.IO.IOException
            at System.IO.__Error.WinIOError(Int32, System.String)
            at System.IO.FileStream.Init(System.String, System.IO.FileMode, System.IO.FileAccess, Int32, Boolean, System.IO.FileShare, Int32, System.IO.FileOptions, SECURITY_ATTRIBUTES, System.String, Boolean, Boolean, Boolean)
            at System.IO.FileStream..ctor(System.String, System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare)
            at System.IO.FileInfo.Open(System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare)
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.AppendToFile(System.IO.FileInfo, System.String)
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.Flush(Boolean)
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.FlushCallback(System.Object)
            at System.Threading.TimerQueueTimer.CallCallbackInContext(System.Object)
            at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
            at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
            at System.Threading.TimerQueueTimer.CallCallback()
            at System.Threading.TimerQueueTimer.Fire()
            at System.Threading.TimerQueue.FireQueuedTimerCompletion(System.Object)
            at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
            at System.Threading.ThreadPoolWorkQueue.Dispatch()
            at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

        </Data>
    </EventData>
</Event>

• Timestamp: 2018-11-06T12:53:45.889
• Function App version (1.0 or 2.0): 1.0
• Invocation ID: d2c4757a-7444-498c-b07c-f766fabef199
• Region: East US

Any suggestions or workarounds?

Thanks