runtime: Deadlock in Linux Socket code

Description

Our production server stopped receiving data and accepting network connections. Our internal business logic kept running, otherwise our watchdog timer would have detected that the server was unresponsive and restarted it automatically. The socket classes were apparently hanging in their async codepath so we could not detect this from inside the process.

I’ve taken a dotnet-dump --collect pid which I can share privately if that is of any interest, or extract further information if you give me instructions.

stack traces from dotnet dump
OS Thread Id: 0x720
        Child SP               IP Call Site
00007FFF201F5530 00007f888c5bcad3 [GCFrame: 00007fff201f5530]
00007FFF201F5610 00007f888c5bcad3 [HelperMethodFrame_1OBJ: 00007fff201f5610] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007FFF201F5740 00007F8811390DC4 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007FFF201F57D0 00007F88113909E2 System.Threading.ManualResetEventSlim.Wait() [/_/src/System.Private.CoreLib/shared/System/Threading/ManualResetEventSlim.cs @ 399]
00007FFF201F57E0 00007F88118719B2 <AppNamespace>.Program.Main(System.String[]) [/home/<user>/public/build/server/Program.cs @ 61]
00007FFF201F5BA8 00007f888ac8749f [GCFrame: 00007fff201f5ba8]
00007FFF201F6090 00007f888ac8749f [GCFrame: 00007fff201f6090]
OS Thread Id: 0x725
        Child SP               IP Call Site
00007F8883484CD0 00007f888c5bcfb9 [DebuggerU2MCatchHandlerFrame: 00007f8883484cd0]
OS Thread Id: 0x727
        Child SP               IP Call Site
OS Thread Id: 0x72a
        Child SP               IP Call Site
00007F87E8C2C5B0 00007f888c5bcad3 [GCFrame: 00007f87e8c2c5b0]
00007F87E8C2C690 00007f888c5bcad3 [HelperMethodFrame_1OBJ: 00007f87e8c2c690] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F87E8C2C7C0 00007F8811891BE9 <AppNamespace>.ServerThread.ExecutionThread() [/home/<user>/public/build/core/ServerThread.cs @ 283]
00007F87E8C2C800 00007F8811385862 System.Threading.ThreadHelper.ThreadStart_Context(System.Object) [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 50]
00007F87E8C2C820 00007F881138F111 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F87E8C2C870 00007F881138597E System.Threading.ThreadHelper.ThreadStart() [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 100]
00007F87E8C2CBD0 00007f888ac8749f [GCFrame: 00007f87e8c2cbd0]
00007F87E8C2CCA0 00007f888ac8749f [DebuggerU2MCatchHandlerFrame: 00007f87e8c2cca0]
OS Thread Id: 0x72b
        Child SP               IP Call Site
00007F87D3FFE430 00007f888c5bcad3 [GCFrame: 00007f87d3ffe430]
00007F87D3FFE520 00007f888c5bcad3 [GCFrame: 00007f87d3ffe520]
00007F87D3FFE580 00007f888c5bcad3 [HelperMethodFrame_1OBJ: 00007f87d3ffe580] System.Threading.Monitor.Enter(System.Object)
00007F87D3FFE6D0 00007F8814BFF2D6 System.Net.Sockets.SocketAsyncContext+OperationQueue`1[[System.__Canon, System.Private.CoreLib]].HandleEvent(System.Net.Sockets.SocketAsyncContext) [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 871]
00007F87D3FFE710 00007F8814C013F4 System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents) [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 1944]
00007F87D3FFE730 00007F881212D2D3 System.Net.Sockets.SocketAsyncEngine.EventLoop() [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs @ 334]
00007F87D3FFE780 00007F881212D219 System.Net.Sockets.SocketAsyncEngine+<>c.<.ctor>b__23_0(System.Object)
00007F87D3FFE790 00007F88113A4D83 System.Threading.Tasks.Task.InnerInvoke() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2449]
00007F87D3FFE7B0 00007F88113A4B91 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2378]
00007F87D3FFE830 00007F88113A49E8 System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2327]
00007F87D3FFE850 00007F88113B196B System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/ThreadPoolTaskScheduler.cs @ 37]
00007F87D3FFE860 00007F881138593A System.Threading.ThreadHelper.ThreadStart(System.Object) [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 83]
00007F87D3FFEBD0 00007f888ac8749f [GCFrame: 00007f87d3ffebd0]
00007F87D3FFECA0 00007f888ac8749f [DebuggerU2MCatchHandlerFrame: 00007f87d3ffeca0]
OS Thread Id: 0x72c
        Child SP               IP Call Site
00007F87D37FD5B0 00007f888c5bcad3 [GCFrame: 00007f87d37fd5b0]
00007F87D37FD690 00007f888c5bcad3 [HelperMethodFrame_1OBJ: 00007f87d37fd690] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F87D37FD7C0 00007F8811891BE9 <AppNamespace>.ServerThread.ExecutionThread() [/home/<user>/public/build/core/ServerThread.cs @ 283]
00007F87D37FD800 00007F8811385862 System.Threading.ThreadHelper.ThreadStart_Context(System.Object) [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 50]
00007F87D37FD820 00007F881138F111 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F87D37FD870 00007F881138597E System.Threading.ThreadHelper.ThreadStart() [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 100]
00007F87D37FDBD0 00007f888ac8749f [GCFrame: 00007f87d37fdbd0]
00007F87D37FDCA0 00007f888ac8749f [DebuggerU2MCatchHandlerFrame: 00007f87d37fdca0]
OS Thread Id: 0xc0c
        Child SP               IP Call Site
00007F87D2FFC2D8 00007f888c5bcfb9 [HelperMethodFrame: 00007f87d2ffc2d8] System.Threading.Thread.SleepInternal(Int32)
00007F87D2FFC420 00007F8814D639EF System.Threading.SpinWait.SpinOnceCore(Int32) [/_/src/System.Private.CoreLib/shared/System/Threading/SpinWait.cs @ 242]
00007F87D2FFC4B0 00007F8815036402 System.Net.Sockets.SocketAsyncContext+AsyncOperation.TryCancel() [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 202]
00007F87D2FFC4E0 00007F8814CF98BB System.Net.Sockets.SocketAsyncContext+OperationQueue`1[[System.__Canon, System.Private.CoreLib]].StopAndAbort(System.Net.Sockets.SocketAsyncContext) [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 1117]
00007F87D2FFC510 00007F8814D23350 System.Net.Sockets.SocketAsyncContext.Close() [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 1192]
00007F87D2FFC540 00007F8814D070A9 System.Net.Sockets.SafeSocketHandle.ReleaseHandle() [/_/src/System.Net.Sockets/src/System/Net/Sockets/SafeSocketHandle.cs @ 141]
00007F87D2FFC560 00007F88135771A9 System.Runtime.InteropServices.SafeHandle.InternalRelease(Boolean) [/_/src/System.Private.CoreLib/shared/System/Runtime/InteropServices/SafeHandle.cs @ 183]
00007F87D2FFC5A0 00007F88135770AA System.StubHelpers.StubHelpers.SafeHandleRelease(System.Runtime.InteropServices.SafeHandle) [/_/src/System.Private.CoreLib/src/System/StubHelpers.cs @ 1644]
00007F87D2FFC5D0 00007f8812a2978d [InlinedCallFrame: 00007f87d2ffc5d0] Interop+Sys.ReceiveMessage(System.Runtime.InteropServices.SafeHandle, MessageHeader*, System.Net.Sockets.SocketFlags, Int64*)
00007F87D2FFC5C0 00007F8812A2978D ILStubClass.IL_STUB_PInvoke(System.Runtime.InteropServices.SafeHandle, MessageHeader*, System.Net.Sockets.SocketFlags, Int64*)
00007F87D2FFC660 00007F8814C002D8 System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle, System.Net.Sockets.SocketFlags, System.Span`1<Byte>, Byte[], Int32 ByRef, System.Net.Sockets.SocketFlags ByRef, Error ByRef) [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketPal.Unix.cs @ 69]
00007F87D2FFC710 00007F8814C00121 System.Net.Sockets.SocketPal.TryCompleteReceiveFrom(System.Net.Sockets.SafeSocketHandle, System.Span`1<Byte>, System.Collections.Generic.IList`1<System.ArraySegment`1<Byte>>, System.Net.Sockets.SocketFlags, Byte[], Int32 ByRef, Int32 ByRef, System.Net.Sockets.SocketFlags ByRef, System.Net.Sockets.SocketError ByRef) [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketPal.Unix.cs @ 644]
00007F87D2FFC770 00007F8814C0B896 System.Net.Sockets.SocketAsyncContext+BufferMemoryReceiveOperation.DoTryComplete(System.Net.Sockets.SocketAsyncContext) [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 473]
00007F87D2FFC7F0 00007F8814C09D35 System.Net.Sockets.SocketAsyncContext+OperationQueue`1[[System.__Canon, System.Private.CoreLib]].ProcessQueuedOperation(System.__Canon) [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 1010]
00007F87D2FFC860 00007F8814C09C1C System.Net.Sockets.SocketAsyncContext+OperationQueue`1[[System.__Canon, System.Private.CoreLib]].ProcessAsyncOperation(System.__Canon) [/_/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 891]
00007F87D2FFC890 00007F8814BFEA46 System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs @ 699]
00007F87D2FFCCA0 00007f888ac8749f [DebuggerU2MCatchHandlerFrame: 00007f87d2ffcca0]
OS Thread Id: 0xc5e
        Child SP               IP Call Site
OS Thread Id: 0xc61
        Child SP               IP Call Site

Is this a known issue? Can this be diagnosed from the dump? Otherwise, from my side, we’ll continue to monitor if it happens again, but I figured this may be of interest and report it early, considering bugs in the Socket implementation may affect ASP.NET servers as well.

Configuration

  • Which version of .NET is the code running on? 3.1.404

  • What OS and version, and what distro if applicable? Ubuntu 18.04.5 LTS

  • What is the architecture (x64, x86, ARM, ARM64)? x64

dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.1.404
 Commit:    470f6754b3

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/3.1.404/

Host (useful for support):
  Version: 3.1.10
  Commit:  1721e39439

.NET Core SDKs installed:
  2.2.402 [/usr/share/dotnet/sdk]
  3.1.404 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.10 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.10 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (17 by maintainers)

Most upvoted comments

I haven’t looked in detail at everything provided here, but

Based on the dump, it looks like the completion of the ReceiveMessage P/Invoke triggers the disposal of SafeSocketHandle. As far as I understand, this could only happen if the underlying socket gets closed during an outgoing receive call.

reminds me of https://github.com/dotnet/runtime/issues/31570.

Ok I can reproduce somewhat nicely by sending bursts of 20 parallel connects followed by garbage data (bytewise, Nagle disabled) and a client socket disposal, to induce the suspected scenario of server side disposal while there are still outstanding TCP packets being handled.

I’ve collected a Microsoft-System-Net-Sockets trace I could share privately, but at this point it might be better to reduce the server application by removing all business logic (its not required to reject garbage data) and build a publically shareable repro scenario?

Will continue working on this so I should have something early next week. (Also will try some of the suggestions, especially how it behaves under .NET 5, this test was the .NET 3.1 build)

[edit] taking a bit longer, turns out it is harder to reproduce locally under WSL2 so I have to experiment with different setups to see what works best

[edit] making progress, can reliably repro with WSL2/Ubuntu and will have the stripped repro ready in a day or two

Can you collect this during the deadlock?

I’ll do once I’m able to repro, we had no further occurrences of the deadlock after the initial DOS attempt was over. Could be that there is no more interest to pressure our server, or could be that switching to .NET 5 is sufficient to fix the problem.

As suggested earlier I’m currently working on reproducing the hang. If theres nothing to learn from the dumps taken during the attack then this is the next best chance to get additional data. Might take a few days to setup everything and iterate on it until it reproduces.

reminds me of #31570.

Yes, this looks very similar

is it possible that SocketUplink gets disposed while there are pending receives?

We are interacting with the Socket and SocketAsyncEventArgs only from the thread running on the SynchronizationContext, but as far as I understand SocketAsyncEventArgs are not synchronized with anything, so of course there could be a receive happening when we decide to terminate the connection. We do not block SocketAsyncEventArgs receiving more data while processing incoming data, so if we see bad data we terminate the connection and the SocketAsyncEventArgs may just have received more data or a remote socket disconnect or whatever.

For me it’s very hard to guess the entire scenario based on the information we’ve got now, so it would be already very helpful if you could do at least the first repro step (simulating the attack on your full app in a test environment).

Alright, I was hoping to learn more from the dump than the stack traces, but if thats not possible I guess I have to try reproducing it. Might take a while.

Note that the issue may have gone with .NET 5 as a side effect of the performance refactors in the Unix socket engine.

We’ve updated our server code to build against .NET 5 after the attack was over. The spinning code still seems to be in the source but on a quick look I couldn’t find the lock the other thread was waiting in, so yes, maybe its gone.

I still will try to reproduce because I think this is a security problem (allowing DOS attacks against .NET Core socket based servers) and its important to understand it more. I’ll start trying to reproduce on .NET Core 3.1 since I know its a problem there, and move on when I have a repro.

Based on the dump, it looks like the completion of the ReceiveMessage P/Invoke triggers the disposal of SafeSocketHandle. As far as I understand, this could only happen if the underlying socket gets closed during an outgoing receive call.

@weltkante is it possible that SocketUplink gets disposed while there are pending receives?

Creating a repro would presumably require setting up a test environment and simulating an attack […] its probably better to work from the stack trace and dump

For me it’s very hard to guess the entire scenario based on the information we’ve got now, so it would be already very helpful if you could do at least the first repro step (simulating the attack on your full app in a test environment). Note that the issue may have gone with .NET 5 as a side effect of the performance refactors in the Unix socket engine. With such a repro it would be possible to determine if updating to .NET 5 fixes the issue, which would be a valuable piece of information for both you and us.

@tmds maybe you can spot something I couldn’t?

I cannot share the full source publically, but if you are just looking for the way the sockets are used here are the relevant classes:

accepting connections
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace AppNamespace
{
    public sealed class Acceptor : IDisposable
    {
        public static event Action<Action<Task>> FatalError;

        private SynchronizationContext mContext;
        private Socket mAcceptor;
        private SocketAsyncEventArgs mAcceptEvent;
        private Action<Socket> mOnAccept;
        private int mPort;

        public Acceptor(IPAddress ip, int port, Action<Socket> onAccept)
        {
            if (onAccept == null)
                throw new ArgumentNullException("onAccept");

            // create the endpoint early to validate the port
            var endpoint = new IPEndPoint(ip, port);

            mContext = Utils.RequireContext();
            mOnAccept = onAccept;
            mPort = port;

            mAcceptEvent = new SocketAsyncEventArgs();
            mAcceptEvent.Completed += HandleAccept;

            mAcceptor = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            try { mAcceptor.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true); }
            catch (SocketException e)
            {
                if (e.NativeErrorCode != (int)SocketError.OperationNotSupported)
                    throw;
            }
            mAcceptor.Bind(endpoint);
            mAcceptor.Listen((int)SocketOptionName.MaxConnections);

            CoreEvents.Log.AcceptorStarted(mPort);

            // we always post the first accept event so it doesn't invoke the callback before we return from the ctor
            if (!mAcceptor.AcceptAsync(mAcceptEvent))
                HandleAccept(mAcceptor, mAcceptEvent);
        }

        public void Dispose()
        {
            CoreEvents.Log.AcceptorStopped(mPort);

            mAcceptor.Dispose();
            mAcceptEvent.Dispose();
        }

        public int Port
        {
            get { return mPort; }
        }

        private void HandleAccept(object sender, SocketAsyncEventArgs e)
        {
            mContext.Post(x => HandleAccept(), null);
        }

        private void HandleAccept()
        {
        repeat:
            if (mAcceptEvent.SocketError == SocketError.Success)
            {
                var socket = mAcceptEvent.AcceptSocket;
                mAcceptEvent.AcceptSocket = null;
                CoreEvents.Log.AcceptorAccept(mPort, socket);
                mOnAccept(socket);
                if (!mAcceptor.AcceptAsync(mAcceptEvent))
                    goto repeat;
            }
            else
            {
                CoreEvents.Log.AcceptorError(mPort, mAcceptEvent.SocketError);

                // something bad happened, stop accepting
                //Dispose();

                // HACK: FailFast will crash too fast for the above message to appear in the log
                ConsoleAccess.Flush();

                var delays = new List<Task>();
                delays.Add(Task.CompletedTask);

                FatalError?.Invoke(task => delays.Add(task));

                Task.WhenAll(delays).ContinueWith(x => Environment.Exit(0));
            }
        }
    }
}
socket connection
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace AppNamespace
{
    public abstract class UplinkBase : IDisposable, INetworkWriterTarget
    {
        private static int mNextLogId;

        private SynchronizationContext mContext;
        private UplinkProtocol mProtocol;
        private int mLogId;
        private bool mDisposed;

        private NetworkReaderSource mInputController;
        private NetworkWriterControl mOutputController;
        private MemoryStream mWriteBuffer = new MemoryStream();
        private byte[] mWriteFrame = new byte[1 << 10];
        private bool mShutdownNotificationPosted;
        private bool mPostedReadyForInput;

        protected UplinkBase()
        {
            mLogId = Interlocked.Increment(ref mNextLogId);
            CoreEvents.Log.UplinkConnected(this);

            mContext = Utils.RequireContext();
            mContext.OperationStarted();

            mOutputController = new NetworkWriterControl(this);
            mInputController = new NetworkReaderSource();
        }

        public void Dispose()
        {
            if (!mDisposed)
            {
                mDisposed = true;

                try
                {
                    HandleDispose();

                    // if these asserts fire then base.HandleDispose was not called
                    Utils.Assert(mInputController.IsDisposed);
                    Utils.Assert(mOutputController.IsDisposed);

                    // this should be last so the protocol implementation sees a properly disposed uplink
                    if (mProtocol != null)
                    {
                        mProtocol.DetachInternal();
                        mProtocol.Dispose();
                    }
                }
                catch (Exception ex)
                {
                    Utils.ReportException(ex, true);
                }

                mContext.OperationCompleted();
            }
        }

        protected virtual void HandleDispose()
        {
            mInputController.Dispose();
            mOutputController.Dispose();
        }

        public bool IsDisposed
        {
            get { return mDisposed; }
        }

        public SynchronizationContext Context
        {
            get { return mContext; }
        }

        public int LogId
        {
            get { return mLogId; }
        }

        public UplinkProtocol SetProtocol(UplinkProtocol protocol, bool disposeOldProtocol)
        {
            if (mProtocol == protocol)
                return null;

            var oldProtocol = mProtocol;
            if (oldProtocol != null)
            {
                mProtocol = null;
                oldProtocol.DetachInternal();

                if (disposeOldProtocol)
                {
                    oldProtocol.Dispose();
                    oldProtocol = null;
                }
            }

            if (protocol != null)
            {
                mProtocol = protocol;
                protocol.AttachInternal(this);
                ReadyForInput();
            }

            return oldProtocol;
        }

        public NetworkWriter Write()
        {
            return mOutputController.Write();
        }

        public void Close(bool hard)
        {
            NotifyShutdown();
        }

        private void NotifyShutdown()
        {
            if (!mDisposed && !mShutdownNotificationPosted)
            {
                mShutdownNotificationPosted = true;
                mContext.OperationStarted();
                mContext.Post(x => ((UplinkBase)x).HandleShutdown(), this);
            }
        }

        private void HandleShutdown()
        {
            if (!mDisposed)
            {
                mInputController.SetComplete();
                using (var p = mInputController.Read())
                {
                    try { mProtocol.ProcessInputInternal(p); }
                    catch (ProtocolException ex)
                    {
                        CoreEvents.Log.UplinkProtocolException(mLogId, ex);
                        Dispose();
                    }
                }
            }

            mContext.OperationCompleted();
        }

        protected void AppendInput(byte[] buffer, int offset, int length)
        {
            CoreEvents.Log.UplinkReceivedData(LogId, length);
            CoreEvents.Log.UplinkReceivedDataPayload(LogId, buffer, 0, length);
            mInputController.Append(buffer, 0, length);
        }

        protected void CompleteInput()
        {
            CoreEvents.Log.UplinkRemoteShutdown(LogId);
            mInputController.SetComplete();
        }

        protected void ProcessInput()
        {
            if (!mPostedReadyForInput)
            {
                using (var p = mInputController.Read())
                {
                    try { mProtocol.ProcessInputInternal(p); }
                    catch (ProtocolException ex)
                    {
                        CoreEvents.Log.UplinkProtocolException(mLogId, ex);
                        Dispose();
                    }
                }
            }
        }

        #region Input

        public bool InputComplete
        {
            get { return mInputController.IsComplete; }
        }

        public int AvailableInput
        {
            get { return mInputController.Available; }
        }

        public void ReadyForInput()
        {
            if (!mPostedReadyForInput)
            {
                mPostedReadyForInput = true;
                mContext.OperationStarted();
                mContext.Post(ReadyForInputCallback, null);
            }
        }

        private void ReadyForInputCallback(object unused)
        {
            Utils.Assert(mPostedReadyForInput);
            mPostedReadyForInput = false;
            mContext.OperationCompleted();

            using (var p = mInputController.Read())
                mProtocol.ProcessInputInternal(p);
        }

        #endregion

        #region Output

        protected abstract void HandleSend(byte[] buffer, int offset, int length);

        public void BeginOutput()
        {
        }

        public ArraySegment<byte> RequestOutputPage(int sizeHint)
        {
            return new ArraySegment<byte>(mWriteFrame);
        }

        public void SubmitOutputPage(ArraySegment<byte> data)
        {
            mWriteBuffer.Write(data.Array, data.Offset, data.Count);
        }

        public void SuspendOutput()
        {
        }

        public void FlushOutput()
        {
            var buffer = mWriteBuffer.ToArray();
            HandleSend(buffer, 0, buffer.Length);
            mWriteBuffer.SetLength(0);
        }

        #endregion

        #region INetworkWriterTarget Members

        void INetworkWriterTarget.Begin()
        {
            BeginOutput();
        }

        void INetworkWriterTarget.Submit(ArraySegment<byte> data)
        {
            SubmitOutputPage(data);
        }

        void INetworkWriterTarget.Complete()
        {
            SuspendOutput();
        }

        #endregion  }
    }

    public sealed class SocketUplink : UplinkBase
    {
        public static SocketUplink Connect(IPEndPoint endpoint)
        {
            var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            socket.Connect(endpoint);
            return new SocketUplink(socket);
        }

        private Socket mSocket;
        private IPEndPoint mRemote;
        private SocketAsyncEventArgs mRecvEvent;
        private byte[] mInputBuffer;
        private SocketAsyncEventArgs mSendEvent;
        private List<ArraySegment<byte>> mSendData;
        private bool mSendActive;

        public SocketUplink(Socket socket)
        {
            if ((mSocket = socket) == null)
                throw new ArgumentNullException("socket");

            mRemote = (IPEndPoint)mSocket.RemoteEndPoint;

            mSendEvent = new SocketAsyncEventArgs();
            mSendEvent.Completed += HandleSendComplete;

            mRecvEvent = new SocketAsyncEventArgs();
            mRecvEvent.Completed += HandleRecvComplete;

            mInputBuffer = new byte[1 << 10];
            mRecvEvent.SetBuffer(mInputBuffer, 0, mInputBuffer.Length);

            Context.OperationStarted();
            if (!mSocket.ReceiveAsync(mRecvEvent))
                Context.Post(x => ((SocketUplink)x).HandleRecvComplete(), this);
        }

        public override string ToString()
        {
            return StringNC.Format("{{SocketUplink Id={0}, Remote={1}}}", LogId, mRemote);
        }

        protected override void HandleDispose()
        {
            mSocket.Dispose();
            mRecvEvent.Dispose();
            mSendEvent.Dispose();
            base.HandleDispose();
        }

        public IPEndPoint RemoteEndpoint
        {
            get { return mRemote; }
        }

        private void HandleRecvComplete(object sender, SocketAsyncEventArgs e)
        {
            Context.Post(x => ((SocketUplink)x).HandleRecvComplete(), this);
        }

        private void HandleRecvComplete()
        {
            bool process = false;

        repeat:
            Context.OperationCompleted();

            if (mRecvEvent.SocketError == SocketError.Success)
            {
                process = true;
                int length = mRecvEvent.BytesTransferred;
                if (length == 0)
                {
                    CompleteInput();
                }
                else
                {
                    AppendInput(mInputBuffer, 0, length);
                    mRecvEvent.SetBuffer(mInputBuffer, 0, mInputBuffer.Length);

                    Context.OperationStarted();
                    if (!mSocket.ReceiveAsync(mRecvEvent))
                        goto repeat;
                }
            }
            else
            {
                process = false;
                CoreEvents.Log.UplinkError(LogId, mRecvEvent.SocketError);
                Dispose();
            }

            if (process)
                ProcessInput();
        }

        protected override void HandleSend(byte[] buffer, int offset, int length)
        {
            try
            {
                if (mSendData == null)
                    mSendData = new List<ArraySegment<byte>>();

                mSendData.Add(new ArraySegment<byte>(buffer, offset, length));

                if (!mSendActive)
                {
                    mSendEvent.BufferList = mSendData;
                    mSendData = null;
                    mSendActive = true;

                    Context.OperationStarted();
                    if (!mSocket.SendAsync(mSendEvent))
                        HandleSendComplete();
                }
            }
            catch (SocketException ex)
            {
                CoreEvents.Log.UplinkError(LogId, ex.SocketErrorCode);
                Dispose();
            }
        }

        private void HandleSendComplete(object sender, SocketAsyncEventArgs e)
        {
            Context.Post(x => ((SocketUplink)x).HandleSendComplete(), this);
        }

        private void HandleSendComplete()
        {
        repeat:
            Context.OperationCompleted();
            mSendActive = false;

            if (mSendEvent.SocketError != SocketError.Success)
            {
                CoreEvents.Log.UplinkError(LogId, mSendEvent.SocketError);
                Dispose();
                return;
            }

            if (mSendData != null && mSendData.Count > 0)
            {
                mSendEvent.BufferList = mSendData;
                mSendData = null;
                mSendActive = true;

                Context.OperationStarted();
                if (!mSocket.SendAsync(mSendEvent))
                    goto repeat;
            }
        }
    }
}

The SynchronizationContext is a custom one running its work on a dedicated thread. If you wanted to know something else you need to be more specific; if you wanted something runnable, thats not possible without investing significant effort to reduce the server application, as I explained above its probably better to work from the stack trace and dump.