websocket-sharp: Server with SSL connection won't accept connection

I have a server that works fine over regular connection. The client is a JavaScript script, using regular WebSockets. Now I change the server to work over secured connection, set the certificate, server starts and listens for connection fine. Now when I change the JS client to work over wss:// protocol then server would not accept the connection and dump this in log file

3/16/2016 12:15:21 AM|Fatal|<>c__DisplayClass60_0.<receiveRequest>b__0|WebSocketSharp.WebSocketException: An exception has occurred while reading an HTTP request/response. —> System.IO.EndOfStreamException: The header cannot be read from the data source. at WebSocketSharp.HttpBase.<>c__DisplayClass13_0.<readHeaders>b__0(Int32 i) at WebSocketSharp.Ext.EqualsWith(Int32 value, Char c, Action1 action) at WebSocketSharp.HttpBase.readHeaders(Stream stream, Int32 maxLength) at WebSocketSharp.HttpBase.Read[T](Stream stream, Func2 parser, Int32 millisecondsTimeout) — End of inner exception stack trace — at WebSocketSharp.HttpBase.Read[T](Stream stream, Func`2 parser, Int32 millisecondsTimeout) at WebSocketSharp.HttpRequest.Read(Stream stream, Int32 millisecondsTimeout) at WebSocketSharp.Net.WebSockets.TcpListenerWebSocketContext…ctor(TcpClient tcpClient, String protocol, Boolean secure, ServerSslConfiguration sslConfig, Logger logger) at WebSocketSharp.Ext.GetWebSocketContext(TcpClient tcpClient, String protocol, Boolean secure, ServerSslConfiguration sslConfig, Logger logger) at WebSocketSharp.Server.WebSocketServer.<>c__DisplayClass60_0.<receiveRequest>b__0(Object state)

Any idea how this can be fixed?