deno: Uncatchable `IO error: Broken pipe (os error 32)` when sending to much data through WebSocket

Hi guys !

Tested in Deno 1.20.5

So there are a lot of broken pipe problems around and a lot of them are workable via try or Promise.catch (from other posts I found);

But after further investigation in my own app, I drilled down an error that I couldn’t catch in any way. I identified that it’s happening when there is too much data sent in a WebSocket. Since it’s not catchable I think it’s a bug, but if you expect me to implement the verification then… (but that would be weird regarding the arbitrary limit that has nothing to do with Deno userland => see reprod)

Here is the reprod (check comment to toggle the bug)


// Typical web server
const server = await Deno.listen({ port: 8080 });

// Delaying the client connection to make sure server is up & running
setTimeout(() => {
  const sock = new WebSocket("ws://localhost:8080")

  sock.onopen = () => {
    console.log("clientop")
  }

  sock.onmessage = () => {
    console.log("recv");
    Deno.exit(0)
  }
}, 1000);

// Wait for a request
for await (const conn of server) {
  Deno.serveHttp(conn)
    .nextRequest()
    .then((event: Deno.RequestEvent | null) => {
      if (event) {
        const { socket, response } = Deno.upgradeWebSocket(event.request)
        
        // string of length Math.pow(2, 24) + 1 WILL WORK but with Math.pow(2, 24) + 2 we trigger the broken pipe
        // I have to admit I don't know why this limit exactly but... found it
        const content = new Array(Math.pow(2, 24) + 2).join(":")

        socket.onopen = () => {
          console.log("open");

          // Wait a bit to fullsend...
          setTimeout(() => {
            console.log("send");

            // try won't do anything, and wrapping in Promise to .catch won't either
            try {
              socket.send(content);
            } catch (error) {
              console.error("Catch")
            }
            
          }, 100);
        }
        
        event.respondWith(response);
      }
    })
}

I didn’t reproduce the bug using only the HTTP without upgrading

Tell me if you need more info, cheers

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 15 (6 by maintainers)

Most upvoted comments

I’m not sure I’d feel confident if the errors just went away. This is clearly some timing / size issue in the underlying rust code / network layer. That happens, code’s buggy, pipes break!

The fact that an underlying error cannot be caught by the calling code is concerning. If the calling code could catch these errors, then we could handle them in the application. I think reporting (and ultimately fixing these errors) should still be prioritized, but if underlying network problems leads to system crashes which cannot be avoided, then the platform is unusable.

irbull on Nov 3, 2022

Here’s a small reproducer that doesn’t depend on message sizes:

import { serve } from "https://deno.land/std@0.154.0/http/mod.ts";

const HTTP_PORT = 4321;

setTimeout(async () => {
  const c = await Deno.connect({ port: HTTP_PORT });
  const d = new TextEncoder().encode(`GET / HTTP/1.1\r
Connection: Upgrade\r
Upgrade: websocket\r
Sec-WebSocket-Key: dummy\r
Sec-WebSocket-Version: 13\r
\r\n\r\n`);
  await c.write(d);
  c.close()
}, 100);

function reqHandler(req: Request) {
  try {
    const { socket: ws, response } = Deno.upgradeWebSocket(req);
    ws.onopen = () => {
      try {
        // by the time we send our message the socket is closed, though
        // we have not noticed it yet and ws.readyState is still OPEN
        ws.send("something")
      } catch (e) {
        console.error(e)
      }
    }
    return response;
  } catch (e) {
    console.error(e)
    return new Response(null, { status: 500 })
  }
}

serve(reqHandler, { port: HTTP_PORT });

⋊> ~ deno run --allow-net repro.ts
Listening on http://localhost:4321/
Error 'IO error: Broken pipe (os error 32)' contains boxed error of unknown type:
  Io(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })
  Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }
error: Uncaught (in promise) Error: IO error: Broken pipe (os error 32)
⋊> ~ echo $status
1

I’ve encountered this with a Deno 1.25.0 server, but the snippet reproduces with 1.26 and current main as well.

Mrmaxmeier on Oct 6, 2022

I got the same error and am able to narrow it down to following code which reliably crashes opened with chromium, but not with firefox.

error: Uncaught (in promise) Error: IO error: Broken pipe (os error 32): Broken pipe (os error 32)

import { Application, Router } from 'https://deno.land/x/oak@v11.1.0/mod.ts'

const router = new Router()

router.get('/', ctx => {
  ctx.response.headers.set('content-type', 'text/html')
  ctx.response.body = `
  <!DOCTYPE html><script>
    const wsc = new WebSocket('ws://localhost:1111/ws', 'test')
    wsc.addEventListener('open', () => { console.log('connected') })
    wsc.addEventListener('error', (event) => { console.error(event) })
  </script>
  `
})
router.get('/ws', ctx => {
  try {
    const sock = ctx.upgrade();
    sock.onopen = () => {
      try {
        sock.send("{}")
        sock.send("{}")
      } catch(err) {
        console.log('onopen', err)
      }
    }
    sock.onerror = (e) => {
      console.log("socket errored:", e);
    }
    sock.onclose = () => {
      console.log("socket closed");
    }
  } catch(err) {
    console.log(err)
  }
});

const app = new Application()
app.use(router.routes());
app.use(router.allowedMethods())
app.listen({ port: 1111 })
console.log(`oak server running at http://localhost:1111/`)

deno 1.31.1
chromium 111.0.5563.64
firefox 110.0.1

alangecker on Mar 11, 2023

This seems to be solved in Deno’s 1.27 release. At least the above reproduction code does no longer throw an error for me after upgrading from 1.25 to 1.27. For reference, I tried this on Manjaro Linux and Ubuntu 20.04.

qkniep on Nov 2, 2022

Yes its that one (https://github.com/denoland/fastwebsockets/commit/93d8aff85020fc95f8a654a97497ddfcd8822990) and relavant commit in Deno: https://github.com/denoland/deno/commit/91dc6fa5f11f5621c21397c783ee899bfd9657ef

littledivy on Aug 12, 2023

Thanks for the report! This was fixed in the latest release (1.36.1). There was a race condition in the recv buffer. I’m going around closing issues related to this but please feel free to reopen if its still happening.

littledivy on Aug 12, 2023

Can confirm that the reproc no longer crashes. (M1, Mac OS 13)

I will see if I can reproduce the error outside of this.

Edit: I was able to reproduce the bug. If you take the first proposed reproc and change the Math.pow to 26, it will throw IO error. Wondering if this is intentional or not? If it is, it’s really annoying.

 const content = new Array(Math.pow(2, 26)).join(":")

lino-levan on Nov 3, 2022

I am also experiencing this error. I have a webserver that streams videos, and its practically unusable. I experience this crash whenever I try to skip a part of the video (or even when loading it sometimes). Here is what Deno says when it crashes:

error: Uncaught (in promise) BrokenPipe: Broken pipe (os error 32)
          numBytesWritten = await this.writer.write(data);
                            ^
    at async write (deno:ext/net/01_net.js:33:12)
    at async BufWriter.write (https://deno.land/std@0.87.0/io/bufio.ts:499:29)
    at async writeResponse (https://deno.land/std@0.87.0/http/_io.ts:273:15)
    at async ServerRequest.respond (https://deno.land/std@0.87.0/http/server.ts:89:7)

The whole thing is in a try-catch block but it does nothing.

Muha0644 on Aug 16, 2022