zap: TLS - Random requests stalling & occasional segfault [raspi aarch64]
Yo. I’m still looking into this and am going to start digging into zap and facil to see if I can’t see what’s going on, but I wanted to drop what I’m seeing here first so I don’t lose it. I’m also going to try and set up a mini version of this in my efforts to narrow down what’s happening… I’ll be back here to update with more info, but until then…
(reproducible version @ https://github.com/Vemahk/zap-issue-64)
After plugging in TLS and getting the certs squared away, I immediately noticed maybe a third of the requests made to the server would stall for about 6 seconds, and there was an accompanying “UUID error”. The client gets the right response after the stall, but it looks like on the server side, it fails and retries internally:
192.168.0.100 - - [Sat, 06 Jan 2024 03:59:21 GMT] "GET / HTTP/1.1" 200 416b 577us
192.168.0.100 - - [Sat, 06 Jan 2024 03:59:27 GMT] "GET / HTTP/1.1" 200 416b 873us
192.168.0.100 - - [Sat, 06 Jan 2024 03:59:27 GMT] "GET /time HTTP/1.1" 200 33b 649us
UUID error: 0x810 (0)
No errno handler: Success
Above, the first request to / was replayed 6 seconds later. The UUID error I think is just late to the party, and the /time call snuck in.
I looked around in zap and found that the UUID error is probably coming from fio.c:3035, which looks to be something to do with actually writing to the response stream…
Some extra notes: it seems like which requests stall are relatively random. I can spam refresh and some make it through, some stall. Eventually, I segfaulted it by continuing to spam the refresh from the browser:
192.168.0.100 - - [Sat, 06 Jan 2024 03:49:28 GMT] "GET / HTTP/1.1" 200 416b 671us
Segmentation fault at address 0xfff06bf95fb0
???:?:?: 0xffff9d9d28d8 in ??? (libcrypto.so.3)
Unwind information for `libcrypto.so.3:0xffff9d9d28d8` was not available, trace may be incomplete
???:?:?: 0x0 in ??? (???)
run http-test: error: the following command terminated unexpectedly:
cd /home/vemahk/coding/zig/http-test/zig-out && /home/vemahk/coding/zig/http-test/zig-out/bin/http-test
Build Summary: 7/9 steps succeeded; 1 failed (disable with --summary none)
run transitive failure
└─ run http-test failure
error: the following build command failed with exit code 1:
/home/vemahk/coding/zig/http-test/zig-cache/o/f6f8278f7afc1bca5b5ecd21b7918d65/build /home/vemahk/.apps/zig/zig /home/vemahk/coding/zig/http-test /home/vemahk/coding/zig/http-test/zig-cache /home/vemahk/.cache/zig run
Which is when I thought I should come back here and start writing things down.
The segfault happened inside libcrypto, which makes me think that it could possibly be something screwy with the version of libcrypto that I have? Geeze, I hope not.
Anyway, gonna go back to investigating. I’ll come back to update with anything else I find.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (14 by maintainers)
Thx. I’m on my way to SYCL in Milan, so I’ll remain in CET. I doubt I’ll have much time next week as it’s a conference week. Just fyi
One more thing: openssl, the command, can be used as a server. You could try to check if you see similar behavior with that:
Serving the current dir:
(…and it’s bedtime for me )