ksh: Lauching ksh with RLIMIT_NOFILE with a value exceeding system's INT_MAX leads to crash EXC_BAD_ACCESS (SIGSEGV)

If ksh is launched with RLIMIT_NOFILE set to a value that exceeds the maximum (positive) value an int can hold, then ksh crashes with a segmentation fault with EXC_BAD_ACCESS (SIGSEGV).

Consider the following trivial reproducer. I am on macos M1 and on my system INT_MAX value is 2147483647. So let’s first launch ksh with RLIMIT_NOFILE set to this 2147483647 value. This run should pass without any issues:

(RLIMIT_NOFILE is set using ulimit -n <val> command):

(ulimit -n 2147483647 && /bin/ksh --version)

This executes correctly and prints out the ksh version as expected:

version         sh (AT&T Research) 93u+ 2012-08-01

Now let’s increment the ulimit value by 1 so that it exceeds the INT_MAX value and so let’s rerun it with 2147483648:

(ulimit -n 2147483648 && /bin/ksh --version)

Running this results in a segmentation fault in ksh:

segmentation fault  ( ulimit -n 2147483648 && /bin/ksh --version; )

Here’s the relevant parts from the generated crash logs:

Process:               ksh [12345]
Path:                  /bin/ksh
Identifier:            ksh
Version:               ???
Code Type:             ARM-64 (Native)
Parent Process:        java [53231]
Responsible:           Terminal [234]

OS Version:            macOS 13.0.1 (22A400)
...

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Codes:       0x0000000000000001, 0x0000000000000000

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [12345]

...

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   ksh                           	       0x104321dac sh_ioinit + 76
1   ksh                           	       0x104321d94 sh_ioinit + 52
2   ksh                           	       0x10431e898 sh_init + 524
3   ksh                           	       0x104308798 sh_main + 72
4   dyld                          	       0x191aebe50 start + 2544


Thread 0 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000000   x1: 0x0000000000000000   x2: 0x000000000000000b   x3: 0x0000000000000000
    x4: 0x0000000000000000   x5: 0x0000600002a10474   x6: 0x000000000000000a   x7: 0x0000000000000001
    x8: 0x00000001043fb680   x9: 0x0000000000000000  x10: 0x0000600001d10000  x11: 0x00000000000000c0
   x12: 0x0000000000000055  x13: 0x00000000000007fb  x14: 0x000000008002a7fb  x15: 0x000000008002a7fb
   x16: 0x0000000191dd77cc  x17: 0x00000001f2328c98  x18: 0x0000000000000000  x19: 0x00000001043fb878
   x20: 0x00000001043f8da0  x21: 0x000000016bb07480  x22: 0x0000000104401de8  x23: 0x0000000000000002
   x24: 0x00000001ed9e83c0  x25: 0x0000000000000000  x26: 0x0000000000000000  x27: 0x0000000000000000
   x28: 0x0000000000000000   fp: 0x000000016bb06b40   lr: 0x0000000104321d94
    sp: 0x000000016bb06b20   pc: 0x0000000104321dac cpsr: 0x20001000
   far: 0x0000000000000000  esr: 0x92000046 (Data Abort) byte write Translation fault

As per the man pages of getrlimit and setrlimit it’s a valid case to have RLIMIT_NOFILE values which don’t fit the int range.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 16

Commits related to this issue

init: Fix crash on huge/tiny 'ulimit -n' (OPEN_MAX, RLIMIT_NOFILE) Reproducers: $ (ulimit -n 2147483648; ksh --version) -ksh: 22206: Memory fault Memory fault $ (ulimit -n 8; ksh --version) -ksh: 22... — committed to ksh93/ksh by McDutchie a year ago

Most upvoted comments

I’ll attempt to build this project locally, along with the patch, just to give it a try today.

I was able to build this project locally. My previous attempts at building ksh from the original repo weren’t successful. So, thank you for making it straightforward to build this project.

After building the project, I applied @JohnoKing’s patch and rebuilt it. As expected it did address the segmentation faults for the large values (outside INT_MAX) but it continued to seg fault for smaller values (like 15 or 16):

(ulimit -n 15 && arch/darwin.arm64-64/bin/ksh --version)
zsh: segmentation fault  ( ulimit -n 15 && arch/darwin.arm64-64/bin/ksh --version; )

I then did a small change, on top of @JohnoKing’s patch, to check the return value of sh_iovalidfd before trying to deal with the sh.sftable. That change looks like:

diff --git a/src/cmd/ksh93/sh/io.c b/src/cmd/ksh93/sh/io.c
index ca3c354e..a0a7fd5a 100644
--- a/src/cmd/ksh93/sh/io.c
+++ b/src/cmd/ksh93/sh/io.c
@@ -453,7 +453,10 @@ void sh_ioinit(void)
 {
        filemapsize = 8;
        filemap = (struct fdsave*)sh_malloc(filemapsize*sizeof(struct fdsave));
-       sh_iovalidfd(16);
+       if (!sh_iovalidfd(16))
+    {
+        error(ERROR_exit(1), "Not enough open file limit");
+    }

After rebuilding the project, this now works for both the large values as well as smaller values and no longer seg faults:

(ulimit -n 9223372036854775807 && arch/darwin.arm64-64/bin/ksh --version)
  version         sh (AT&T Research) 93u+m/1.1.0-alpha+b865478b/MOD 2022-10-31

$> (ulimit -n 15 && arch/darwin.arm64-64/bin/ksh --version)
ksh: Not enough open file limit
$> echo $?
1

Of course, this was just a crude way to quickly check this code path and my patch in itself may not be the right way to do it. Furthermore, I see that there are a lot more places where sh_iovalidfd gets called and those places too may need a review and fix.

jaikiran on Dec 23, 2022

So far I haven’t been able to reproduce the segfault on Linux, FreeBSD or illumos, and I don’t have a Mac to test ksh on. Linux rejects excessive RLIMIT_NOFILE values and FreeBSD silently caps it at 58014 without throwing an error. Interestingly, illumos sets values close to and above INT_MAX as ‘unlimited’ while setting OPEN_MAX to zero, and that works without a segfault (although attempting to run any external programs results in OOM crashes). Below is a patch that might fix the crash. It simply extends the max variable to a long to avoid overflow issues (astconf_long is a macro that usually returns long values from sysconf(3), or strtol in the unlikely fallback).

Patch

diff --git a/src/cmd/ksh93/sh/io.c b/src/cmd/ksh93/sh/io.c
index 8521c9ab1..ca3c354e1 100644
--- a/src/cmd/ksh93/sh/io.c
+++ b/src/cmd/ksh93/sh/io.c
@@ -388,13 +388,14 @@ static short		filemapsize;
 int  sh_iovalidfd(int fd)
 {
 	Sfio_t		**sftable = sh.sftable;
-	int		max,n, **fdptrs = sh.fdptrs;
+	int		n, **fdptrs = sh.fdptrs;
 	unsigned char	*fdstatus = sh.fdstatus;
+	long		max;
 	if(fd<0)
 		return(0);
 	if(fd < sh.lim.open_max)
 		return(1);
-	max = (int)astconf_long(CONF_OPEN_MAX);
+	max = astconf_long(CONF_OPEN_MAX);
 	if(fd >= max)
 	{
 		errno = EBADF;

JohnoKing on Dec 22, 2022

It happens on an Intel Mac as well.

posguy99 on Dec 22, 2022