ksh: Test and build failures on arm7 Ubuntu box

Multiple errors occur when running shtests under Ubuntu 18.04 arm7 (32-bit hardware). dmesg output as a result of running shtests:

[1356568.629304] Alignment trap: not handling instruction ed910b00 at [<004b5960>]
[1356568.635251] Unhandled fault: alignment exception (0x011) at 0x00579fb5
[1356568.641904] pgd = ecb40000
[1356568.644684] [00579fb5] *pgd=b5326835
[1356568.656694] Alignment trap: not handling instruction ed910b00 at [<00423960>]
[1356568.662729] Unhandled fault: alignment exception (0x011) at 0x004e7fe6
[1356568.669282] pgd = ecb40000
[1356568.672107] [004e7fe6] *pgd=b3cdc835
[1356573.237540] Alignment trap: not handling instruction ed910b00 at [<00449960>]
[1356573.243570] Unhandled fault: alignment exception (0x011) at 0x0050dfb5
[1356573.250225] pgd = ecb20000
[1356573.253007] [0050dfb5] *pgd=b5b5e835
[1356573.266498] Alignment trap: not handling instruction ed910b00 at [<00431960>]
[1356573.272403] Unhandled fault: alignment exception (0x011) at 0x004f5fe6
[1356573.279071] pgd = ecb20000
[1356573.281923] [004f5fe6] *pgd=b6532835
[1356577.830056] Alignment trap: not handling instruction ed910b00 at [<0046b960>]
[1356577.836006] Unhandled fault: alignment exception (0x011) at 0x0052ffb5
[1356577.842708] pgd = eca5c000
[1356577.845616] [0052ffb5] *pgd=b5a76835
[1356577.856898] Alignment trap: not handling instruction ed910b00 at [<00457960>]
[1356577.862806] Unhandled fault: alignment exception (0x011) at 0x0051bfe6
[1356577.869492] pgd = eca5c000
[1356577.872270] [0051bfe6] *pgd=b536e835

failures from shtests:

$ bin/shtests
test io begins at 2021-04-04+09:41:32
	io.sh[243]: long seek not working
test io failed at 2021-04-04+09:41:36 with exit code 1 [ 130 tests 1 error ]
test io(C.UTF-8) begins at 2021-04-04+09:41:36
	io.sh[243]: long seek not working
test io(C.UTF-8) failed at 2021-04-04+09:41:40 with exit code 1 [ 130 tests 1 error ]
test io(shcomp) begins at 2021-04-04+09:41:40
	shcomp-io.ksh[long seek not working]:
test io(shcomp) failed at 2021-04-04+09:41:44 with exit code 1 [ 130 tests 1 error ]
/dev/shm/ksh93/src/cmd/ksh93/tests/variables.sh: line 994: 20419: Bus error
	variables.sh[1014]: Failure in making one or more special variables readonly in a subshell (exit status 263/BUS)
/dev/shm/ksh93/src/cmd/ksh93/tests/variables.sh: line 1018: 20421: Bus error
	variables.sh[1031]: One or more special variables leak out of a subshell (exit status 263/BUS)
test variables failed at 2021-04-04+09:46:34 with exit code 2 [ 166 tests 2 errors ]
test variables(C.UTF-8) begins at 2021-04-04+09:46:34
/dev/shm/ksh93/src/cmd/ksh93/tests/variables.sh: line 994: 22224: Bus error
	variables.sh[1014]: Failure in making one or more special variables readonly in a subshell (exit status 263/BUS)
/dev/shm/ksh93/src/cmd/ksh93/tests/variables.sh: line 1018: 22225: Bus error
	variables.sh[1031]: One or more special variables leak out of a subshell (exit status 263/BUS)
test variables(C.UTF-8) failed at 2021-04-04+09:46:39 with exit code 2 [ 166 tests 2 errors ]
test variables(shcomp) begins at 2021-04-04+09:46:39
/tmp/ksh93.shtests.10157.30876/shcomp-variables.ksh: line 994: 24032: Bus error
	shcomp-variables.ksh[Failure in making one or more special variables readonly in a subshell]: (exit status 263/BUS)
/tmp/ksh93.shtests.10157.30876/shcomp-variables.ksh: line 1018: 24033: Bus error
	shcomp-variables.ksh[One or more special variables leak out of a subshell]: (exit status 263/BUS)
test variables(shcomp) failed at 2021-04-04+09:46:43 with exit code 2 [ 166 tests 2 errors ]
Total errors: 9

And one time while rerunning shtests -p I got:

test path begins at 2021-04-04+10:21:01
	path.sh[569]: warning: 'command -x' test killed, probably due to lack of memory; skipping test
test path passed at 2021-04-04+10:21:09 [ 103 tests 0 errors ]
test variables begins at 2021-04-04+10:22:11
/home/hyenias/ksh93/src/cmd/ksh93/tests/variables.sh: line 994: 9472: Bus error
	variables.sh[1014]: Failure in making one or more special variables readonly in a subshell (exit status 263/BUS)
/home/hyenias/ksh93/src/cmd/ksh93/tests/variables.sh: line 1018: 9474: Bus error
	variables.sh[1031]: One or more special variables leak out of a subshell (exit status 263/BUS)
test variables failed at 2021-04-04+10:22:16 with exit code 2 [ 166 tests 2 errors ]
Total errors: 3

Then there is the following:

$ bin/shtests -p
shtests: shell not found: /dev/shm/ksh93/arch/linux.arm-64/bin/ksh
Specify a shell like:  KSH=path/to/ksh bin/shtests
$ ls arch
linux.arm

I played around with the following but I do not believe it is the right fix to correct the host type for an arm7 cpu.

diff --git a/bin/package b/bin/package
index 9b0ded8..d5b8298 100755
--- a/bin/package
+++ b/bin/package
@@ -1525,7 +1525,7 @@ use)	case $1 in
 			wow=`uname -i`
 			case $bit in
 			32)	case $HOSTTYPE in
-				*-64)	HOSTTYPE=${HOSTTYPE%-64} ;;
+				*-32)	HOSTTYPE=${HOSTTYPE%-32} ;;
 				esac
 				;;
 			64)	case $HOSTTYPE in

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 43

Commits related to this issue

Most upvoted comments

Here’s another test case, this time one where we get wrong output because of this rather than a report or crash about a misaligned access:

ksh -c 'SECONDS=10; echo SECONDS=$SECONDS; (SECONDS=20); sleep 5; echo SECONDS=$SECONDS'

Expected output: 10.000, 15.000 (possibly plus some small amount) Actual output: 10.000, 5.000 (possibly plus some small amount)

Edit: the actual output will depend on build options, but regardless, it was a problem that existed even in ksh 93v and presumably older versions too. With ksh 93v, I see 10.000, 4322232590209746483367325708968564867973327636718750000000000000000000000000000000000000000000000.000.

+2
hvdijk on Apr 7, 2021

I think the right fix might be as simple as

--- a/src/cmd/ksh93/sh/subshell.c
+++ b/src/cmd/ksh93/sh/subshell.c
@@ -392,9 +392,9 @@ static void nv_restore(struct subshell *sp)
 		}
 		mp->nvflag = np->nvflag|(flags&NV_MINIMAL);
 		if(nv_cover(mp))
-			nv_putval(mp, nv_getval(np),np->nvflag|NV_NOFREE|NV_RDONLY);
+			nv_putval(mp,nv_getval(np),NV_RDONLY);
 		else
-			mp->nvalue.cp = np->nvalue.cp;
+			mp->nvalue = np->nvalue;
 		if(nofree && np->nvfun && !np->nvfun->nofree)
 			free((char*)np->nvfun);
 		np->nvfun = 0;

We should not need NV_NOFREE to prevent freeing the previous value of mp, as there is an earlier _nv_unset(mp,NV_RDONLY|NV_CLONE); which should ensure there is no previous value if I understand correctly (warning: I probably don’t). We should not use anything in np->nvflag as nv_getval serialized it to a string.

Removing the nv_cover check entirely does break things:

LANG=en_US.UTF-8 ksh -c 'var=äëïöü; echo ${#var}; (LANG=C; echo ${#var}); echo ${#var}'

This should print 5, 10, 5. Removing the nv_cover check causes this to print 5, 10, 10.

+1
hvdijk on Apr 7, 2021

Using the ast-open-history repo I’ve determined the SECONDS bug was introduced in the 2009-08-18 version, specifically by this diff:

--- a/src/cmd/ksh93/sh/subshell.c
+++ b/src/cmd/ksh93/sh/subshell.c
@@ +346,11 -346,7 @@ static void nv_restore(struct subshell *sp)
 		mp->nvfun = np->nvfun;
 		mp->nvflag = np->nvflag;
 		if(nv_cover(mp))
+		{
+			nv_putval(mp, nv_getval(np),np->nvflag|NV_NOFREE);
+			if(!nv_isattr(np,NV_NOFREE))
+				nv_offattr(mp,NV_NOFREE);
+		}
-			nv_putval(mp, nv_getval(np),0);
 		else
 			mp->nvalue.cp = np->nvalue.cp;
 		np->nvfun = 0;

When I revert to the old code, the SECONDS test case works as expected on x86_64:

# Patch against ksh93u+m
--- a/src/cmd/ksh93/sh/subshell.c
+++ b/src/cmd/ksh93/sh/subshell.c
@@ -392,7 +392,7 @@ static void nv_restore(struct subshell *sp)
 		}
 		mp->nvflag = np->nvflag|(flags&NV_MINIMAL);
 		if(nv_cover(mp))
-			nv_putval(mp, nv_getval(np),np->nvflag|NV_NOFREE|NV_RDONLY);
+			nv_putval(mp, nv_getval(np), 0);
 		else
 			mp->nvalue.cp = np->nvalue.cp;
 		if(nofree && np->nvfun && !np->nvfun->nofree)

$ arch/linux.i386-64/bin/ksh -c 'SECONDS=10; echo SECONDS=$SECONDS; (SECONDS=20); sleep 5; echo SECONDS=$SECONDS'
SECONDS=10.000
SECONDS=15.000
+1
JohnoKing on Apr 7, 2021

Clearly that seek is very far beyond the end of the file. Why is this not expected to fail?

lseek() allows moving beyond the end of the file. I assume ksh intends to allow all seek offsets that lseek() allows.

+1
hvdijk on Apr 7, 2021

Then there is the following:

$ bin/shtests -p
shtests: shell not found: /dev/shm/ksh93/arch/linux.arm-64/bin/ksh
Specify a shell like:  KSH=path/to/ksh bin/shtests
$ ls arch
linux.arm

This bug is here: https://github.com/ksh93/ksh/blob/6b9703ffdd9a71c3fb7418ba7744fd734e39f454/bin/package#L2572-L2585

Tracing (which required exec 2>/dev/tty; set -x in the script itself, as stderr was redirected elsewhere) revealed the output of file includes a build hash like BuildID[sha1]=8934dd61657aac875c190535066466849687a56b:

+ bits=': ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=8934dd61657aac875c190535066466849687a56b, not stripped'

If that build hash happens to contain the string 64 then the code wrongly sets a 64-bit architecture on line 2580. This explains why the problem is intermittent: the build hash is different every time.

Hopefully a fix for this won’t break it on other systems. The fundamental problem is that the AT&T way of doing these things is horribly overcomplicated and unreliable.

+1
McDutchie on Apr 7, 2021

@McDutchie, I have created an other_boxes.txt file on the freebsd RPi box detailing access to my other ARM boxes if you are so inclined: Ubuntu 20.04 aarch64 and Ubuntu 18.04 armv7.

+1
hyenias on Apr 4, 2021
Read more comments on GitHub
← ksh: Tab completion menu causes inconsistent editor state in emacs
homeassistant-apsystems_ecur: 1.2.30b breaks integration →