shellspec: Shellspec can not collect data with the profile option enabled

Hello, I try to use the profile option, but the formatter get no proper data.

# shellspec --profile spec/em_spec.sh
Running: /bin/bash [bash 5.1.4(1)-release]
............/lib/shellspec/lib/general.sh: line 538: * 10: syntax error: operand expected (error token is "* 10")
Aborted with status code [executor: 0] [reporter: 0] [error handler: 102]
Fatal error occurred, terminated with exit status 102.

This is a problem in shellspec_shift10() because “$2” is empty. Therefore, I try find the way the function will be called.

But this ended because I can not find the code where “$time_real” filled with proper values. See /lib/shellspec/lib/libexec/reporter/profiler_formatter.sh in profiler_end() in the line

  read_profiler callback "$profiler_tick_total" "$time_real" \
    < "$SHELLSPEC_PROFILER_LOG" \
    > "$SHELLSPEC_PROFILER_REPORT"

The time_real is empty, is ‘’. If I assign “0” to the empty “$time_real” I get no useful report but Shellspec finished without an error.

My question, where the variable “$time_real” came from?

Sorry, I can not find, what is going wrong, Lutz

p.s. I use shellspec 0.28.1 dist version.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 34 (19 by maintainers)

Commits related to this issue

Most upvoted comments

Hello Koichi Nakashima, nice to see IBM make AIX available via a cloud service for developers. I was not aware of this option to get access to a IBM AIX system.

You are right, the KSH88 is very old and I do not know the reason IBM still use KSH88 as a default shell. I will start testing in a real AIX system next week, based on the github master.

Last week I make MKSH ([mksh @(#)MIRBSD KSH R59 2020/10/31]) available on AIX and shellspec works well, I can not find errors based on the shellspec tests, the tests are passed only 47 are skipped.

AIX 7.2

shellspec -s /home/j123450/bin/mksh spec

Running: /home/j123450/bin/mksh [mksh @(#)MIRBSD KSH R59 2020/10/31] Finished in 96.09 seconds (user 27.02 seconds, sys 3.18 seconds) 1619 examples, 0 failures, 47 skips (muted 33 skips)

Linux SLES 12/15 and RHEL 9 (I think)

shellspec -s /bin/mksh spec

Running: /bin/mksh [mksh @(#)MIRBSD KSH R50 2015/04/19 SLES] Finished in 43.89 seconds (user 38.07 seconds, sys 11.15 seconds) 1619 examples, 0 failures, 45 skips (muted 33 skips)

MacOS

shellspec -s /bin/mksh spec

Running: /bin/mksh [mksh @(#)MIRBSD KSH R59 2020/10/31] Finished in 69.46 seconds (user 51.00 seconds, sys 23.35 seconds) 1619 examples, 0 failures, 45 skips (muted 33 skips)

I compare the skipped tests to see the differences, for the MKSH. And I compare the KSH93 based tests on AIX and Linux.

Thank you for taking the time to look into this issue.

It’s good to see you back in action

Thanks for all your help/support. I’ll be back next week, Lutz

  • Failure of UseFD (ksh93)
    • Fixed in 2b7216ba40b7604833b60704a5fddc8967468cf7

However, it is not clear whether this fix has completely fixed the issue.

Fixes were added to master.

  1. Reasons for not getting the time
  2. echo: write to 1 failed (ksh93)
  3. Failure of UseFD (ksh93)
    • pending (This issue has not been fixed)

The time should be measured correctly and the profiler should work correctly.

# LC_ALL=DE_DE bash ./shellspec --profile --no-banner spec/semver_spec.sh
Running: /usr/bin/bash [bash 5.2.15(1)-release]
...............................................

Finished in 2.113 seconds (user: 0.457s, sys: 0.031s) [time: bash-builtin]
47 examples, 0 failures

# Top 10 slowest examples of the 47 examples
#   1 0.0441 spec/semver_spec.sh:56-60
#   2 0.0430 spec/semver_spec.sh:88-91
#   3 0.0393 spec/semver_spec.sh:88-91
#   4 0.0382 spec/semver_spec.sh:49-52
#   5 0.0378 spec/semver_spec.sh:49-52
#   6 0.0361 spec/semver_spec.sh:88-91
#   7 0.0321 spec/semver_spec.sh:25-28
#   8 0.0315 spec/semver_spec.sh:88-91
#   9 0.0313 spec/semver_spec.sh:88-91
#  10 0.0313 spec/semver_spec.sh:88-91

Bonus: The reimplemented locale-independent shellspec-time.sh can be run alone.

# LC_ALL=DE_DE libexec/shellspec-time.sh sleep 1
real:1.10 user:0.00 sys:0.00 type:ksh88-builtin

# LC_ALL=DE_DE ksh93 libexec/shellspec-time.sh sleep 1
real:1.000 user:0.000 sys:0.000 type:ksh93-builtin

# LC_ALL=DE_DE bash libexec/shellspec-time.sh locale
LANG=en_US
LC_COLLATE="DE_DE"
LC_CTYPE="DE_DE"
LC_MONETARY="DE_DE"
LC_NUMERIC="DE_DE"
LC_TIME="DE_DE"
LC_MESSAGES="DE_DE"
LC_ALL=DE_DE
real:0.004 user:0.001 sys:0.001 type:bash-builtin

Hello, based on the shellepc master (45b02e0) and “LANG=C” the tests are passed successfully with the KSH93 also.

LANG=C shellspec -s /bin/ksh93 spec
Running: /bin/ksh93 [ksh Version M 93t+ 2009-05-01]
Finished in 50.40 seconds (user 0.05 seconds, sys 0.01 seconds)
1696 examples, 0 failures, 56 skips (muted 39 skips)


Notable examples: (Listed here do not affect your suite's status)

shellspec spec/core/matchers/be/empty_spec.sh:123 # 1) core/matchers/be/empty.sh be empty directory matcher when enabled failglob in bash matches SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:114 # 2) core/matchers/be/stat.sh be symlink matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:142 # 3) core/matchers/be/stat.sh be pipe matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:169 # 4) core/matchers/be/stat.sh be socket matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:196 # 5) core/matchers/be/stat.sh be readable matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:225 # 6) core/matchers/be/stat.sh be writable matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:253 # 7) core/matchers/be/stat.sh be executable matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:281 # 8) core/matchers/be/stat.sh be block device matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:309 # 9) core/matchers/be/stat.sh be character device matcher example SKIPPED
shellspec spec/core/matchers/have/stat_spec.sh:14 # 10) core/matchers/have/stat.sh has setgid matcher example SKIPPED
shellspec spec/core/matchers/have/stat_spec.sh:43 # 11) core/matchers/have/stat.sh has setuid matcher example SKIPPED
shellspec spec/core/matchers/have/stat_spec.sh:72 # 12) core/matchers/have/stat.sh should have setgid matcher example SKIPPED
shellspec spec/core/matchers/have/stat_spec.sh:101 # 13) core/matchers/have/stat.sh should have setuid matcher example SKIPPED
shellspec spec/core/subjects/fd_spec.sh:18 # 14) core/subjects/fd.sh fd subject example example SKIPPED
shellspec spec/core/utils_spec.sh:115 # 15) core/utils.sh shellspec_shopt() shopt option sets option SKIPPED
shellspec spec/install_spec.sh:45 # 16) ./install.sh prompt() when not pre answered returns the answer SKIPPED
shellspec spec/install_spec.sh:76 # 17) ./install.sh confirm() when not pre answered returns the answer SKIPPED

The tests with the KSH93 looks similar to the BASH (version 5.2!) tests now.

LANG=C shellspec -s /bin/bash spec
Running: /bin/bash [bash 5.2.15(1)-release]
Finished in 114.48 seconds (user 32.27 seconds, sys 4.44 seconds)
1696 examples, 0 failures, 49 skips (muted 36 skips)


Notable examples: (Listed here do not affect your suite's status)

shellspec spec/core/matchers/be/stat_spec.sh:114 # 1) core/matchers/be/stat.sh be symlink matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:142 # 2) core/matchers/be/stat.sh be pipe matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:169 # 3) core/matchers/be/stat.sh be socket matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:196 # 4) core/matchers/be/stat.sh be readable matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:225 # 5) core/matchers/be/stat.sh be writable matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:253 # 6) core/matchers/be/stat.sh be executable matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:281 # 7) core/matchers/be/stat.sh be block device matcher example SKIPPED
shellspec spec/core/matchers/be/stat_spec.sh:309 # 8) core/matchers/be/stat.sh be character device matcher example SKIPPED
shellspec spec/core/matchers/have/stat_spec.sh:14 # 9) core/matchers/have/stat.sh has setgid matcher example SKIPPED
shellspec spec/core/matchers/have/stat_spec.sh:43 # 10) core/matchers/have/stat.sh has setuid matcher example SKIPPED
shellspec spec/core/matchers/have/stat_spec.sh:72 # 11) core/matchers/have/stat.sh should have setgid matcher example SKIPPED
shellspec spec/core/matchers/have/stat_spec.sh:101 # 12) core/matchers/have/stat.sh should have setuid matcher example SKIPPED
shellspec spec/core/subjects/fd_spec.sh:18 # 13) core/subjects/fd.sh fd subject example example SKIPPED

Based on “LANG=C” no NLSPATH is found in my AIX environment, therefore no messages are translated and the expected message text is found. The runtime time values are available also.

Have a nice weekend, Lutz

p.s. Nice to see, how easy a (inappropriate) LANG setting confuse lots of scripts. On the other hand, I know this problem, some of the used Java environments/application switch to “LANG=de_DE” or “LANG=de_DE.UTF-8”, therefore some messages are in german and somtimes partial messages are in german only. However, I was not aware that “LANG=en_US” does not necessarily use the default message text.

Hello, nice to see a german language pack create german messages, the NLS packages are not installated in our environments, the “en_US” language is installed only, by default.

bash-5.2# LC_ALL=DE_DE env time -p sleep 0

Real   0,00
Benutzer 0,00
System 0,00

In general I use “en_US” or “en_US.UTF-8”. This fit well (I thought) in the past, but I try do some tests based on “C”, tomorrow.

From my point of view the “/bin/time -p” problem can not fixed, as long as the NLS messages are used, the messages can not handled properly. The fix from LukeSavefrogs fix the “.”/“,” problem, but if a NLS package is available, the messages are not properly handled, because values are not properly assigned.

But “C” is a good idea.

Lutz

Hello, thanks for your answer. The used AIX environment is similar.

# oslevel
7.2.0.0
# oslevel -s
7200-05-05-2246

Based on your sample to gather some more information, my output looks similar, but the test failed.

# /bin/ksh93 ./shellspec -s /bin/ksh93 spec/libexec/prechecker_spec.sh
Running: /bin/ksh93 [ksh Version M 93t+ 2009-05-01]
.total 0
dr-x------    1 j123450  user            0 May 29 19:38 .
dr-xr-xr-x    1 j123450  user            0 May 29 19:38 ..
c---------    1 j123450  user        1,  0 May 29 19:37 0
c---------    1 j123450  user        1,  0 May 29 19:37 1
--w-------    1 j123450  user            0 May 29 19:38 2
lr-xr-xr-x    1 j123450  user            0 May 29 19:38 3 -> /proc/17301990/fd/
FF............../home/j123450/shellspec-master/libexec/shellspec-executor.sh[111]: echo: write to 1 failed [Bad file number]
/home/j123450/shellspec-master/libexec/shellspec-runner.sh[208]: echo: write to 1 failed [Bad file number]

The KSH93 version number is the same, but the files size is not the same.

# ls -la /bin/ksh93
-r-xr-xr-x    2 bin      bin         2387302 Aug 26 2022  /bin/ksh93

# shasum /bin/ksh93
a7072e229502350104cce7c136e5c7c063a21a01  /bin/ksh93

The only ugly environment variables are set/used:

TZ=MEZ-1MESZ,M3.5.0/02:00:00,M10.5.0/03:00:00
LANG=en_US
NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat
LC_ALL is not set

And “/opt/freeware/bin” is in the path, to make some Linux/GNU tools available in the AIX environment.

The used LANG and NLSPATH variables are the reason for the problem with the “time -p” command output, but this is an other storry. But you can check your NLSPATH to find the translated messages, in your AIX environment.

Finished in ? seconds (user ? seconds, sys ? seconds)
1696 examples, 1 failure, 56 skips (muted 39 skips)

If I check the NLSPATH, I find messages for en_US only.

# ls -la /usr/lib/nls/msg/*/time.cat
-rw-r--r--    1 bin      bin             252 Jun 01 2015  /usr/lib/nls/msg/en.8859-15/time.cat
-rw-r--r--    1 bin      bin             252 Jun 01 2015  /usr/lib/nls/msg/en.ISO8859-1/time.cat
-rw-r--r--    1 bin      bin             252 Jun 01 2015  /usr/lib/nls/msg/en.ISO8859-15/time.cat
-rw-r--r--    1 bin      bin             252 Jun 01 2015  /usr/lib/nls/msg/en_US.8859-15/time.cat
-rw-r--r--    1 bin      bin             252 Jun 01 2015  /usr/lib/nls/msg/en_US.ISO8859-1/time.cat
-rw-r--r--    1 bin      bin             252 Jun 01 2015  /usr/lib/nls/msg/en_US/time.cat

A funny test result, if I do this (based on your last sample).

# LC_ALL=DE_DE /bin/ksh93 ./shellspec -s /bin/ksh93 spec/libexec/prechecker_spec.sh
Running: /bin/ksh93 [ksh Version M 93t+ 2009-05-01]
.total 0
dr-x------    1 j949350  siuser            0 May 29 19:51 .
dr-xr-xr-x    1 j949350  siuser            0 May 29 19:51 ..
c---------    1 j949350  siuser        1,  0 May 29 19:46 0
c---------    1 j949350  siuser        1,  0 May 29 19:46 1
--w-------    1 j949350  siuser            0 May 29 19:51 2
lr-xr-xr-x    1 j949350  siuser            0 May 29 19:51 3 -> /proc/18612700/fd/
................

Finished in 0.59 seconds (user 0.00 seconds, sys 0.00 seconds)
17 examples, 0 failures

Every think works well. And “LANG=EN_US” work well too, but “LANG=en_US” does not. With the suggest LANG “DE_DE” or “de_DE” the “time -p” looks like expected, too.

# LANG=DE_DE /bin/time -p sleep 1
real   1.05
user   0.00
sys    0.00

# LANG=en_US /bin/time -p sleep 1
Real   1.00
User   0.00
System 0.00

It is a AIX LANG environment setting/handling problem? Lutz

The BASH will simulate it (on AIX), I think.

# test -e /dev/stdin; echo $?
0

But MKSH and KSH test the dev filesystem.

# test -e /dev/stdin; echo $?
1

But the file does not exist.

Nice to see

Very nice indeed 😳

Hello (back to MKSH), tests with MKSH on AIX (compared to Linux and MacOS) also skips the following tests, because “/dev/stdin” does not exist.

  13) ./install.sh prompt() when not pre answered returns the answer
      13.1) SKIPPED: /dev/stdin not exists

            # spec/install_spec.sh:46

  14) ./install.sh confirm() when not pre answered returns the answer
      14.1) SKIPPED: /dev/stdin not exists

            # spec/install_spec.sh:77

Finished in 124.48 seconds (user 27.90 seconds, sys 3.05 seconds)
1619 examples, 0 failures, 47 skips (muted 33 skips)

shellspec spec/install_spec.sh:45 # 13) ./install.sh prompt() when not pre answered returns the answer SKIPPED
shellspec spec/install_spec.sh:76 # 14) ./install.sh confirm() when not pre answered returns the answer SKIPPED

The BASH will simulate it (on AIX), I think.

# test -e /dev/stdin; echo $?
0

But MKSH and KSH test the dev filesystem.

# test -e /dev/stdin; echo $?
1

But the file does not exist.

Nice to see, Lutz

@lutzmad

I have registered with IBM cloud and modified it to work with ksh93t+ on AIX 7.2 and ksh93u+ on AIX 7.3. The fixes are included in the latest master. Please note that the problems that did not work with ksh93 are not related to locale issues. This is a bug related to the virtual subshell of ksh93.

There are no plans to support sh (Version M-11/16/88f). This ksh88 is very old (released around 1995?), and although AIX seems to claim that it is POSIX compliant, it probably has bugs in its errexit-related behaviour. It is quite hard to support this.

Thank you for taking the time to look into this issue.

It’s good to see you back in action 😄

@lutzmad

I have registered with IBM cloud and modified it to work with ksh93t+ on AIX 7.2 and ksh93u+ on AIX 7.3. The fixes are included in the latest master. Please note that the problems that did not work with ksh93 are not related to locale issues. This is a bug related to the virtual subshell of ksh93.

There are no plans to support sh (Version M-11/16/88f). This ksh88 is very old (released around 1995?), and although AIX seems to claim that it is POSIX compliant, it probably has bugs in its errexit-related behaviour. It is quite hard to support this.

Thanks for the report. It is very interesting.

When I wrote these codes, I apparently did not know about the time command. ShellSpec will use time -p in preference to the external command time if it is present.

https://github.com/shellspec/shellspec/blob/166e4f0523c958033b6baea5c1151c432419f5e6/shellspec#L414-L423

In ksh, the time command is an internal command, but in some cases an external command is invoked. (sigh)

(Invoke internal command)
$ time sleep 1

real    0m1.00s
user    0m0.00s
sys     0m0.00s

(Attempts to invoke internal commands but no -p option)
$ time -p sleep 1
ksh: -p:  not found

real    0m0.00s
user    0m0.00s
sys     0m0.00s

(Invoke external command)
$ t="time -p"
$ $t sleep 1

real 1.00
user 0.00
sys 0.00

The report shows that /usr/bin/time on AIX uses locale-dependent decimal point symbol. The same applies to bash. My understanding is that NLSPATH is the path to the message catalog and I assume it has nothing to do with the decimal point symbol. This issue will be fixed in merge #281. It will be merged soon.

However, there is another issue. Implementations of the various time commands can change their format via environment variables. The current implementation does not work well when a non-default format is specified. Eventually I am considering addressing this issue.

Hello, no problem.

However something that i do not understand is how is this possible… The documentation for AIX time command clearly shows that the output is the same as linux:

I will add additional “time” command output to the issue and I try to find out which “time” command is used. Lutz

@LukeSavefrogs Thanks for your help. And I’m sorry for neglecting you guys for so long.

Perhaps #281 can be merged without problems. ShellSpec must work in all locales. I did not pay enough attention to the locale when I wrote that code. Shells and commands have historically respected locales. However, not all.

The time command is built into the shell in some shells and not in others. In ksh, the time command invokes the built-in shell when no options are specified, but invokes an external command when options are specified. That is very confusing.

bash is usually invoked by the shell built-in time command. Its behavior differs from that of the external (GNU) time command.

$ bash -c 'export LC_ALL=de_DE.UTF-8; time -p true'
real 0,00
user 0,00
sys 0,00

$ bash -c 'export LC_ALL=de_DE.UTF-8; env time -p true'
real 0.00
user 0.00
sys 0.00

Hello Luca, I try to add your suggested modification tomorrow. I used my MacOS, but Shellspec collect profile data as expected in the MacOS environment. Lutz

p.s. I use the Bash shell in the Linux (and AIX) environment. The LANG is not C, we use de_DE with/without UTF-8 in general.

In a MacOS environment, the time output does not depend to the used LANG.

$ export LANG=de_DE (or en_US)
$ time -p id
uid= ...
real 0.00
user 0.00
sys 0.00

In the AIX ksh environment (my default shell, but I like the ksh, sorry), the time output honour the used LANG. This is the time command used by Shellspec, I think.

# export LANG=en_US
# /usr/bin/time -p id
uid= ...

Real   0.00
User   0.00
System 0.00
# export LANG=de_DE
# /usr/bin/time -p id
uid= ...

real   0,01
user   0,00
sys    0,00

And in the Linux and AIX bash environment, also.

# export LANG=de_DE
# time -p id
uid= ...
real 0,00
user 0,00
sys 0,00
# export LANG=en_US
# time -p id
uid= ...
real 0.00
user 0.00
sys 0.00