nushell: scripts can't operate as part of a pipeline

Describe the bug

with a simple script that returns a simple data structure, it does not appear as true data in the REPL 🤔

How to reproduce

in foo.nu, put the following

let prs = [
    [foo bar];
    [1 2]
    [3 4]
]

$prs

run the script with nu foo.nu => this will show the table as expected
run the script as nu foo.nu | columns

Error: nu::shell::only_supports_this_input_type

  × Input type not supported.
   ╭─[entry #4:1:1]
 1 │ nu foo.nu | columns
   · ─┬          ───┬───
   ·  │             ╰── only record or table input data is supported
   ·  ╰── input type: raw data
   ╰────

run the script as nu foo.nu | get 0

Error: nu::shell::io_error

  × I/O error
  help: can't follow stream paths

and finally nu foo.nu | select foo does not give anything…

👉 looks like foo.nu is returning some strange output 🤔

Expected behavior

i expected nu foo.nu to return the table as-is, allowing all the classic operations on tables.

Note i get the exact same behaviour with the following more explicit script
def main [] {
    let prs = [
        [foo bar];
        [1 2]
        [3 4]
    ]

    return $prs
}

Screenshots

No response

Configuration

key	value
version	0.80.1
branch	main
commit_hash	acd2fe8c51eba9f3a707d596b1fae10d2986936f
build_os	linux-x86_64
build_target	x86_64-unknown-linux-gnu
rust_version	rustc 1.68.2 (9eb3afe9e 2023-03-27)
rust_channel	1.68.2-x86_64-unknown-linux-gnu
cargo_version	cargo 1.68.2 (6feb7c9cf 2023-03-26)
build_time	2023-06-03 13:10:15 +02:00
build_rust_channel	release
features	default, sqlite, trash, which, zip
installed_plugins

Additional context

No response

About this issue

Original URL
State: open
Created a year ago
Reactions: 4
Comments: 17 (10 by maintainers)

Commits related to this issue

make "can't follow stream paths" error a bit better (#10569) related to - https://github.com/nushell/nushell/issues/9373 - https://github.com/nushell/nushell/issues/8639 might be able to close h... — committed to nushell/nushell by amtoine 9 months ago
make "can't follow stream paths" error a bit better (#10569) related to - https://github.com/nushell/nushell/issues/9373 - https://github.com/nushell/nushell/issues/8639 might be able to close h... — committed to hardfau1t/nushell by amtoine 9 months ago

Most upvoted comments

Since the above comment with the problem statement and workaround got a bit long, I decided to split out “solution brainstorming” to a separate reply. Only I’m just now getting back around to finishing it up, 5 days later 😉.

I don’t really mind modifying scripts to handle the nuon conversion, but the requirement to have to add explicit to/from nuon on the command-line seems like something that might could/should be simplified. Right now, as @amtoine mentioned, the workaround on a pipeline script foo.nu would look like:
ls | to nuon | ./foo.nu | from nuon | ...
Ultimately it would be nice to eliminate the CLI/REPL to/from noun’s. E.g., the much cleaner:
ls | ./foo.nu | ...
Perhaps this could be done if there was a way for (any of the below):
1. `nu` to detect when it is being invoked as a subshell _of another Nu_ and, if so, automatically coerce it's input and output to nuon.  Bonus:  Default behavior would be nice, with it being able to be _disabled_ via `nu --raw-input/--raw-output` (or something).

2. Or, and I don't know if this is possible, but if the pipeline parser can detect whether the next or previous command in a pipeline is `nu`, automatically coerce to/from nuon.

3. Or, if there _is_ some way to create a builtin `nu` that overrides the external-command `nu`, then perhaps we get to something similar to what @fdncred hypothesized, where the internal command _could_ handle native data.  Probable downside, though:  This would be unlikely to work with a shebang, since that's (IIRC) an OS construct.

what about nushell automatically detecting NUON and putting a from nuon in there? This would also open the way to external commands capable of parsing and returning NUON being usable in a pipeline ‘natively’.

gabevenberg on Mar 22, 2024

During the last team meeting I think we talked about potentially adding a run command or something like that, which would allow you to run a nu script and pipe in/out of it without polluting scope. I think it would still run within the same process.

Generally it seems like we don’t want to hijack the normal execution of nu or try to detect nu scripts and run them differently without explicitly opting into this. There are too many things that would behave very unusually if we did that - one example being print which prints directly to the nu stdout, which you currently could capture in a pipeline, but wouldn’t be able to if we did this without running it externally.

So it makes more sense to have a separate command with semantics that are more like running a command.

devyn on Mar 27, 2024

Since the above comment with the problem statement and workaround got a bit long, I decided to split out “solution brainstorming” to a separate reply. Only I’m just now getting back around to finishing it up, 5 days later 😉.

I don’t really mind modifying scripts to handle the nuon conversion, but the requirement to have to add explicit to/from nuon on the command-line seems like something that might could/should be simplified. Right now, as @amtoine mentioned, the workaround on a pipeline script foo.nu would look like:

ls | to nuon | ./foo.nu | from nuon | ...

Ultimately it would be nice to eliminate the CLI/REPL to/from noun’s. E.g., the much cleaner:

ls | ./foo.nu | ...

Perhaps this could be done if there was a way for (any of the below):

nu to detect when it is being invoked as a subshell of another Nu and, if so, automatically coerce it’s input and output to nuon. Bonus: Default behavior would be nice, with it being able to be disabled via nu --raw-input/--raw-output (or something).
Or, and I don’t know if this is possible, but if the pipeline parser can detect whether the next or previous command in a pipeline is nu, automatically coerce to/from nuon.
Or, if there is some way to create a builtin nu that overrides the external-command nu, then perhaps we get to something similar to what @fdncred hypothesized, where the internal command could handle native data. Probable downside, though: This would be unlikely to work with a shebang, since that’s (IIRC) an OS construct.

NotTheDr01ds on Jul 20, 2023

I think this is because it’s output is raw input and not any type of structured data.

yeah 👍

but that’s unexpected and makes scripts way less useful 😕

amtoine on Jun 7, 2023

#5929 was closed as a duplicate of the earlier request articulated in #3551 This issue can also be subsumed by #3551

The current design classifies nu calls or calls to scripts that resolve via shebang to nu as external/process calls and thus communicates to the outside with the traditional POSIX assumptions. This has the benefit that the behavior of a script doesn’t change whether it is called from inside nushell or outside nushell (e.g. in bash calling nu my_script.nu).

If we follow the sentiment of this issue of having structured output from a script if invoked inside a host nu process, there is a wide open design space. How do we define that the arbitrary output of a free form script should be coerced to structured data. Would only the last expression count (viewed like a nu custom command) or should all print operations that don’t explicitly go to stderr feed into a stream of structured data (closer to how the world of posix behaves)?

sholderbach on Dec 12, 2023

I’m running into this while trying to make management scripts for a data set in nu. The only viable alternative I can think of to implementing this properly might be to have a sort of nu-direnv that can do use x y in the current directory, localized custom commands.

endigma on Aug 26, 2023