expectrl: Need non-blocking reads for interactive()

I can’t use expect() directly to read lines for line processing because I need non-blocking reads. Using non-blocking reads allows more control while waiting for data to be available. For example, I add an animated cursor if the data takes too long to appear.

Here’s the current code, roughly. It’s loosely based on the expectrl source code using try_read here.

The following code shows a number of useful expectrl features. Specifically, it:

Catches \r and \n as the command writes lines, allowing them to be rewritten. This is useful because some commands emit just \r to overwrite the previous line in a terminal rather than scrolling.
Allows lines to be read individually and processed.
Allows timeouts to be checked.
Allows other processing in the loop, such as starting an animated cursor or waiting message if the command is taking too long to emit any output.

    use expectrl::{Session, Signal, WaitStatus};
    use std::io::{stdout, Write};
    use std::{process::Command, thread, time::Duration, time::SystemTime};

    fn main() {
      let mut cmd = Command::new("ping");
      cmd.args(&["www.time.org"]);

      let mut session = Session::spawn(cmd).expect("failed to execute process");
      thread::sleep(Duration::from_millis(300)); // Give the process a moment to start.

      let mut buf = vec![0; 512]; // Line buffer.
      let mut b = [0; 1]; // Read buffer.
      let sys_time = SystemTime::now();
      loop {
        match session.try_read(&mut b) {
          Ok(0) => {
            println!("==> EoF");
            break; // EoF
          }
          Ok(_) => {
            buf.push(b[0]);
            if b[0] == 10 || b[0] == 13 {
              stdout().write_all(&buf).unwrap();
              stdout().flush().unwrap();

              // (1) Further process the line as needed. 
              // ...

              buf.clear();
            }
          }
          Err(err) if err.kind() == std::io::ErrorKind::WouldBlock => {}
          Err(err) => {
            println!("err {}", err);
            break;
          }
        }
        // Check for timeout.
        // Start/update animated cursor if there's no output
      }

      match session.wait() {
        Ok(WaitStatus::Exited(pid, status)) => {
          println!("child process {} exited with: {:?}", pid, status);
        }
        // Signaled() is expected if the process is terminated.
        Ok(WaitStatus::Signaled(_, s, _)) => {
          println!("child process terminated with signal: {}", s);
        }
        Err(e) => {
          println!("error waiting for process: {:?}", e);
        }
        s => {
          println!("unhandled WaitStatus: {:?}", s);
        }
      }
    }

This code works because Session provides a try_read() that doesn’t block. Currently,expect() already provides many of these features, including timeouts and the Any() method to detect specific line endings. But expect() blocks. With expect() blocking, I would have to multithread cursor animation and synchronize terminal writes. Doable, but it adds complexity that the try_read() approach avoids.

The problem: How to allow interactive typing into the terminal, with the line processing as above? There’s interactive()but it can’t be used in this context because it blocks.

Possible solutions:

Could there be a version of interactive() that emits lines for processing?
Could there be a version of interactive() that accepts a passed-in function for line processing or other work? (eg, at (1) above).
Could there be a try_interactive() allows interactivity, but checks for data written to stdout without blocking?

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 59 (39 by maintainers)

Most upvoted comments

I’m seeing some differences with the new macro and the original code I had that used try_read(). I’m trying to create individual issue reproductions now.

Meanwhile, here’s a program that demonstrates many of the kinds of outputs that I think expectrl should handle. Can expectrl process each line (meaning apply functions like watching for keywords), and write each line to the screen, so that it appears the same as when the program is run directly? Note that this program includes user input in the middle, which ideally would just be entered by the user without having to also enter/exit a special interactive mode. Wdyt? Is this a good test for expectrl? There are likely other tests that could be added, such as multiline cursor movement and more ANSI codes.

#!/usr/bin/env python3
import enum
import sys
import time
import getpass

NUM_LINES = 5
SAME_LINE_ESC = "\033[F"


class Color(enum.Enum):
    RESET = "\33[0m"
    RED = "\33[31m"
    GREEN = "\33[32m"
    YELLOW = "\33[33m"


def colorize(text: str, color: Color) -> str:
    """Wrap `text` in terminal color directives.

    The return value will show up as the given color when printed in a terminal.
    """
    return f"{color.value}{text}{Color.RESET.value}"


def main():
    """Demonstrate several kinds of terminal outputs.

    Examples including ANSI codes, "\r" without "\n", writing to stdin, no-echo
    inputs.
    """

    # Show a color.
    print("status: ", colorize("good", Color.GREEN))

    # Show same-line output via "\r".
    for i in range(NUM_LINES):
        sys.stdout.write(f"[{i+1}/{NUM_LINES}]: file{i}\r")
        time.sleep(1)
    print("\n")

    # Show same-line output via an ANSI code.
    for i in range(NUM_LINES):
        print(f"{SAME_LINE_ESC}[{i+1}/{NUM_LINES}]: file{i}")
        time.sleep(1)

    # Handle prompts which don't repeat input to stdout.
    print("Here is a test password prompt")
    print(colorize("Do not enter a real password", Color.RED))
    getpass.getpass()

    # Handle simple input.
    ans = input("Continue [y/n]:")
    col = Color.GREEN if ans == "y" else Color.RED
    print(f"You said: {colorize(ans, col)}")
    if ans == "n" or ans == "":
        sys.exit(0)

    # Handle long-running process, like starting a server.
    print("[Starting long running process...]")
    print("[Ctrl-C to exit]")
    while True:
        print("status: ", colorize("good", Color.GREEN))
        time.sleep(1)


if __name__ == "__main__":
    main()

GaryBoone on Sep 23, 2021

ping

Heh. Was coding… 😃 I wanted to create some demo code that showed a couple of things:

Using the same function in multiple on_output() handlers.
Creating on_output() handlers programmatically from a mapping of keywords to functions. This is necessary because in some cases we don’t know how many keywords we’ll need to recognize until we read a config or obtain it from the user.

Here’s what I came up with. Notes:

It’s easy it is to pass a struct into the context via state. Eg: the dc variable.
The handler insertion wasn’t quite as simple as above where we had:

    for (key, handler) in keyword_handler_map {
      opts = opts.on_output(key, handler)
    }

The reason was that the map ended up being mutably borrowed twice, once when passed into the state, next when use in the for loop. So that failed to compile. What worked was to pull the keys out separately and iterate over those. Note that the for loop was actually over closures, not simple functions, so slightly more wordy, but it worked. If you know of a simpler way, please share it!

BTW, note that the on_output() closures could reuse the Found value given in the closure parameters, but that would involve more processing of them to utf8 decode them, slowing things down, and shouldn’t be necessary since the mapping of keyword to function was given in the map.

use std::collections::HashMap;

trait MyHandler {
  fn inc(&mut self);
  fn val(&self) -> &u32;
}

struct FileCounter {
  file_counter: u32,
}
impl MyHandler for FileCounter {
  fn inc(&mut self) {
    self.file_counter += 1;
  }
  fn val(&self) -> &u32 {
    &self.file_counter
  }
}

struct StatusCounter {
  status_counter: u32,
}
impl MyHandler for StatusCounter {
  fn inc(&mut self) {
    self.status_counter += 1;
  }
  fn val(&self) -> &u32 {
    &self.status_counter
  }
}

struct DelayChecker {
  last_update: std::time::SystemTime,
}
impl DelayChecker {
  fn update(&mut self) {
    self.last_update = std::time::SystemTime::now();
  }
}

fn main() {
  let mut session = expectrl::spawn("./terminal_example.py").expect("Can't spawn a session");

  let fc = FileCounter { file_counter: 0 };
  let sc = StatusCounter { status_counter: 0 };
  let dc = DelayChecker { last_update: std::time::SystemTime::now() };

  let mut map: HashMap<&str, Box<dyn MyHandler>> = HashMap::new();
  map.insert("file", Box::new(fc));
  map.insert("status", Box::new(sc));
  let keys = map.keys().cloned().collect::<Vec<&str>>();

  let state = (dc, map);
  let mut opts = expectrl::interact::InteractOptions::terminal()
    .unwrap()
    .state(state)
    .on_idle(|mut ctx| {
      if let Ok(elapsed) = ctx.state().0.last_update.elapsed() {
        if elapsed > std::time::Duration::from_millis(3000) {
          println!("\r\n[waiting...]");
          ctx.state().0.update();
        }
      }
      Ok(())
    });

  for key in keys {
    opts = opts.on_output(key, move |mut ctx, _| {
      ctx.state().0.update();
      if let Some(handler) = ctx.state().1.get_mut(&key) {
        handler.inc();
      }
      Ok(())
    });
  }

  opts.interact(&mut session).unwrap();

  println!("Last update at {:?}", opts.get_state().0.last_update);
  println!("files: {}", opts.get_state().1["file"].val());
  println!("status count: {}", opts.get_state().1["status"].val());
}

I think the additions you made for this bug greatly increase the value of expectrl because it means that it can be used in two ways: 1) the original way where the coder knows what to expect from given commands and expectrl will confirm them, and now 2) a new way where the coder doesn’t know exactly the commands or their order, such as if they’re given by the user or a user-written config file, but can watch the pty for keywords.

I appreciate your help on this issue and enjoyed working with you on it. Thanks, Maxim!

GaryBoone on Oct 6, 2021

Well, what’s interesting is that it only consumes that input that partially matches the the target string.

So to reproduce, use the example program I just added above, then

Wait for the “Continue [y/n]:” prompt.
Type a sequence containing parts of the key input phrase “123”. Note that whenever the sequence contains those characters in order, the characters don’t appear until they don’t match. Eg type “wq1q12q123”. You’ll see

What you type	What you see
w	w
q	wq
1	wq
q	wq1q
1	wq1q
2	wq1q
q	wq1q12q
1	wq1q12q
2	wq1q12q
3	wq1q12qYou typed a magic word…

Note that the result from the python input function (ans = input("Continue [y/n]:")) is missing the matched part:

“You said: wq1q12q”

GaryBoone on Oct 1, 2021

What do you call it here?

Oh, I was just wondering if options.input.read() should be called options.input.try_read() to match elsewhere where try_ indicated non-blocking.

Meaning on_output will be called just when there was something produced by process and that’s it right?

Yes, that’s what I was thinking. Hmmm. I’m not sure about the options for handling buffers. I assume most pty output we’re watching is line-oriented, so buffering by lines would make sense, if possible. Ie, calling the callback function would happen on “\r”/“\n”. But I’m not sure about the efficiency and speed of how that would be done. As long as we don’t copy. 😃

Re, @dmgolembiowski’s comment, the suggestion would work for the situation where all of the input from the command was read, so that wouldn’t work for the use case where we’re watching, printing, and reacting to output lines as they are generated by the command being executed. In another comment, the suggestion was to use Python’s pty library, which would be simplifying so might be a workaround, but would incur all of the overhead of adding a Python layer. So perhaps slightly different conversations… 😃

GaryBoone on Sep 28, 2021

@GaryBoone maybe you’ll find something useful here https://github.com/ProgrammingRust/async-chat/blob/master/src/bin/client.rs

@dmgolembiowski do you mean this reading from stdin? https://github.com/ProgrammingRust/async-chat/blob/abbd763289a1cf367b1c2183e7d432de83b1c650/src/bin/client.rs#L13-L14

 let mut command_lines = io::BufReader::new(io::stdin()).lines();
    while let Some(command_result) = command_lines.next().await {

Indeed If you do.

*I tend to think such a thing is provided in expectrl as well.

    use std::io::BufRead;
    let mut session = expectrl::spawn("cat").expect("Can't spawn a session");
    while line in session.lines() {
    }

**the same must be possible with async feature.

zhiburt on Sep 28, 2021

Alternatively, it may be healthier to directly interface with Python’s own pty module, except instead of using subprocess.PIPE for your stdin, stdout, and stderr arguments you instead use something that has the same behavior as socket.socket and acquire its file descriptors from Rust to listen on and forward appropriately.

def ask_rust_for_fd():
    # ...
    return (STDIN, STDOUT, STDERR)

in_, out_, err_ = ask_rust_for_socket()

tmp  = tempfile.NamedTemporaryFile(dir=".", suffix=".py", mode="w", delete=False)
Self = sys.executable   
subcmd = shlex.split('{python} -i {startup}'.format(python=Self, startup=tmp.name)) 

proc = subprocess.Popen(subcmd, 
        stdin=in_, #subprocess.PIPE,
        stdout=out_, #subprocess.PIPE,
        stderr=err_, #subprocess.PIPE,
        shell=True, 
        cwd=os.curdir,
        env=os.environ)

pty.spawn(subcmd)

dmgolembiowski on Sep 28, 2021

@GaryBoone maybe you’ll find something useful here https://github.com/ProgrammingRust/async-chat/blob/master/src/bin/client.rs

dmgolembiowski on Sep 28, 2021

Yes, please use the script as you like to create tests and demos in expectrl.
Re how expectrl handles ansi-sequences, it should read them into a buffer that can be repeated to stdout and work. For example, when I run the demo script by itself, I see the lines like[3/10] all print on the same line. I see colored output.

On my screen, it looks like this:

Kapture 2021-09-24 at 08 37 19

So expectrl should be able to read the lines, process them if desired, then write them to stdout and the result should appear identical to directly running the script.

In your examples,

    println!("{:?}", String::from_utf8_lossy(f.first()));

is great for converting the data to strings for processing. from_utf8_lossy() helpfully drops the ANSI codes. But how to print the line, including the ANSI codes? It should something like:

let mut session = spawn_bash().unwrap();
session.send_line("./terminal_example.py").unwrap();
loop {
  let _err = expectrl::check!(
      session,
      line_ending = Any(vec!["\n", "\r"]) => {
        stdout().write_all(line_ending.before()).unwrap();
        stdout().write_all(line_ending.first()).unwrap();
        // Do additional processing of the line.
      },
      _ = Eof => break,
  );
  // Do additional processing while watching the command.
 }

… which I know works, except for the interactive parts of terminal_example.py. expectrl does read ANSI sequences just fine!

We really have two cases: blocking and not blocking. In the above examples, you showed several session.expect() examples. I’m working with check!() and previously, try_read() because I need do additional processing inside the read loop. So this bug is about the non-blocking case.
Also, note that in real-world examples, we don’t always know if a program being run under expectrl will request a password or other input. Therefore, we need the above to work with interactive. Also, we don’t know how long a program will run or even whether it will exit, such as if it’s a server. These reasons are why my test includes all of these examples in one file. Is it possible to handle interactive input with this non-blocking case?

GaryBoone on Sep 24, 2021

Another reason to include idle processing: Without it, the system can only react as data is written to stdout. Except for a terminating timeout or adding additional threading, there’s no mechanism to check progress or react to delayed output.

GaryBoone on Sep 16, 2021

Essentially you’re proposing a way to write something to process input when some action happens like a expected string found inside output while being in interactive mode, right?

Yes. Ideally, to me there would be no distinction between interactive and non-interactive modes. Ideally, user code would be able to watch for specific strings in the terminal whether the command emitted it, the user typed it, or it was an unpredictable output in a program caused by something the user typed.

+1 to the interact! macro idea. It’s very concise and readable.

Would it possible to have some kind of input for ‘idle’ when there’s no output to the pty?

interact!{
 session,
 "\r" | "\n" => {}
  "CURSOR_SYMBOL" => {}
  IDLE => {
    // user code to start "wait" animation or show "waiting" message or other idle processing
  }
}

+1 to the non-blocking expect as well. It would allow the most flexibility. It would need to handle interactivity though…

GaryBoone on Sep 16, 2021