tonic: RPC with streaming input and unary output: unexpected internal error encountered

Bug Report

Version

cargo tree | grep tonic
└── tonic v0.1.0-beta.1
└── tonic-build v0.1.0-beta.1

Platform

OpenSuSE Leap 15

Linux linux-fgdx 4.12.14-lp151.28.36-default #1 SMP Fri Dec 6 13:50:27 UTC 2019 (8f4a495) x86_64 x86_64 x86_64 GNU/Linux

Description

Endpoint works when using a Rust client. Fails with a Python 3 client using the official grpc libraries: that specific payload cannot be decoded it seems, and I don’t know why. What kind of tips can you provide to either fix this, or further debug the encoding/decoding?

I’m currently running the server as:

RUST_LOG=debug RUST_BACKTRACE=1 cargo run --bin server

Server error

[2019-12-31T23:57:57Z INFO  server] [traj push states]
[2019-12-31T23:57:57Z DEBUG tonic::codec::decode] decoder inner stream error: Status { code: Unknown, message: "error reading a body from connection: protocol error: unexpected internal error encountered" } 
[2019-12-31T23:57:57Z DEBUG server] got state
[src/traj_srv/srv.rs:567] &traj_state = Err(
    Status {
        code: Unknown,
        message: "error reading a body from connection: protocol error: unexpected internal error encountered",
    },
)
[2019-12-31T23:57:57Z DEBUG hyper::proto::h2::server] send response error: user error: unexpected frame type
[2019-12-31T23:57:57Z DEBUG hyper::proto::h2::server] stream error: http2 error: user error: unexpected frame type

Relevant Rust server code

async fn traj_push_states(
        &self,
        request: Request<tonic::Streaming<TrajState>>,
    ) -> Result<Response<Empty>, Status> {
        info!("[traj push states]");

        let stream = request.into_inner();

        // Pin the inbound stream to the stack so that we can call next on it
        futures::pin_mut!(stream);

        while let Some(traj_state) = stream.next().await {
            debug!("got state");
            dbg!(&traj_state);
            let traj_state = traj_state?;
            debug!("unwrapped");
            let req_traj_id = traj_state.id as i32;
            // Fails _before_ the unwrapped debug message

Python code

The RPC stubs are generated as per https://grpc.io/docs/quickstart/python/ . All of the endpoints work apart from this one. Every endpoint is either a unary -> unary or a unary -> stream.

The Python implementation follows the same code as the RecordRoute from the routes gRPC tutorial.

RPC description

rpc TrajPushStates(stream TrajState) returns (Empty);

Where

message TrajState {
    uint32 id = 1; // Trajectory ID
    xb.EphemRegistry.State state = 2;
}
// ...
message State {
    // Absolute epoch
    Epoch epoch = 1;
    // The position may be unset, check the is_zero flag of the Vector.
    Vector position = 2;
    // The velocity may be unset, check the is_zero flag of the Vector.
    Vector velocity = 3;
    /* If the covariance is composed entirely of zeros, it is not informative, and therefore
     * can be assumed to be unset.*/
    Covariance covariance = 4;
    /* The covariance exponent specifies an optional exponent for all of the components of the
    * covariance. This enables storing high precision covariance while not losing precision of
    * floating point values.
    */
    double covariance_exponent = 5;
    // An optional map of parameters associated to this state
    map<string, Parameter> parameters = 6;
  }

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 21 (4 by maintainers)

Most upvoted comments

So you’re saying that the problem might be when the Python client tried to encode and send the next TrajState in the stream, is that correct?

Yes, that’s what I suspect. At least that’s what my code above is doing. If, like you said before, your Python client is closely modeled after the gRPC example (which mine is) and since we are seeing the exact the same error, there is a good chance your code has a similar issue.

And if so, any idea how I would go about solving that?

I would first try to confirm if this is actually an encoding issue on the Python side by encoding/decoding messages without making any calls to the server.

  • You said all other requests succeed, except this one, so try to identify which messages are not being used in the other calls. Maybe the error is in one of those messages.
  • Add some error handling to the client calls.
  • Implement a simple server with just the problematic method in another language, is the error still there?
  • If all else fails, maybe implement a simple unary method that uses the exact same messages and data as the client streaming method and call it in a loop. Does it fail?

I’ll take a look at your code tonight or tomorrow morning to see if something jumps out.