pystan: Decode Error when extracting protobuf message (macOS)

Just playing around with the package, I find that beyond a certain number of data or parameters, I seem to run into a Decode Error.

Taking the sample model…


schools_code = """
data {
  int<lower=0> J;         // number of schools
  real y[J];              // estimated treatment effects
  real<lower=0> sigma[J]; // standard error of effect estimates
}
parameters {
  real mu;                // population treatment effect
  real<lower=0> tau;      // standard deviation in treatment effects
  vector[J] eta;          // unscaled deviation from mu by school
}
transformed parameters {
  vector[J] theta = mu + tau * eta;        // school treatment effects
}
model {
  target += normal_lpdf(eta | 0, 1);       // prior log-density
  target += normal_lpdf(y | theta, sigma); // log-likelihood
}
"""

This succeeds:

schools_data = {"J": 8,
                "y": [28,  8, -3,  7, -1,  1, 18, 12],
                "sigma": [15, 10, 16, 11,  9, 11, 10, 18]}

posterior = stan.build(schools_code, data=schools_data)
fit = posterior.sample(num_chains=4, num_samples=1000)

This succeeds:

schools_data = {"J": 8*10,
                "y": [28,  8, -3,  7, -1,  1, 18, 12]*10,
                "sigma": [15, 10, 16, 11,  9, 11, 10, 18]*10}

posterior = stan.build(schools_code, data=schools_data)
fit = posterior.sample(num_chains=4, num_samples=1000)

But this fails:

schools_data = {"J": 8*11,
                "y": [28,  8, -3,  7, -1,  1, 18, 12]*11,
                "sigma": [15, 10, 16, 11,  9, 11, 10, 18]*11}

posterior = stan.build(schools_code, data=schools_data)
fit = posterior.sample(num_chains=4, num_samples=1000)

Stack trace:

DecodeError                               Traceback (most recent call last)
<ipython-input-3-815b881aecc2> in <module>
      1 posterior = stan.build(schools_code, data=schools_data)
----> 2 fit = posterior.sample(num_chains=1, num_samples=1000)

~/.pyenv/versions/3.8.5/envs/stan/lib/python3.8/site-packages/stan/model.py in sample(self, **kwargs)
    239
    240         try:
--> 241             return asyncio.run(go())
    242         except KeyboardInterrupt:
    243             pass

~/.pyenv/versions/3.8.5/lib/python3.8/asyncio/runners.py in run(main, debug)
     41         events.set_event_loop(loop)
     42         loop.set_debug(debug)
---> 43         return loop.run_until_complete(main)
     44     finally:
     45         try:

~/.pyenv/versions/3.8.5/lib/python3.8/asyncio/base_events.py in run_until_complete(self, future)
    614             raise RuntimeError('Event loop stopped before Future completed.')
    615
--> 616         return future.result()
    617
    618     def stop(self):

~/.pyenv/versions/3.8.5/envs/stan/lib/python3.8/site-packages/stan/model.py in go()
    188                         if resp.status != 200:
    189                             raise RuntimeError((await resp.json())["message"])
--> 190                         stan_outputs.append(tuple(extract_protobuf_messages(await resp.read())))
    191
    192                 def is_nonempty_logger_message(msg):

~/.pyenv/versions/3.8.5/envs/stan/lib/python3.8/site-packages/stan/model.py in extract_protobuf_messages(fit_bytes)
    131                 msg = callbacks_writer_pb2.WriterMessage()
    132                 next_pos, pos = varint_decoder(fit_bytes, pos)
--> 133                 msg.ParseFromString(fit_bytes[pos : pos + next_pos])
    134                 yield msg
    135                 pos += next_pos

DecodeError: Error parsing message

Running on MacOS 10.15.6 with 32GB, Python 3.8.5, stan 3.0.0b4, httpstan 2.3.0

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 17 (15 by maintainers)

Commits related to this issue

Most upvoted comments

Worked on this a bit today. Clearly something has gone wrong with the protobuf encoding.

When I have the time, I’ll try doing what the error message suggests, switching from (protobuf) “string” to (protobuf) “bytes” in our protobuf definition file, https://github.com/stan-dev/httpstan/blob/main/protos/callbacks_writer.proto

Looking at the protobuf docs, it seems like we are, in fact, doing the wrong thing. A C++ std::string is not a protobuf string.