pystan: Decode Error when extracting protobuf message (macOS)
Just playing around with the package, I find that beyond a certain number of data or parameters, I seem to run into a Decode Error.
Taking the sample model…
schools_code = """
data {
int<lower=0> J; // number of schools
real y[J]; // estimated treatment effects
real<lower=0> sigma[J]; // standard error of effect estimates
}
parameters {
real mu; // population treatment effect
real<lower=0> tau; // standard deviation in treatment effects
vector[J] eta; // unscaled deviation from mu by school
}
transformed parameters {
vector[J] theta = mu + tau * eta; // school treatment effects
}
model {
target += normal_lpdf(eta | 0, 1); // prior log-density
target += normal_lpdf(y | theta, sigma); // log-likelihood
}
"""
This succeeds:
schools_data = {"J": 8,
"y": [28, 8, -3, 7, -1, 1, 18, 12],
"sigma": [15, 10, 16, 11, 9, 11, 10, 18]}
posterior = stan.build(schools_code, data=schools_data)
fit = posterior.sample(num_chains=4, num_samples=1000)
This succeeds:
schools_data = {"J": 8*10,
"y": [28, 8, -3, 7, -1, 1, 18, 12]*10,
"sigma": [15, 10, 16, 11, 9, 11, 10, 18]*10}
posterior = stan.build(schools_code, data=schools_data)
fit = posterior.sample(num_chains=4, num_samples=1000)
But this fails:
schools_data = {"J": 8*11,
"y": [28, 8, -3, 7, -1, 1, 18, 12]*11,
"sigma": [15, 10, 16, 11, 9, 11, 10, 18]*11}
posterior = stan.build(schools_code, data=schools_data)
fit = posterior.sample(num_chains=4, num_samples=1000)
Stack trace:
DecodeError Traceback (most recent call last)
<ipython-input-3-815b881aecc2> in <module>
1 posterior = stan.build(schools_code, data=schools_data)
----> 2 fit = posterior.sample(num_chains=1, num_samples=1000)
~/.pyenv/versions/3.8.5/envs/stan/lib/python3.8/site-packages/stan/model.py in sample(self, **kwargs)
239
240 try:
--> 241 return asyncio.run(go())
242 except KeyboardInterrupt:
243 pass
~/.pyenv/versions/3.8.5/lib/python3.8/asyncio/runners.py in run(main, debug)
41 events.set_event_loop(loop)
42 loop.set_debug(debug)
---> 43 return loop.run_until_complete(main)
44 finally:
45 try:
~/.pyenv/versions/3.8.5/lib/python3.8/asyncio/base_events.py in run_until_complete(self, future)
614 raise RuntimeError('Event loop stopped before Future completed.')
615
--> 616 return future.result()
617
618 def stop(self):
~/.pyenv/versions/3.8.5/envs/stan/lib/python3.8/site-packages/stan/model.py in go()
188 if resp.status != 200:
189 raise RuntimeError((await resp.json())["message"])
--> 190 stan_outputs.append(tuple(extract_protobuf_messages(await resp.read())))
191
192 def is_nonempty_logger_message(msg):
~/.pyenv/versions/3.8.5/envs/stan/lib/python3.8/site-packages/stan/model.py in extract_protobuf_messages(fit_bytes)
131 msg = callbacks_writer_pb2.WriterMessage()
132 next_pos, pos = varint_decoder(fit_bytes, pos)
--> 133 msg.ParseFromString(fit_bytes[pos : pos + next_pos])
134 yield msg
135 pos += next_pos
DecodeError: Error parsing message
Running on MacOS 10.15.6 with 32GB, Python 3.8.5, stan 3.0.0b4, httpstan 2.3.0
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17 (15 by maintainers)
Commits related to this issue
- test: Verify a model with many parameters works Check that a schools model with many parameters (160) works. Prompted by a unusual macOS bug in late 2020 (#163). — committed to stan-dev/pystan by riddell-stan 4 years ago
Worked on this a bit today. Clearly something has gone wrong with the protobuf encoding.
When I have the time, I’ll try doing what the error message suggests, switching from (protobuf) “string” to (protobuf) “bytes” in our protobuf definition file, https://github.com/stan-dev/httpstan/blob/main/protos/callbacks_writer.proto
Looking at the protobuf docs, it seems like we are, in fact, doing the wrong thing. A C++
std::stringis not a protobufstring.