appengine: Proto Error returned when using aetest context -- `proto: invalid UTF-8 string`
The error is set inside the Iterator
on the call to datastore.Query.Run
.
It is initially returned by the call to internal.Call
line 534 in datastore/query.go
Which returns the error at the call to proto.Unmarshall
line 534 (also coincidently lol) in internal/api.go
The error is: proto: invalid UTF-8 string
This occurs with the most recent version of github.com/golang/protobuf
which was updated in response to this issue: https://github.com/golang/protobuf/issues/484
The following code reproduces the issue:
func TestB() {
inst, err := aetest.NewInstance(nil)
if err != nil {
panic(err.Error())
}
req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
panic(err.Error())
}
ctx := appengine.NewContext(req)
iter := datastore.NewQuery("link").KeysOnly().Run(ctx)
_, err = iter.Cursor()
if err != nil {
panic(err.Error())
}
}
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 5
- Comments: 22 (12 by maintainers)
Thanks, @dsnet. The error is now:
main_test.go:24: testB: proto: string field "appengine.CompiledQuery.PrimaryScan.index_name" contains invalid UTF-8
Much appreciated! For reference/if others run into this, this fixed the original issue:
Thank you for the bug report, I can reproduce this. This appears to be due to https://github.com/golang/protobuf/pull/499, which changes how the protobuf library handles strings: they are now expected to be valid UTF-8 strings.
Unfortunately, the way we encode protos in other protos in the datastore library doesn’t produce valid UTF-8 strings.
@egag Can you show me what your code does in as much detail as you’re able? I suspect it’s something akin to the following, but I want to be sure.
This indicates we might actually be able to fix this instead of pushing back against protobuf, but I need to sync up with someone on the datastore team to be sure.
@egag thanks, did something similar:
@lukebjerring I think that’s happening because your system’s
github.com/golang/protobuf
is out of date, but the docker image is using the latest protobuf package.The error message does make it sound like it’s a locale issue, but it’s actually because
protobuf
started validating allstring
fields to be validUTF-8
, which an encodedprotobuf
message cast to a string is not.