appengine: Proto Error returned when using aetest context -- `proto: invalid UTF-8 string`

The error is set inside the Iterator on the call to datastore.Query.Run. It is initially returned by the call to internal.Call line 534 in datastore/query.go Which returns the error at the call to proto.Unmarshall line 534 (also coincidently lol) in internal/api.go The error is: proto: invalid UTF-8 string

This occurs with the most recent version of github.com/golang/protobuf which was updated in response to this issue: https://github.com/golang/protobuf/issues/484

The following code reproduces the issue:

func TestB() {
	inst, err := aetest.NewInstance(nil)
	if err != nil {
		panic(err.Error())
	}
	req, err := inst.NewRequest("GET", "/", nil)
	if err != nil {
		panic(err.Error())
	}
	ctx := appengine.NewContext(req)
	iter := datastore.NewQuery("link").KeysOnly().Run(ctx)
	_, err = iter.Cursor()
	if err != nil {
		panic(err.Error())
	}
}

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 5
  • Comments: 22 (12 by maintainers)

Commits related to this issue

Most upvoted comments

Thanks, @dsnet. The error is now:

main_test.go:24: testB: proto: string field "appengine.CompiledQuery.PrimaryScan.index_name" contains invalid UTF-8

Much appreciated! For reference/if others run into this, this fixed the original issue:

pushd $(dirname $(which gcloud))/../platform/google_appengine/google/appengine/datastore
sed -i "s/query.Encode()/'devindex'/" datastore_stub_util.py
popd
go get -u github.com/golang/protobuf

Thank you for the bug report, I can reproduce this. This appears to be due to https://github.com/golang/protobuf/pull/499, which changes how the protobuf library handles strings: they are now expected to be valid UTF-8 strings.

Unfortunately, the way we encode protos in other protos in the datastore library doesn’t produce valid UTF-8 strings.

@egag Can you show me what your code does in as much detail as you’re able? I suspect it’s something akin to the following, but I want to be sure.

k := datastore.NewKey(ctx, "Data", "invalidbytes", 0, nil)
b := "\xa0\x01"  // \xa0 is an invalid utf-8 byte
_, err := datastore.Put(ctx, k, b)

This indicates we might actually be able to fix this instead of pushing back against protobuf, but I need to sync up with someone on the datastore team to be sure.

@egag thanks, did something similar:

 go_medium_test: go_deps dev_appserver_deps
   # Hack to work around https://github.com/golang/appengine/issues/136
   cd $(GOPATH)/src/github.com/golang/protobuf; git checkout ac606b1
   cd $(WPTD_GO_PATH); go test -tags=medium -v $(FLAGS) ./...

@lukebjerring I think that’s happening because your system’s github.com/golang/protobufis out of date, but the docker image is using the latest protobuf package.

The error message does make it sound like it’s a locale issue, but it’s actually because protobuf started validating all string fields to be valid UTF-8, which an encoded protobuf message cast to a string is not.