go: proposal: encoding/json: reject unknown fields in Decoder
Currently, json.Unmarshal (and json.Decoder.Decode) ignore fields in the incoming JSON which are absent in a target struct.
For example, the JSON { "A": true, "B": 123 } will successfully unmarshal into struct { A bool }.
This makes it difficult to do strict JSON parsing. One place this is an issue is in API creation, since you want to reject unknown fields on incoming JSON requests to allow safe introduction of new fields in the future (I’ve had this use case previously when working on a JSON REST API). It also can lead to typos leading to bugs in production: when a field is almost always the zero value in incoming JSON, you might not realise you’re not even reading a misspelled field.
I propose that a new method is added to Decoder to turn on this stricter parsing, in much the same way UseNumber is used today. When in strict parsing mode, a key in the incoming JSON which cannot be applied to a struct will result in an MissingFieldError error. Like UnmarshalTypeError, decoding would continue for the remaining incoming data.
d := json.NewDecoder(r.Body)
d.UseStrictFields()
err := d.Decode(myStruct)
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 35
- Comments: 36 (20 by maintainers)
Commits related to this issue
- fixup! add comment because of golang/go#15314 — committed to sloppyio/cli by philippfranke 8 years ago
- Units handling in the Catalog We had several options how to implement that: * one plan and user provided json to specify number of units: we dont like this option * multiple plans and we hardcode nu... — committed to alphagov/paas-compose-broker by deleted user 7 years ago
- Units handling in the Catalog We had several options how to implement that: * one plan and user provided json to specify number of units: we dont like this option * multiple plans and we hardcode nu... — committed to alphagov/paas-compose-broker by henrytk 7 years ago
- Units handling in the Catalog We had several options how to implement that: * one plan and user provided json to specify number of units: we don't like this option * multiple plans and we hardcode n... — committed to alphagov/paas-compose-broker by henrytk 7 years ago
Ah, thanks very much. OK, then UseStrictFields is definitely a bad name. 😃
We could certainly solve Dave’s problem but I wonder if we should do something a bit more generic. In the XML decoder there is a “,any” tag that you can use to say “unexpected things go here”. I wonder if JSON should allow a map[K]V (K=string, V=interface{} would be common but not strictly required) to have a tag
json:",any"to collect otherwise ignored key-value pairs. Then the decoder can do whatever it needs to do with them. Similarly on encoding those would have to be reinserted into the marshalled object.It might be too late for Go 1.7 to work all this out.
Change https://golang.org/cl/74830 mentions this issue:
encoding/json: disallow unknown fields in DecoderGiven that
map[string]interface{}is used if you were to Unmarshal an object intointerface{}, wouldn’t requiring anmap[string]interface{}type for,anybe sufficient? The purpose is to save the data instead of dropping it.Even if you could support other types such as
map[string]string, I’m not sure handling “what do you do with a non-string value” is a necessary pursuit.What if
,anyrequiredmap[string]interface{}and other types are revisited at a later time?Not sure why this was still in proposal mode. I think everyone agrees this can be added. Will move to Go1.9Early milestone. We missed Go1.8, sorry.
+1 for DisallowUnknownFields(). The interface is similar to the existing
UseNumber()method.