oj: Rendering JSON with ActiveModel::Serializers returns object

Hello!

The situation we got into is quite unfortunate, as we have multiple JSON libraries fighting over each other about very generic method names like to_json 😞

Quick Summary

1. Rails developers should always override #as_json instead of #to_json
2. Rails expects that calling #to_json will always go through the Rails encoder (custom pure-Ruby encoder for <= 4.0, json gem for 4.1+), which considers the #as_json hooks among other things
3. To use Oj explicitly, developers should call Oj.dump(obj)
4. To use json gem explicitly, developers should call JSON.generate(obj) (this only work reliably on Rails 4.1+)
5. The #to_json definition simply call rails_json_encoder_encode(self.as_json) in all version (slightly simplified)
6. Oj can/should invoke #as_json on objects it doesn’t natively know how to encode

Before we get into the details, here is some backstory and history of how we got ourselves into the current situation.

Rails JSON encoder

Ancient History (Rails 2.3 era)

Once upon a time, Ruby doesn’t ship with any JSON library. As Rails needed to generate JSON objects, we wrote our own encoder, and eventualy settled on the to_json-based API as that seemed like the obvious choice at the time. You simply override to_json on your classes, construct a hash to represent your data and recursively call to_json on that Hash, and you have your desired JSON string. (Fun fact: that commit also included the initial JSON decoder that essentially worked by converting JSON into YAML using gsub so it could be parsed with Syck. Fun times!)

Rails 3 era

At some point, Rubyists started writing other JSON libraries/wrappers that are more efficient than the pure-Ruby encoder that comes with Rails, such as Oj, yajl, etc. There is a problem though – the to_json API does not offer any way for us to cooperate with these libraries. Because to_json is responsible for returning the resulting JSON string, there is no way for these alternative encoders to “hook into” this process.

The key observation here is that most application programmers don’t override to_json because they want control of how Ruby objects are encoded into a JSON string; they simply want to describing how they want their data represented (e.g. what fields to include, whether the object should map to an Array or an Object…).

Thus, as_json was born. The idea is that developers can just override as_json to return the Ruby representation of their data (i.e. return a Hash/Array/etc) and let the encoder take care of turning that into the JSON string.

For backwards compatibility reasons, the #to_json method is kept as an “entry point” to our Ruby-based encoder (1.8.7 doesn’t have an JSON encoder in the standard library). However, it should now be possible to use an alternative encoder such as Oj, e.g. by calling Oj.dump(some_rails_object_that_oj_doesnt_know_about). In this case, the oj encoder can simply invoke #as_json on the object to get a (hopefully*) simplified representation of the object that it will be able to encode.

(*the as_json hook is provided as a “hint” to the encoder – the developer is free to return any strange objects in the hook and it is up to the encoder to decide what to do with them. In practice though, the meaning of things like symbols, hashes, arrays, booleans and strings are well understood. This is not required to be recurrsive, so if the developer returned, say, an array of non-literals, the encoder should still try to call as_json on its elements. I took some detailed notes about how a reasonable encoder should behave here.)

Rails 4 era

When Ruby 1.9.1 was released, the json gem officially become part of the Ruby standard library. Since 4.0 dropped support for Ruby 1.8.7, in theory everything should be nice from here on. Unfortunately, it uses the #to_json architecture, which prevented it from being very useful inside Rails because of the flaws described above. Thus things remained largely unchanged.

As the popularity of the JSON gem grew though, another annoying issue arose. As mentioned above, the json gem also defines a to_json method that is not quite the same as the one shipped with Rails. (e.g. one takes an option hash, one expects a State object; one considers as_json, the other doesn’t). This created all sorts of problems and confusion.

Rails 4.1+

I picked up the task of cleaning up some of these mess. A few things changed:

• Previously, Rails and json gem would appear to play nice with each other until things suddenly blow up in some edge cases. The main issue is that the JSON gem expects and passes a hash-like-but-not-really State object as the second argument to to_json, where Rails expects a real hash. To avoid these problems, we detect and completely bypass the Rails encoder when the JSON gem is involved. So when you call ActiveSupport::JSON.encode(obj) or obj.to_json, you get the Rails stuff, whereas when you explicitly call JSON.{generate|dump}(obj) you get the “bare” result from JSON gem without any of the Rails stuff (as_json).
• Since Ruby 1.9+ ships with a json gem, there is no point maintaining the pure-Ruby encoder inside Rails anymore. I ripped out all the encoding related stuff inside Rails (i.e. code that converts Ruby object into the actual JSON strings… e.g. nil => "null"), and simply shim’ed the JSON gem. So when you call to_json, Rails would first recursively call as_json on the object and pass that to JSON.generate.
• Since the maintainers of multijson seemed uninterested in continuing to maintain that gem, it has been pulled out of Rails in favor of just using the built-in json gem. This only affects parsing! In previous versions, Rails has only used multi-json on the parsing side. The encoding side always went through Rails’ own encoder (through #to_json) with or without multi-json/oj/json gem activated. Except for the spots that you explicitly used MultiJson.dump(...), you have always been using Rails’ internal encoder. Installing Oj gem does not tell Rails to use the Oj encoder regardless of your MultiJson settings. Any effect on encoding speed you noticed is probably psychological 😃

Rails controller

When you call render json: some_object, Rails will call #to_json on the object, which should go through the Rails encoder out-of-the-box.

Active Model Serializer

v0.8 & 0.9

These two versions of AM::S has a completely different codebase, but for our purpose they are virtually the same. When you do a render json: object, it tries to wrap object in the appropriate serializer object (which implements as_json), and pass that object (still a Ruby object at this point) to the controller’s json renderer, which calls to_json as described above.

If you are using Rails 4.2, you’ll need 0.8.2 / 0.9.0 for things to work properly because Rails renamed some of the controller hook methods.

v0.10

This is another complete rewrite, but currently it defines to_json instead of as_json.

(@guilleiguaran we need to ✂️ these to_json in AM::S in favor of as_json as I pointed out here)

Where things went wrong with Oj + Rails

The Rails codebase relies on to_json across the board. By default, this should resolve to the Rails JSON encoder, which uses the as_json hook. It appears that somewhere in the Oj gem it overrides to_json with a definition that ignores the as_json hooks (probably here?).

Where does that leave us…

Option 1: Explicitly opt-in

1. Configure Oj to…
   1. Not override to_json
   2. Honor as_json
   3. (I believe the above amounts to “don’t use mimic json gem mode, and enable compat mode”, but I could be wrong)
2. Explicitly use render json: Oj.dump(obj) and render json: Oj.dump(WhateverSerializer.new(obj)) in your controllers

This is pretty much how things always worked with previous versions of Oj. (Again MultiJson never actually helped you for encoding.)

Presumably, you can monkey patch the json renderer to do this automatically for you (in that case the AM::S renderer additions should Just Work™ also).

Option 2: Automatically use Oj for everything…

This requires patching Oj to offer a mode that would override to_json to encode objects in a Rails-compatible way. This is super invasive and you’ll probably end up dealing with a lot of the same bugs that we had dealt with before, so I don’t know if I could recommend this.

Also, it’s quite likely that we would deviate slightly on the encoding result, but if anything comes up I’m happy to work with you to figure out what the “correct” behavior should be (or whether that’s an edge case that should be left undefined).

Option 3: Automatically use Oj for everything (Rails 4.1+ only)…

If supporting only Rails 4.1 and above is an option, there’s a slightly less invasive option. (In this case it probably shouldn’t try to mimic json gem.) It still has the “deviate slightly on the encoding result” problem, but it’d at least shield you from the json gem compat State object nonsense.