airy: Parse (but don't map) API message response content

Current state

We deliver in-/outbound message content on the APIs as a string, which is the way we store it internally. I.e. a text message looks like this:

{ //...
    "content": "{\"text\":\"on the road again\"}",
}

Desired state

We still do not want to maintain a schema for the source content. However, it would be useful to clients if we at least parse the message content to an untyped JSON object to make it easier to consume. The same message would look like this:

{ //...
    "content": {"text":"on the road again"},
}

This works for all sources that support JSON. For Twilio, this is not the case as we get messages in a url-encoded string. Therefore we can keep representing message content that we cannot parse to JSON as a string like so:

{
  "content": "ApiVersion=2010-04-01&Body=hello%20how%20are%20you%0A"
}

TODO:

If possible parse the content for the MessageResponsePayload to a JsonNode
Remove JSON.parse code from the render library

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 2
Comments: 24 (23 by maintainers)

Commits related to this issue

[#934] Parse (but don't map) API message response… …content — committed to airyhq/airy by paulodiniz 3 years ago
[#934] Parse (but don't map) API message response content — committed to airyhq/airy by paulodiniz 3 years ago
[#934] Parse (but don't map) API message response content — committed to airyhq/airy by paulodiniz 3 years ago
[#934] Parse (but don't map) API message response content — committed to airyhq/airy by paulodiniz 3 years ago
[#934] Parse (but don't map) API message response content (#977) — committed to airyhq/airy by paulodiniz 3 years ago

Most upvoted comments

Yeah I understand the principle in the the backend, and I wouldn’t break it. If the principle is “You send whatever you want, you will get it back in the other side” we have to be ready (and I think it is implicit in the principle) to render any type of response.

AitorAlgorta on Feb 10, 2021

So since I am gathering from Aitor and Bodo’s comments that the frontend is also okay with this change I am removing the “needs discussion” label.

chrismatix on Feb 10, 2021

maybe i get this wrong, but if you stringify a perfect JSON object you get from a source and make it a string, in order to support the restrictions of the other weakest source you have, don’t you modify it by doing so and start to have an opinion about the content not being an object but a string?

yes that’s a valid argument.

I had just discussed this with @chrismatix offline and if we want to stay true to our own design principle then we should return json objects when the source sent us json objects and strings when the source sent us that. What put me off is the accent on parsing. It’s not about parsing, it’s about being truly transparent. So I agree we should be transparent, after all the past weeks were all about supporting this design principle. To recap, I’m suggesting:

return JSON objects for the sources but twilio
return the string for the twilio one

iven they don’t care about twilio and their shortcomings (also since twilio currently makes up for 0.1% of our traffic, most people won’t even have that issue).

In the context of the open source project, this is irrelevant.

lucapette on Feb 10, 2021

I think there’s no solution that works for all cases. This actually reflects very well on the core problem of this design: we’re dealing with data we don’t own and you either have an opinion about the data or you don’t. I think we now know not having an opinion is an healthier trade-off and so far the fact we “break the contract” seems to be the only problem we encounter.

In theory, I agree with the point made by @paulodiniz as I also don’t feel very good that we return data types that are dependent on the source. But then, again, as soon as I read the sentence I just wrote I also realize that’s “by design”. If we truly want to stay away from having an opinion about the content, then I would argue it must happen that different sources have different data types for the content. After all that’s true anyway even if you have all JSON… they’re different JSON structures so while there’s an argument for “they’re aal JSON objects” the argument is not so strong at closer examination as unless you know what’s the source you’re dealing with, you’re still left with no idea of what’s the data you’re dealing with. And from that perspective, different JSON objects or “JSON object in one case, string to decode in some funny way in an other” leads to the same exact responsibility on the client code. To be fair, (using the practical example we’re dealing with here), by changing twilio responses we’d have the worst of both words (in terms of general design): we’d have an opinion on the content for some sources and we wouldn’t for others. As the consumers of the API would have to know the contract either way, you can indeed argue that by always returning JSON we’re doing more work than we should and so would the consumers of the API.

tl;dr while it’s counterintuitive to return different data types depending on the sources, that’s in line with our philosophy of not having an opinion on the content

lucapette on Feb 17, 2021

So we have to put the parsing somewhere. Either into the frontends or into our backend. I would vote for our backend to make the API easier to consume. The target of this airy thing is devs and having a clean api to the outside is more important to me than having a cleaner code inside.

bitboxer on Feb 10, 2021

I think I get what @lucapette is saying. It’s a principle of least surprise.

Meaning: it’s a surprise that the API returns one type and then returns another type depending on the source. I agree with that. We could transform the twilio data to a json before delivering it, but that would break the whole principle of it (not change content)

paulodiniz on Feb 10, 2021