spring-boot: Consider using RFC 7807 problem details for error responses

At bol.com, the biggest online retailer in the Netherlands and Belgium, RFC 7807 is our bread and butter. Judging from the amount of work Zalando has also put into this (problem and problem-spring-web), I think we are not alone in this effort. This said, there is a significant amount of work that we needed to implement to make Problems work with Spring. Here I want to share some highlights from our iceberg.

Model

In our custom Problem model, we enhance the one in RFC 7807 with the following fields:

• String host – The host on which this problem has occurred.
• List<Violation> violations – The list of constraint violations.
• Problem cause – The underlying cause, if any.
• Map<String, Object> metadata – Collection of fields that are unknown during deserialization.

Problem is an interface. Its default implementation is DefaultProblem accessible via Problem.builder(). DefaultProblem extends from AbstractThrowableProblem, which extends from RuntimeException and implements Problem.

There are, IIRC, at least 3 Jackson bugs one need to work around to make serialization work using annotations. For instance, if cause extends from Throwable, given Throwable is final, quite some mixin magic is needed to avoid serializing stack traces and such. Hence, the necessary (de)serialization plumbing is far from trivial.

Models are published in a standalone artifact with only Jackson dependency.

Integration

We encourage our programmers to always throw standard exceptions (e.g., RuntimeException) in their business logic. If need arises, they are only advised to catch exceptions at the controller level and map them to Problems manually. Put another way, we only advise explicit use of Problems to communicate a failure if current mappers fall short of addressing the need; else, standard exceptions are the way to go.

Mapping exceptions to Problems

Our plethora of Spring sauce does its best to catch thrown exceptions and create a meaningful Problem out of that. We have custom mappers for a variety of Spring exceptions:

• BindException
• ConstraintViolationException
• HttpMediaTypeNotAcceptableException
• HttpMediaTypeNotSupportedException
• HttpMessageNotReadableException
• HttpRequestMethodNotSupportedException
• MethodArgumentNotValidException
• MethodArgumentTypeMismatchException
• MethodNotAllowedException
• MissingServletRequestParameterException
• NotAcceptableStatusException
• ResponseStatusException
• ServerWebInputException
• UnsupportedMediaTypeStatusException
• WebExchangeBindException

While this works 99% of the cases, it indeed is a moving target. Yet, it works pretty good so far for us.

Logging exceptions

This is the curse and bliss of our implementation. We support logging of Problems. That is, whenever a REST controller responds with a Problem, we log it. The logging behaviour can be configured to set the logging level and filter on type. For instance, “log Problems of type /problems/unhandled-exception at ERROR level”.

Our users really much depend on this feature and love it! Yet, it is practically impossible to cover every case. There are a couple of bug reports with really weird combinations (e.g., “exceptions thrown from exception handlers in a Web MVC controller fails to log the response Problem”) we still need to figure out how to tackle. This issue is simply solved via introducing a bean extending from HandlerResultHandler and access to the response model via HandlerResult#getReturnValue(). Though this only works for the WebFlux backend. Web MVC doesn’t have such a global interceptor where one can access to the response model. WebFlux’s HandlerResultHandler equivalent in Web MVC is HandlerInterceptor and that only exposes the raw I/O stream. Long story short, intercepting Problems via a global handler that works with both WebFlux and Web MVC is not possible at this stage, to the best of our knowledge.

Conclusion

We are totally unhappy with the myriad of enhancements we need to add to make Problems work. Worse, occassionally we stumble upon corner cases that doesn’t work. We will be extremely happy to see this feature being shipped by Spring itself. I also need to note that we can afford company time to contribute to this effort.

/cc @lkleeven @mzeijen @breun @sagacity

My two cents regarding JSON errors in general: in basically every project using Spring Boot for HTTP services, and a JavaScript framework (such as Angular or Vue), I always need to customize the JSON error sent by Spring Boot in order to

• allow the client to distinguish between unexpected errors, that shouldn’t happen if the client is correctly implemented (such as forgetting to provide a required input), and expected ones that must be handled in a specific way by the client (such as providing a value that only the server can check as invalid, like a duplicate identifier or a multipart file with invalid data)
• allow the client to get an error key, used to display an i18ned error message (the i18n of the error messages is done at client side, as everything else in the pages), sometimes with a map of parameter keys and values (example: { "errorKey": "error.accountBanalnceTooLow", errorParams: { "currentBalance": 100 } })

I haven’t read the RFC very carefully yet, but if Spring Boot could provide a standard way to support this common (in my experience) need, it would be great. Or at least not lose the possibility to do it ourselves, and maybe document what the best strategy is to do it by respecting the standard.

I like to map exceptions to rfc-7807 compliant bodies on the server side and/or back to exceptions on the client side (note: they don’t have to be the same exception, they only have to be compatible). I defined an API and wrote an implementation where you can rely on useful defaults for the fields or annotate your exceptions when you have specific requirements: https://github.com/t1/problem-details

I’ve blogged about the topic in more detail: https://blog.codecentric.de/en/2020/01/rfc-7807-problem-details-with-spring-boot-and-jax-rs/

I also created a PR for Microprofile Sandbox, as I think it should be a common standard.

I would be glad to contribute the Spring Boot implementation.

In order to make the client side feel more organic, maybe we would need to scan for all exceptions?

Any feedback (except for silence 😉 is welcome!

I’ve created an issue in the Spring Framework https://github.com/spring-projects/spring-framework/issues/27052 for this.

Spring HATEOAS provides basic support for RFC-7807 since 1.1, see spring-projects/spring-hateoas#1057 and the relevant chapter of reference manual.

Now, this might not mean too much for Spring (Boot) users that don’t use Spring HATEOAS, but since RFC-7807 is IMO a generally very useful spec, I wonder whether this basic support could be moved to Spring Framework proper? Because it is something that has a clear use even without the involvement of Spring HATEOAS and/or Spring Boot. Both could then leverage the Framework support and build their own extended support on top of it. WDYT @bclozel and @gregturn?

Also, seeing this was twice marked as for: team-attention, could Spring Boot team share outcome of those discussions?

I don’t find the security angle a very compelling argument against supporting RFC 7807 Problems. I’ve been using Problem JSON for a couple of years now in a micro services architecture with many hundreds of REST API applications, and in my experience developers that come across Problem JSON mostly start producing errors that are way clearer and easier to understand for users. Over-disclosure can happen through any format, and a plain text stack trace is generally way worse than a Problem JSON object. I don’t feel that we shouldn’t have nice things just because they could contain information that shouldn’t be exposed. Not having a nice standard for errors isn’t a way to get more secure applications. Things like training, awareness, testing, etc. are what will get you more secure applications.

Just to be clear, i was posting the argument because I thought it was worth analyzing. It deserves a response. But that doesn’t mean I agree with it. I think the benefits of a general mechanism outweigh the drawbacks.

It’s an old dispute going forth and back: should we prevent good things just because they can be misused? Don’t get me wrong: this dispute is difficult and there is no simple answer. But in this case, I stand clearly on one side, because it’s a very good thing and the risk is not seriously big. And best of all, we can even implement it in a way that doesn’t reveal anything by default, so developers have to actively decide what they want to be exposed… and they do that every day with non-exceptional code — I think they can handle it.

using latest SB 3.1 with webflux, I can successfully use problem details when raising ResponseStatusException and can successfully handle other exceptions with @ExceptionHandler. But if I make a rest request that doesn’t match any of my controllers the server responds with the old format and the only way I found to fix this was by adding spring.web.resources.add-mappings=false in my application.properties file. I tried to understand this but couldn’t find any useful explanations in the docs, internet and even spent several hours digging into spring’s source code but I still cannot figure out why this is happening and how to “change” this outcome without having to disable the resources mappings. any ideas?

update: interesting thing… I managed to fix this by adding spring.webflux.static-path-pattern=/resources/** to my application.properties - so now if I make a request to a non-existing resource it returns the correct ProblemDetails response… even if I request a non-existing resource to /resources/** it comes back correctly. So I’m wondering if that’s a bug 🤔

I didn’t see much difference either, maybe the references in documentation/code to RFC 7807 should be updated in order to reference the new RFC 9457? To highlight the support of the new non-obsoleted RFC.

As RFC 7807 has been obsoleted by RFC 9457:

@rstoyanchev @bclozel are there any plans to support the format defined in the new RFC 9457?

I couldn’t find any reference to it either in Spring Framework repository nor in this Spring Boot one.

Thanks in advance

Additionally, there is already a precedent in the Spring Boot white label error page which can can be configured to show stack trace information.

@vpavic I don’t think there really was an outcome from the team discussions. We probably flagged it to discuss how easy it would be to get it into a release, but ultimately decided to prioritize other issues.