spring-framework: Spring is inconsistent in the encoding/decoding of URLs [SPR-16860]

J. Pablo Fernández opened SPR-16860 and commented

I have a Spring Boot server application and a client that uses RestTemplate to talk to it. I’m finding that RestTemplate doesn’t encode plus signs (“+”) while Spring decodes them as spaces.

For example, this snippet of code:

String foo = "fo+o";
String bar = "ba r";
restTemplate.exchange("http://example.com/?foo={foo}&bar={bar}", HttpMethod.GET, null, foo, bar)

will make a request to

http://example.com/?foo=fo+o&bar=ba%20r

which means, inside Spring, the variable foo will contain “fo o” and the variable bar will contain “ba r”. Using UriTemplate directly, it’s easier to reproduce the generation of that URL:

String foo = "fo+o";
String bar = "ba r";
UriTemplate uriTemplate = new UriTemplate("http://example.com/?foo={foo}&bar={bar}");
Map<String, String> vars = new HashMap<>();
vars.put("foo", foo);
vars.put("bar", bar);
URI uri = uriTemplate.expand(vars);
System.out.println(uri);

which prints

http://example.com/?foo=fo+o&bar=ba%20r

Not even UriComponentsBuilder encodes the plus sign:

String foo = "fo+o";
String bar = "ba r";
UriComponentsBuilder ucb = UriComponentsBuilder.fromUriString("http://example.com/?foo={foo}&bar={bar}");
System.out.println(ucb.build().expand(foo, bar).toUri());
System.out.println(ucb.build().expand(foo, bar).toString());
System.out.println(ucb.build().expand(foo, bar).toUriString());
System.out.println(ucb.build().expand(foo, bar).encode().toUri());
System.out.println(ucb.build().expand(foo, bar).encode().toString());
System.out.println(ucb.build().expand(foo, bar).encode().toUriString());
System.out.println(ucb.buildAndExpand(foo, bar).toUri());
System.out.println(ucb.buildAndExpand(foo, bar).toString());
System.out.println(ucb.buildAndExpand(foo, bar).toUriString());
System.out.println(ucb.buildAndExpand(foo, bar).encode().toUri());
System.out.println(ucb.buildAndExpand(foo, bar).encode().toString());
System.out.println(ucb.buildAndExpand(foo, bar).encode().toUriString());

That prints:

http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba r
http://example.com/?foo=fo+o&bar=ba r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba r
http://example.com/?foo=fo+o&bar=ba r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba%20r

all of which are incorrect, not encoding the plus sign.

I have posted questions about this on StackOverflow that further point to other bug reports and there’s a lot of confusion around this subject:

https://stackoverflow.com/questions/50270372/why-is-spring-de-coding-the-plus-character-on-application-json-get-requests

https://stackoverflow.com/questions/50432395/whats-the-proper-way-to-escape-url-variables-with-springs-resttemplate-when-ca


Affects: 5.0.5

Issue Links:

  • #22006 Not encoding ‘+’ in URLs anymore breaks backwards compatibility with apps running on spring 4 (“is duplicated by”)
  • #21577 Support stricter encoding of URI variables in UriComponents (“is superseded by”)

0 votes, 6 watchers

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 5
  • Comments: 20

Most upvoted comments

Christophe Levesque commented

I did have a look at the reference and assume you refer about this bit:

By default UriComponents encodes only characters that are illegal within a given URI component, but not all characters with reserved meaning.

I guess here the “+” sign is not illegal but has a reserved meaning (it’s decoded as a space). So I understand that the behavior is somewhat explained by the documentation. What is really unfortunate, is that the UriComponentBuilder was a great utility to safely and concisely generate URLs without having to manually escape / encode parameters. All you had to do is write:

UriComponentsBuilder.fromHttpUrl(baseUrl).queryParam("foo", foo).toUriString() 

You could throw pretty much any query param value at it (with or without “+” sign or special character) and it would just work (the server receiving the request would get the param with the same value as what you passed to the queryParam method). This was the behavior for years.

Now, the new UriComponentsBuilder feels broken and violates the Principle of least astonishment. The code above works in almost all cases except if the param contains a “+” sign. In which case you’re pretty much SOL as there is no simple workaround to get it encoded as %2B other than jumping through hoops:

  • instantiate a DefaultUriBuilderFactory,
  • switch the encoding more to VALUES_ONLY,
  • switch to full blown param templating (since “strict encoding is only applied to expanded URI variable values”).

A simple one-liner becomes an odyssey. 😛

Rossen Stoyanchev: I hope you’ll consider addressing this!

Mike Placentra commented

Rossen pointed out to me in #21473 that UriUtils#encode(String, Charset) has the behavior we’re looking for, that’s a good workaround in cases where an URL builder isn’t needed.

That being said, in the time leading up to submitting that ticket and until now, I’ve been unsuccessful in convincing myself that having the encoder encode “+” as “%2B” by default would “break” compliance with RFC 3986. (This is separate from decoding “+” as a space by default – I agree that’s bad.) I’ve looked through the RFC several times (basically each time I stub my toe on this issue in Spring Web) and it certainly gives us these points:

  • Since “+” (and the other reserved characters) are valid, a decoder would be non-compliant if it threw an exception when it saw them (as #19394 addresses)
  • When “normalizing” an already-formed URI for comparison with another URI (e.g. to look it up in a cache), “+” (and other reserved characters) must be left alone (must be considered different from “%2B”) because they may be implementation-specific delimiters.

It goes far enough to ensure that 1) once a URI component is formed it is composed of only characters that make it distinguishable from other components in the URI, 2) a (percent) encoding mechanism is available for escaping other characters, and 3) URI normalizers (for comparing URIs) treat (potentially) implementation-specific delimiters as first-class delimiters.

Otherwise, the RFC largely leaves the implementation-specific encoding of data and data structures in a URI component out-of-scope. As far as I can tell, there’s no rule against overzealously encoding data “abcd” as “%61%62%63%64” while forming the URI, and likewise there’s no rule against encoding data “+” as “%2B” while forming the URI.

In fact, the RFC says this in section 2.2:

If data for a URI component would conflict with a reserved character’s purpose as a delimiter, then the conflicting data must be percent-encoded before the URI is formed.

It could be argued, given that a vanilla Spring Web application relies on Tomcat to decode request URIs, and Tomcat has an implementation-specific purpose for “+” as a delimiter (it decodes it as a space, I suppose because it assumes the value is a phrase or prose and we’re delimiting the words), and a URI generated by a Spring Web application is likely to be subsequently decoded by the same or another Spring Web application, +data “+” must be encoded as “%2B” according to RFC 3986+ because the data “would conflict with a reserved character’s purpose as a delimiter”.

Also, I don’t know how much weight it holds, but here’s a W3C recommendation that says to encode “+” as “%2B”.

I couldn’t imagine encoding “+” as “%2B” by default would cause grief for anyone because any URI parser will properly decode that back to a “+”. It could be an issue if someone is designing their own implementation-specific delimiter system for the URI component (encoding complex data structures), but then I probably wouldn’t use UriComponentsBuilder, or I would research its functionality first and discover that DefaultUriBuilderFactory is more configurable.

Brian Clozel commented

Christophe Levesque, I understand your frustration - especially since this was working a way that matched your needs for quite a while.

Looking at #19394, that’s actually the issue violating that principle of least astonishment: "+" was the only character that wasn’t dealt with the way it’s meant to be by the RFC. Why make that exception?

We don’t have many options here:

  • revert that fix, post 5.0 and break RFC compliance and all developers counting on the fix in #19394
  • improve even more the reference documentation on that (suggestions are welcome)

It’s quite unfortunate, but you (and many others) were relying on a bug, or at least an implementation out of sync with the RFC. URL encoding is a very complex matter and we’re trying to make things developer-friendly while maintaining RFC compliance.

The DefaultUriBuilderFactory is in my opinion not a bad choice here, as it’s helping you to clarify the hard parts about URL encoding while still being flexible about your opinions.

DefaultUriBuilderFactory factory = new DefaultUriBuilderFactory();
factory.setEncodingMode(DefaultUriBuilderFactory.EncodingMode.VALUES_ONLY);
URI uri = factory.uriString("https://spring.io/")
			.queryParam("query", "{query}")
			.build("spring+framework"); // http://spring.io/?query=spring%2Bframework

I also got this problem (originally from #14464), and my workaround is to use Apache URIBuilder instead: https://stackoverflow.com/a/50959115/122441

URIBuilder ub = new URIBuilder("http://example.com/query");
ub.setParameter("q", "foo+bar"); // foo%2Bbar
URI uri = ub.build();

I hope this is useful for other people having the same problem.