spring-framework: Spring is inconsistent in the encoding/decoding of URLs [SPR-16860]
J. Pablo Fernández opened SPR-16860 and commented
I have a Spring Boot server application and a client that uses RestTemplate to talk to it. I’m finding that RestTemplate doesn’t encode plus signs (“+”) while Spring decodes them as spaces.
For example, this snippet of code:
String foo = "fo+o";
String bar = "ba r";
restTemplate.exchange("http://example.com/?foo={foo}&bar={bar}", HttpMethod.GET, null, foo, bar)
will make a request to
http://example.com/?foo=fo+o&bar=ba%20r
which means, inside Spring, the variable foo will contain “fo o” and the variable bar will contain “ba r”. Using UriTemplate directly, it’s easier to reproduce the generation of that URL:
String foo = "fo+o";
String bar = "ba r";
UriTemplate uriTemplate = new UriTemplate("http://example.com/?foo={foo}&bar={bar}");
Map<String, String> vars = new HashMap<>();
vars.put("foo", foo);
vars.put("bar", bar);
URI uri = uriTemplate.expand(vars);
System.out.println(uri);
which prints
http://example.com/?foo=fo+o&bar=ba%20r
Not even UriComponentsBuilder encodes the plus sign:
String foo = "fo+o";
String bar = "ba r";
UriComponentsBuilder ucb = UriComponentsBuilder.fromUriString("http://example.com/?foo={foo}&bar={bar}");
System.out.println(ucb.build().expand(foo, bar).toUri());
System.out.println(ucb.build().expand(foo, bar).toString());
System.out.println(ucb.build().expand(foo, bar).toUriString());
System.out.println(ucb.build().expand(foo, bar).encode().toUri());
System.out.println(ucb.build().expand(foo, bar).encode().toString());
System.out.println(ucb.build().expand(foo, bar).encode().toUriString());
System.out.println(ucb.buildAndExpand(foo, bar).toUri());
System.out.println(ucb.buildAndExpand(foo, bar).toString());
System.out.println(ucb.buildAndExpand(foo, bar).toUriString());
System.out.println(ucb.buildAndExpand(foo, bar).encode().toUri());
System.out.println(ucb.buildAndExpand(foo, bar).encode().toString());
System.out.println(ucb.buildAndExpand(foo, bar).encode().toUriString());
That prints:
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba r
http://example.com/?foo=fo+o&bar=ba r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba r
http://example.com/?foo=fo+o&bar=ba r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba%20r
http://example.com/?foo=fo+o&bar=ba%20r
all of which are incorrect, not encoding the plus sign.
I have posted questions about this on StackOverflow that further point to other bug reports and there’s a lot of confusion around this subject:
Affects: 5.0.5
Issue Links:
- #22006 Not encoding ‘+’ in URLs anymore breaks backwards compatibility with apps running on spring 4 (“is duplicated by”)
- #21577 Support stricter encoding of URI variables in UriComponents (“is superseded by”)
0 votes, 6 watchers
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 5
- Comments: 20
Christophe Levesque commented
I did have a look at the reference and assume you refer about this bit:
I guess here the “+” sign is not illegal but has a reserved meaning (it’s decoded as a space). So I understand that the behavior is somewhat explained by the documentation. What is really unfortunate, is that the
UriComponentBuilderwas a great utility to safely and concisely generate URLs without having to manually escape / encode parameters. All you had to do is write:You could throw pretty much any query param value at it (with or without “+” sign or special character) and it would just work (the server receiving the request would get the param with the same value as what you passed to the
queryParammethod). This was the behavior for years.Now, the new
UriComponentsBuilderfeels broken and violates the Principle of least astonishment. The code above works in almost all cases except if the param contains a “+” sign. In which case you’re pretty much SOL as there is no simple workaround to get it encoded as%2Bother than jumping through hoops:DefaultUriBuilderFactory,VALUES_ONLY,A simple one-liner becomes an odyssey. 😛
Rossen Stoyanchev: I hope you’ll consider addressing this!
Mike Placentra commented
Rossen pointed out to me in #21473 that UriUtils#encode(String, Charset) has the behavior we’re looking for, that’s a good workaround in cases where an URL builder isn’t needed.
That being said, in the time leading up to submitting that ticket and until now, I’ve been unsuccessful in convincing myself that having the encoder encode “+” as “%2B” by default would “break” compliance with RFC 3986. (This is separate from decoding “+” as a space by default – I agree that’s bad.) I’ve looked through the RFC several times (basically each time I stub my toe on this issue in Spring Web) and it certainly gives us these points:
It goes far enough to ensure that 1) once a URI component is formed it is composed of only characters that make it distinguishable from other components in the URI, 2) a (percent) encoding mechanism is available for escaping other characters, and 3) URI normalizers (for comparing URIs) treat (potentially) implementation-specific delimiters as first-class delimiters.
Otherwise, the RFC largely leaves the implementation-specific encoding of data and data structures in a URI component out-of-scope. As far as I can tell, there’s no rule against overzealously encoding data “abcd” as “%61%62%63%64” while forming the URI, and likewise there’s no rule against encoding data “+” as “%2B” while forming the URI.
In fact, the RFC says this in section 2.2:
It could be argued, given that a vanilla Spring Web application relies on Tomcat to decode request URIs, and Tomcat has an implementation-specific purpose for “+” as a delimiter (it decodes it as a space, I suppose because it assumes the value is a phrase or prose and we’re delimiting the words), and a URI generated by a Spring Web application is likely to be subsequently decoded by the same or another Spring Web application, +data “+” must be encoded as “%2B” according to RFC 3986+ because the data “would conflict with a reserved character’s purpose as a delimiter”.
Also, I don’t know how much weight it holds, but here’s a W3C recommendation that says to encode “+” as “%2B”.
I couldn’t imagine encoding “+” as “%2B” by default would cause grief for anyone because any URI parser will properly decode that back to a “+”. It could be an issue if someone is designing their own implementation-specific delimiter system for the URI component (encoding complex data structures), but then I probably wouldn’t use UriComponentsBuilder, or I would research its functionality first and discover that DefaultUriBuilderFactory is more configurable.
Brian Clozel commented
Christophe Levesque, I understand your frustration - especially since this was working a way that matched your needs for quite a while.
Looking at #19394, that’s actually the issue violating that principle of least astonishment:
"+"was the only character that wasn’t dealt with the way it’s meant to be by the RFC. Why make that exception?We don’t have many options here:
It’s quite unfortunate, but you (and many others) were relying on a bug, or at least an implementation out of sync with the RFC. URL encoding is a very complex matter and we’re trying to make things developer-friendly while maintaining RFC compliance.
The
DefaultUriBuilderFactoryis in my opinion not a bad choice here, as it’s helping you to clarify the hard parts about URL encoding while still being flexible about your opinions.I also got this problem (originally from #14464), and my workaround is to use Apache URIBuilder instead: https://stackoverflow.com/a/50959115/122441
I hope this is useful for other people having the same problem.