symphonycms: Input field: control characters must be stripped

can we agree that Symphony requires unicode support ?

You mean PHP unicode support? Yes, I suggest to require this. It makes things much easier.

---

I thought about “not breaking XML (processing)”, and I assumed that it could/should be done “late”, i.e. when a text node gets added to XML. Simply because the “allowed character range” is XML specific. (I assumed that the restriction might be different if Symphony was using other templating solutions.)

I have not included any (opinionated) whitespace handling (like I myself do with the Input field). I still think that this would be the task of the field itself, or a developer might hack it into the event(s). I also do not intend to “protect the database/system from unwanted characters” here. This would be a different story (if ever needed). All I try is to “not break XML”.

My test string

Lorem 😍🙏🐪 ipsum

😍 is a 4-byte character in UTF-8. Plus, there were two DEL characters between “ipsu” and “m”! (Obviously GitHub replaces them, just like I want to do in Symphony!)

Search with PHP regex

Look at the spec again:

Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
/* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */

The simplest search regex seems to be:

'/[^\x{0009}\x{000A}\x{000D}\x{0020}-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]/u'

And if we add DEL a.k.a. \x7F (because we also consider this an unwanted control character, even if it is allowed by the spec), we have:

'/[^\x{0009}\x{000A}\x{000D}\x{0020}-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]|\x{007F}/u'

The replacement character in PHP regex

In order to output the REPLACEMENT CHARACTER in PHP 7+, we might use the “Unicode codepoint escape” syntax "\u{FFFD}". However, this is not possible in PHP 5 — here the best way seems to be json_decode('"\uFFFD"').

Putting it together

Here’s my test file:

<?php

$data = 'Lorem 😍ipsum';

echo preg_replace(
	'/[^\x{0009}\x{000A}\x{000D}\x{0020}-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]|\x{007F}/u'
	json_decode('"\uFFFD"'),
	$data
);

Again, GitHub replaces the DEL characters here. So if you want to use the code, you must re-add these control characters.

The output of this file will be Lorem 😍ipsu��m. So far, so good.

Symphony already has it — partly

I found out that since 2.3, Symphony already has XMLElement::stripInvalidXMLCharacters. But this function:

• does not allow the 4-byte characters and
• strips unwanted characters instead of replacing them.

Fortunately, the function is only used in FrontendPage::sanitizeParameter, so there’s probably no security issue caused by stripping the characters.

I suggest to either improve this function or add another one. If we keep it like it is, it should be renamed (because it does not do what the name tells us).

Adding the regex replacement to Symphony

Now here comes the problem.

I have not found a simple way to add the replacement to frontend text nodes only, without also changing the string that is displayed in the backend’s textarea after saving (which means that when saving a second time, the replacememnt character will be saved). It would be even more problematic with frontend forms, which require data source output. Honestly, I think it is not possible at all to keep a control character in the database without ever outputting it to XML — because Symphony uses XML internally.

So I guess that the idea to keep the input “as it is” and apply the character range filtering for output only has failed.

Still, I hope that my findings will help us to come up with a (simple) solution. Here are the questions:

• Do you like my test file’s preg_replace? (Suggested answer: “Yes”.)
• Shall we accept that the “XML character range” is a limitation for Symphony in general (because it uses XML internally), no matter if other template engines will be added? (Suggested answer: “Yes”.)
• Where and when should the replacement be applied?