sentry-php: Could not encode value into JSON format. Error was: \"Malformed UTF-8 characters, possibly incorrectly encoded

I am using the laravel-sentry integration and I ran into this issue because I have a database table with a column of type VARBINARY and the sql.query breadcrumbs ocassionaly contained binary strings that caused the json_encode() to fail. I was able to update my queries so that instead of inserting binary data directly, I ran in through MySQL’s HEX() function to convert it to a hex string which then prevented the UTF-8 error.

I think this is an example of a simple way to reproduce this error:

        addBreadcrumb(new Breadcrumb(
            Breadcrumb::LEVEL_ERROR,
            Breadcrumb::TYPE_ERROR,
            'error_reporting',
            pack('H*', md5('binary string'))
        ));

        throw new \Exception('testing sentry');

I think the only possible options are to ignore the JSON UTF-8 errors or else to convert all the strings to UTF-8.

If you want to ignore the errors, with PHP >= 7.2 you could add JSON_INVALID_UTF8_IGNORE to Sentry\Util\JSON so that it looks like:

$encodedData = json_encode($data, JSON_UNESCAPED_UNICODE|JSON_INVALID_UTF8_IGNORE);

I did not have these errors until I upgraded from version 1.9 to 2.0 of sentry/sentry-php. It looks like in the old Raven client that it had code like mb_convert_encoding($value, 'UTF-8'); Raven Serializer handles converting everything to UTF-8 and didn’t have this issue with binary data. I don’t think Sentry can predict what type of data will be sent to it. Maybe the new version needs to convert all strings to UTF-8 to prevent this issue.

Our temporary “fix” is ignoring the errors regarding this issue, so we have something like:

/**
 * Report or log an exception.
 *
 * @param  \Exception  $exception
 * @return void
 */
public function report(Exception $exception)
{
    if ( ! app()->isLocal() && app()->bound('sentry') && $this->shouldReport($exception))
    {
        try
        {
            app('sentry')->captureException($exception);
        }
        catch (Exception $e)
        {
            // https://github.com/getsentry/sentry-php/issues/828
        }
    }

    parent::report($exception);
}

I know how ironic it is and we are missing some errors with Sentry, but still it’s the only solution we have found.

I am using the laravel-sentry integration and I ran into this issue because I have a database table with a column of type VARBINARY and the sql.query breadcrumbs ocassionaly contained binary strings that caused the json_encode() to fail. I was able to update my queries so that instead of inserting binary data directly, I ran in through MySQL’s HEX() function to convert it to a hex string which then prevented the UTF-8 error.

I think this is an example of a simple way to reproduce this error:

        addBreadcrumb(new Breadcrumb(
            Breadcrumb::LEVEL_ERROR,
            Breadcrumb::TYPE_ERROR,
            'error_reporting',
            pack('H*', md5('binary string'))
        ));

        throw new \Exception('testing sentry');

I think the only possible options are to ignore the JSON UTF-8 errors or else to convert all the strings to UTF-8.

If you want to ignore the errors, with PHP >= 7.2 you could add JSON_INVALID_UTF8_IGNORE to Sentry\Util\JSON so that it looks like:

$encodedData = json_encode($data, JSON_UNESCAPED_UNICODE|JSON_INVALID_UTF8_IGNORE);

I did not have these errors until I upgraded from version 1.9 to 2.0 of sentry/sentry-php. It looks like in the old Raven client that it had code like mb_convert_encoding($value, 'UTF-8'); Raven Serializer handles converting everything to UTF-8 and didn’t have this issue with binary data. I don’t think Sentry can predict what type of data will be sent to it. Maybe the new version needs to convert all strings to UTF-8 to prevent this issue.

wow. this can fix that Error if ($data instanceof Jsonable) { $this - > data = $data - > toJson($this - > encodingOptions); } elseif($data instanceof JsonSerializable) { $this - > data = json_encode($data - > jsonSerialize(), $this - > encodingOptions); } elseif($data instanceof Arrayable) { $this - > data = json_encode($data - > toArray(), $this - > encodingOptions); } else { // $this->data = json_encode($data, $this->encodingOptions); $ᴛʜɪꜱ - > ᴅᴀᴛᴀ = ᴊꜱᴏɴ_ᴇɴᴄᴏᴅᴇ($ᴅᴀᴛᴀ, ᴊꜱᴏɴ_ᴜɴᴇꜱᴄᴀᴘᴇᴅ_ᴜɴɪᴄᴏᴅᴇ | ᴊꜱᴏɴ_ɪɴᴠᴀʟɪᴅ_ᴜᴛꜰ8_ɪɢɴᴏʀᴇ); }

if (!$this - > hasValidJson(json_last_error())) { throw new InvalidArgumentException(json_last_error_msg()); }

What can be done to resolve this? I am in favour of the 7.2 workaround, for those of us using modern PHP it at least removes an awful hack…

As others have indicated, the primary issue is failure to encode json if accumulated event data contains binary (just surfaced for us). There’s a secondary issue here as well – the Json exception being thrown is escaping the Sentry API boundary (captureEvent). Failure to capture an event should not create a new unexpected (and undocumented) exception condition.

My solution while not fixed in repository:

composer.json "autoload": { "exclude-from-classmap": ["vendor/sentry/sentry/src/Util/JSON.php"], "psr-4": { "App\\": "src/", "Sentry\\Util\\": "src/Infrastructure/Sentry/Util/" } }

Copied \Sentry\Util\JSON to rc/Infrastructure/Sentry/Util/ folder, added JSON_INVALID_UTF8_IGNORE in json_encode function.

Any chance we could push this ?

I (have to) temporary remove sentry package, since it doesn’t report Exception due to this error

I would like to use JSON_INVALID_UTF8_SUBSTITUTE which seems to better fit this since it would replace the invalid UTF-8 character with the standard replacement char, however the polyfill cannot replace existing core functions so it won’t work. We should solve the issue on our side, probably not everything is running through the serializer and converted to UTF-8