laravel-aws-worker: UTF-8 characters unserialize() error at offset

To piggyback on @jdempster 's answer, I had permission issues using a prebuild hook, so I decided to put it all in an an ebextension file called .ebextensions/fix-xml-parsing.config, if that’s helpful to anyone. Not sure yet if this will actually solve the problem, as I can’t reliably replicate the error, but I’m hopeful! I’ll post an update once enough time has passed that I can be confident it’s solved.

UPDATE: This appears to have worked, we haven’t run into any new unserialize() errors in the past 2 days. Thanks to @jdempster for this solution!

packages:
  yum:
    patch: []

commands:
  01_apply_patch:
    command: "/tmp/fix-xml-parsing.sh"

files:
  "/tmp/aws-sdk-core-xml-parser-engines-ox.patch":
      mode: "000755"
      content: |
        --- ox.rb       2021-04-09 20:20:20.862069509 +0000
        +++ ox-new.rb   2021-04-09 20:21:05.181225573 +0000
        @@ -14,7 +14,7 @@
                  Ox.sax_parse(
                    @stack, StringIO.new(xml),
                    :convert_special => true,
        -            :skip => :skip_white
        +            :skip => :skip_none
                  )
                end
  "/tmp/fix-xml-parsing.sh":
      mode: "000755"
      content : |
        #!/bin/bash

        if [ -f "/.aws-sdk-core-xml-parser-engines-ox-fix-installed" ]; then
            exit 0
        fi

        if systemctl list-units --full -all | grep -Fq "sqsd.service"; then
            /usr/bin/patch --ignore-whitespace -u /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.6.0/gems/aws-sdk-core-2.11.546/lib/aws-sdk-core/xml/parser/engines/ox.rb -i /tmp/aws-sdk-core-xml-parser-engines-ox.patch
            touch /.aws-sdk-core-xml-parser-engines-ox-fix-installed
            systemctl restart sqsd
            echo "Installed xml parsing patch!"
            exit 0
        fi

        exit 0

Since upgrading elastic beanstalk to Amazon Linux 2 we’ve come across the same issue. In the end, I’ve found the problem is in the ruby aws SDK XML parser, it’s configured to strip whitespace. /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.6.0/gems/aws-sdk-core-2.11.546/lib/aws-sdk-core/xml/parser/engines/ox.rb

The solution for me was to create a prebuild hook to patch the ruby file.

#!/bin/bash -eu

if [ -f "/.aws-sdk-core-xml-parser-engines-ox-fix-installed" ]; then
    exit 0
fi

sudo yum install -y patch
patch -u /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.6.0/gems/aws-sdk-core-2.11.546/lib/aws-sdk-core/xml/parser/engines/ox.rb -i .platform/files/aws-sdk-core-xml-parser-engines-ox.patch
touch /.aws-sdk-core-xml-parser-engines-ox-fix-installed
systemctl restart sqsd

--- ox.rb       2021-04-09 20:20:20.862069509 +0000
+++ ox-new.rb   2021-04-09 20:21:05.181225573 +0000
@@ -14,7 +14,7 @@
           Ox.sax_parse(
             @stack, StringIO.new(xml),
             :convert_special => true,
-            :skip => :skip_white
+            :skip => :skip_none
           )
         end

We are using beanstalk and we were able to talk to an AWS tech and they said they were able to reproduce this issue on their side. It apparently is some sort of bug with the daemon they have that reads from the sqs queue and shoves it into where the laravel worker can read it. We weren’t giving a timeline on a fix or if they will even fix it but that’s what we found out.

@Nilanth We have a separate container for jobs and laravel reads directly from SQS. Pretty sure it’s just built into laravel (I don’t use laravel we are helping other products on our team that do).

@Nilanth No updates at all. We are now moving to docker and ECS to use SQS more directly to go around their daemon that’s breaking everything. We have some silly hacky fixes similar to what @JulioPablo did for the time being. technically the base64 encoding of your message would be the best “fix” we had that work also but feels terrible to code this way. best of luck!

Just upgraded to Laravel 8 and getting this issue. Any idea to find out which ones are an issue?

We are running into unserialization issues also, for us it seems to be based around two spaces in a row. We haven’t been able to figure out how or why this is happening. There are no special characters or anything, if we remove one of the spaces in the sqs message it works fine so two spaces is somehow an issue.