core: HAproxy config can't be synced to secondary node - fails with "parse error. not well formed"

Describe the bug After upgrading to 20.7.7 - I’m unable to sync my primary to secondary node. Basic troubleshooting reveals that when turning off HAproxy in config sync, the sync continues.

Before the upgrade, things were running fine. Downgrading via opnsense-revert -r 20.7.6 opnsense doesn’t change the behaviour. Note that the HAproxy config is valid and running fine.

To Reproduce Upgrade from 20.7.6 to 20.7.7 with an elaborate haproxy config; following error is thrown on sync: parse error. not well formed

Expected behavior I expect the sync to work. I’ve already checked if no special characters are used in descriptions of backends / rules / etc… and this isn’t the case. After taking a snapshot in proxmox of the systems in question, and starting to “remove” parts of the haproxy config; at some point I’m greeted with an API error and it might be resolved - but it’s inconsistend in when it’s resolved. (sometimes it’s magically fixed when removing a rule, sometimes a backend or virtual service,…)

Relevant log files

</struct></value>
</data></array></value></member>
</struct></value>
</data></array></value></member>
  <member><name>dnssec</name><value><string>1</string></value></member>
</struct></value></member>
</struct></value></param>
</params></methodCall>received >>>
<?xml version="1.0"?>
<methodResponse>
  <fault>
    <value>
      <struct>
        <member>
          <name>faultCode</name>
          <value><int>-32700</int></value>
        </member>
        <member>
          <name>faultString</name>
          <value><string>parse error. not well formed</string></value>
        </member>
      </struct>
    </value>
  </fault>
</methodResponse>
error >>>
parse error. not well formed

Environment OpnSense 20.7.7 / Proxmox / CARP setup

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 94 (62 by maintainers)

Commits related to this issue

xmlrpc: err, what, send EOL for #4533 — committed to opnsense/core by fichtner 4 years ago
HAsync(xmlrpc) : cleanup rc.filter_synchronize and expose "debug" parameter to output exchanged data. /usr/local/etc/rc.filter_synchronize debug ref https://github.com/opnsense/core/issues/4533 — committed to opnsense/core by AdSchellevis 4 years ago
system: pick up recommendations from #4533 Since the new default is TLS 1.2 only use the upstream recommended list and fall back to TLS 1.1 and above for less secure ciphers if the user specificed th... — committed to opnsense/core by fichtner 3 years ago
HAsync(xmlrpc) : cleanup rc.filter_synchronize and expose "debug" parameter to output exchanged data. /usr/local/etc/rc.filter_synchronize debug ref https://github.com/opnsense/core/issues/4533 — committed to DynFi/opnsense-core by AdSchellevis 4 years ago

Most upvoted comments

lighttpd developer here: My sincere apologies for the trouble with lighttpd 1.4.56 and lighttpd 1.4.57.

lighttpd 1.4.56 was a major, major release with large rewrites for HTTP/2 and multiple TLS library options, as well as HTTP/1.1 chunked enhancement (which was the source of bugs here)

With lighttpd 1.4.58, I believe all the bugs have been shaken out. At least all the reported bugs have been addressed.

In the future, if you do notice regressions, please report them upstream on https://github.com/lighttpd/lighttpd1.4 or https://redmine.lighttpd.net/projects/lighttpd/issues

Thanks.

gstrauss on Jan 4, 2021

FYI, while looking at src/etc/inc/plugins.inc.d/webgui.inc, here are a couple things you might want to do to simplify your config and improve performance:

server.event-handler  = "freebsd-kqueue"
server.network-backend  = "writev"

If you leave these out, lighttpd will prefer to use kqueue on systems where available, so that is the default. If you leave out server.network-backend, then lighttpd will prefer the system sendfile if available. In the distant past, you may have needed to specify server.network-backend = "writev" on FreeBSD on embedded systems, since not all filesystems support sendfile, especially some flash filesystems. However, lighttpd has since been enhanced to fall back to writev if sendfile fails.

Most of the OPNsense ssl.* configuration directives are either the default or have been superceded. lighttpd 1.4.56 and later use "MinProtocol" => "TLSv1.2" by default. Also, when your cipher list is restricted to a set which only supports Perfect Forward Secrecy, it is beneficial (and not a reduction in security) to allow the client to choose its preferred cipher. This can greatly speed up mobile clients, which might prefer CHACHA20 over AESGCM since some mobile devices do not have hardware accelerated AESGCM. Much of this is docmented at https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_SSL though can be pretty dense reading, which is why simpler SSL configs are better.

ssl.pemfile = "..."
ssl.ca-file = "..."
ssl.openssl.ssl-conf-cmd = ("MinProtocol" => "TLSv1.2",
                            "Options" => "-ServerPreference",
                            "CipherString" => "EECDH+AESGCM:AES256+EECDH:CHACHA20:!SHA1:!SHA256:!SHA384")

$SERVER["socket"] == "*:443"     { ssl.engine = "enable" }
$SERVER["socket"] == "*[::]:443" { ssl.engine = "enable" }

With the above config, it is not necessary to set ssl.disable-client-renegotiation = "enable", which is the default in lighttpd, and it is not necessary to set ssl.dh-file since a proper set of parameters are used by (modern) TLS libraries. Similar for ssl.ec-curve.

lighttpd sets "HTTPS" => "on" in the environment when necessary, so you should not need to do so.

The above config "MinProtocol" => "TLSv1.2" supercedes ssl.use-sslv2 = "disable" and ssl.use-sslv3 = "disable" and, along with a restricted cipher list, changes ssl.honor-cipher-order. However, if a custom cipher list is configured by the OPNsense user, you may want to omit "Options" => "-ServerPreference" and have lighttpd use the lighttpd default which historically has been the equivalent of "Options" => "ServerPreference" (ssl.honor-cipher-order = "enable")

In addition, as you currently do, "Strict-Transport-Security" should be set when configured.

$HTTP["scheme"] == "https" {
            setenv.add-response-header = ("Strict-Transport-Security" => "max-age=15768000" )
}

https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModSetEnv#HTTP-Strict-Transport-Security-HSTS

gstrauss on Jan 5, 2021

hotfix out in a couple of minutes

fichtner on Jan 4, 2021

@fichtner I did a quick test build of lighttpd-1.4.58 (OpenSSL) on my private poudriere. I can confirm that it fixes the issue. The build options are different, so I can’t share the package with other people.

I’ve repeated the test multiple times with 1.4.56 and 1.4.58 – .56 always failed, .58 always OK.

fraenki on Dec 29, 2020

@fichtner Hotfix works as expected, thank you!

fraenki on Jan 4, 2021

worth a try, thanks!

fichtner on Dec 21, 2020

# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/lighttpd-1.4.57.txz && configctl webgui restart

fichtner on Dec 21, 2020

if it’s fixed with downgrading lighttpd, you probably just have to wait for an update, the only odd thing is that the xml content received by the client is ok, which doesn’t make sense in that case.

AdSchellevis on Dec 21, 2020

Didn’t we already establish latest lighttpd was broken somehow? I assume they have issues with request compression or chunking. Let me distribute a 1.4.57 version for you from last week…

fichtner on Dec 21, 2020

only the shaper section is also ok?

AdSchellevis on Dec 21, 2020

the output in this thread isn’t complete, the script executes multiple command, one fails and the last one doesn’t (look for multiple send : tags and the responses it received).

AdSchellevis on Dec 21, 2020

better remove the auth header 😃

AdSchellevis on Dec 21, 2020

can you post the complete message including the “send :” text?

AdSchellevis on Dec 21, 2020