skupper: skupper router fails to connect on edge site

i’ve got a skupper edge site with the skupper router failing to finish setting up. as a result the service controller never fully starts on the edge. skuppered services are generated in the cloud, but not on the edge. (specific IPs redeacted.)

2021-03-31 23:55:40.975467 +0000 SERVER (info) Container Name: boop-skupper-router-7595f6b8ff-2699f
2021-03-31 23:55:40.975709 +0000 ROUTER (info) Router started in Edge mode
2021-03-31 23:55:40.975739 +0000 ROUTER (info) Version: 1.16.0-SNAPSHOT
2021-03-31 23:55:40.976282 +0000 ROUTER_CORE (info) Core module enabled: edge_router
2021-03-31 23:55:40.976350 +0000 ROUTER_CORE (info) Core module present but disabled: core_test_hooks
2021-03-31 23:55:40.976365 +0000 ROUTER_CORE (info) Core module present but disabled: edge_addr_tracking
2021-03-31 23:55:40.976377 +0000 ROUTER_CORE (info) Core module present but disabled: address_lookup_server
2021-03-31 23:55:40.976391 +0000 ROUTER_CORE (info) Core module enabled: address_lookup_client
2021-03-31 23:55:40.976442 +0000 ROUTER_CORE (info) Stuck delivery detection: Scan interval: 30 seconds, Delivery age threshold: 10 seconds
2021-03-31 23:55:40.976459 +0000 ROUTER_CORE (info) Core module enabled: stuck_delivery_detection
2021-03-31 23:55:40.976471 +0000 ROUTER_CORE (info) Core module present but disabled: mobile_sync
2021-03-31 23:55:40.976490 +0000 ROUTER_CORE (info) Streaming link scrubber: Scan interval: 30 seconds, max free pool: 128 links
2021-03-31 23:55:40.976509 +0000 ROUTER_CORE (info) Core module enabled: streaming_link_scrubber
2021-03-31 23:55:40.976529 +0000 ROUTER_CORE (info) Protocol adaptor registered: http/1.x
2021-03-31 23:55:40.976547 +0000 ROUTER_CORE (info) Protocol adaptor registered: tcp
2021-03-31 23:55:40.976562 +0000 ROUTER_CORE (info) Protocol adaptor registered: http2
2021-03-31 23:55:40.976975 +0000 ROUTER_CORE (info) Router Core thread running. 0/boop-skupper-router-7595f6b8ff-2699f
2021-03-31 23:55:40.977225 +0000 ROUTER_CORE (info) Protocol adaptor registered: amqp
2021-03-31 23:55:40.980888 +0000 ROUTER_CORE (info) In-process subscription M/$management
2021-03-31 23:55:40.981143 +0000 ROUTER_CORE (info) In-process subscription L/$management
2021-03-31 23:55:41.014647 +0000 AGENT (info) Activating management agent on $_management_internal
2021-03-31 23:55:41.014840 +0000 ROUTER_CORE (info) In-process subscription L/$_management_internal
2021-03-31 23:55:41.020459 +0000 POLICY (info) Policy configured maxConnections: 65535, policyDir: '',access rules enabled: 'false', use hostname patterns: 'false'
2021-03-31 23:55:41.023929 +0000 POLICY (info) Policy fallback defaultVhost is defined: '$default'
2021-03-31 23:55:41.025079 +0000 CONN_MGR (info) Created SSL Profile with name skupper-amqps
2021-03-31 23:55:41.026923 +0000 CONN_MGR (info) Created SSL Profile with name default-connection-token-profile
2021-03-31 23:55:41.037242 +0000 CONN_MGR (info) Configured Connector: xxx.xxx.xxx.xxx:45671 proto=any, role=edge, sslProfile=default-connection-token-profile
2021-03-31 23:55:41.040646 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:5671 proto=any, role=normal, sslProfile=skupper-amqps
2021-03-31 23:55:41.055430 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:9090 proto=any, role=normal, http
2021-03-31 23:55:41.055874 +0000 SERVER (info) HTTP server thread running
2021-03-31 23:55:41.056139 +0000 SERVER (notice) Listening for HTTP on 0.0.0.0:9090
2021-03-31 23:55:41.060412 +0000 CONN_MGR (info) Configured Listener: localhost:5672 proto=any, role=normal
2021-03-31 23:55:41.065971 +0000 SERVER (notice) Operational, 4 Threads Running (process ID 1)
2021-03-31 23:55:41.066326 +0000 SERVER (notice) Process VmSize 185.86 MiB (5.64 GiB available memory)
2021-03-31 23:55:41.066806 +0000 SERVER (notice) Listening on 0.0.0.0:5671
2021-03-31 23:55:41.066901 +0000 SERVER (notice) Listening on localhost:5672
2021-03-31 23:55:41.140350 +0000 SERVER (info) [C1] Connection to xxx.xxx.xxx.xxx:45671 failed: amqp:connection:framing-error SSL Failure: error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding
2021-03-31 23:55:46.214091 +0000 SERVER (info) [C2] Connection to xxx.xxx.xxx.xxx:45671 failed: amqp:connection:framing-error SSL Failure: error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding
....

the invalid padding error repeats indefinitely.

both central and edge clusters are running Skupper 0.5.2 cloud site cm data:

  cluster-local: "false"
  console: "true"
  console-authentication: internal
  console-user: <REDACTED>
  console-password: <REDACTED>
  console-ingress: "none"
  edge: "false"
  name: beep
  router-console: "true"
  service-controller: "true"
  service-sync: "true"

edge site cm data:

  cluster-local: "true"
  console: "false"
  edge: "true"
  name: boop
  router-console: "false"
  service-controller: "true"
  service-sync: "true"

also seeing this repeating in cloud’s skupper-router logs:

SERVER (info) [C15] Accepted connection to 0.0.0.0:45671 from 10.168.0.40:13011
SERVER (info) [C15] Connection from 10.168.0.40:13011 (to 0.0.0.0:45671) failed: amqp:connection:framing-error SSL Failure: Unknown error