MeshCentral: Can't consistently connect to devices with peering setup

Describe the bug I’m running two MeshCentral v1.1.19 instances configured in peering setup and I have difficulty consistently connecting to devices terminal sessions. I have experienced three different cases. It is worth to mention that my servers connect to external MongoDB and have exact the same configs excepts for serverId in peers section.

In the first case I can connect to device without any issues and there is no logs on the servers.

The second case is that it freezes on Connected and I can’t do anything. image

The third case is that it get stucked on Setup… image

In the second and the third case I can see below logs:

PEER: FTunnel meshc2: Start connect to ws://MeshCentralServer2Url:443/meshrelay.ashx?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=fq6bywz056d&rauth=QyH2Ep8vAO4R0i01$zzPFYPno6i1un5A@nqH98UQMeZoXu9GbUUkh3HlkKXNg4@SF7sLbWy9VQ0AYTG1eh20KTjr7rzBmB2JMECdDMhzEZpB3HJjbLSltA==
PEER: FTunnel meshc2: Connected

and when I click on Disconnect button…

PEER: FTunnel disconnect meshc2
PEER: FTunnel1: Soft disconnect
PEER: FTunnel2: Soft disconnect
PEER: FTunnel disconnect meshc2

Expected behavior I would expect not get stucked while trying to connect to the device.

Server Software:

  • OS: Amazon Linux 2023
  • Version: [e.g. 1.1.19]
  • Node: [e.g. 18.17.1]

Remote Device:

  • Device: Server
  • OS: Linux
  • Network: Remote over WAN
  • Current Core Version: Nov 21 2022, 598124670

Your config.json file

{
  "settings": {
    "cert": "my.domain.url",
    "WANonly": true,
    "LANonly": false,
    "sessionKey": "MyReallySecretPassword1",
    "mongoDb": "mongodb://MeshCentralMongoDBUser:MeshCentralMongoDBPassword@MeshCentralMongoDBDNS:27017/meshcentral?replicaSet=rs0",
    "mongoDbChangeStream": true,
    "port": 443,
    "_aliasPort": 443,
    "redirPort": 80,
    "_redirAliasPort": 80,
    "nightMode": 0,
    "tlsOffload": true,
    "_mpsTlsOffload": true,
    "maxInvalidLogin": {
      "time": 10,
      "count": 10,
      "coolofftime": 10
      },
    "rootCertCommonName": "xxx",
    "mpshighsecurity": true,
    "allowLoginToken": true,
    "authLog": "meshcentral-data/auth.log"
  },
  "domains": {
    "": {
      "title": "MeshCentral",
      "_title2": "Servername",
      "_minify": true,
      "newAccounts": false,
      "userNameIsEmail": true,
      "amtAcmActivation": {
        "log": "amtactivation.log",
        "certs": {
          "mycertname": {
            "certfiles": [ "xxx.crt", "XXX.crt" ],
            "keyfile": "xxx.key"
          }
        }
      },
      "ssh": true,
      "authStrategies": {
        "saml": {
          "__callbackurl": "xxx",
          "NewAccounts": true,
          "entityid": "xxx",
          "idpurl": "xxx",
          "cert": "xxx.pem"
        },
        "oidc":{
          "_authorizationURL": "xxx",
          "callbackURL": "xxx",
          "clientid": "xxx",
          "clientsecret": "xxx",
          "issuer": "xxx",
          "tokenURL": "xxx",
          "userInfoURL": "xxx",
          "newAccounts": true,
          "_groups": {
            "required": ["default"],
            "siteadmin": ["xxx"],
            "sync": {
              "enabled": true
            }
          }
        }
      }
    }
  },
  "peers": {
    "serverId": "MeshCentralServerId",
    "servers": {
      "meshc1": {
        "url": "ws://MeshCentralServerUrl:443/"
      },
      "meshc2": {
        "url": "ws://MeshCentralServer2Url:443/"
      }
    }
  }
}

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 22 (9 by maintainers)

Commits related to this issue

Most upvoted comments

I believe that did the trick. I reverted the meshcore.js changes and applied the changes to multiserver.js. I can now consistently connect to devices from different peers.

i have properly fixed it now! works a treat! https://github.com/Ylianst/MeshCentral/commit/5a410ccd5bbd8a2da3f248dc0294b731e9f9a261 the issue was actually because when the data came in from the remote device, it was in a arraybuffer! and when it was passing it through to your web browser meshcore.js was expecting it to be either string or binary! (either a command OR the desktop stream!) so just copy the fix from the commit and replace it, you can also restore the other meshcore.js as that wasnt the issue! basically it now checks if its binary and if not, convert to string first then send!

PS: notes about where the fix came from - https://github.com/websockets/ws/blob/8.17.1/doc/ws.md#event-message

@si458

Absolutely, happy to provide anything that would be of help.

  • Server OS: Windows Server 2022 Standard
  • Node Version: 22.6.0
  • MongoDB Version: 7.0.12
  • Mesh Version: 1.1.27

Setup is comprised of 3 peers communicating to a 4th database server hosting the mongodb instance.

  • mesh01
  • mesh02
  • mesh03
  • meshdb

Config:

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.",
  "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.",
  "settings": {
    "cert": "mesh.url",
    "MongoDb": "mongodb://meshdb:27017/?replicaSet=rs0",
    "mongoDbChangeStream": true,
    "mongoDbBulkOperations": true,
    "sessionKey": "MyReallySecretPassword1",
    "WANonly": true,
    "port": 443,
    "redirPort": 0,
    "mpsPort": 0,
    "exactPorts": true
  },
  "peers": {
    "servers": {
      "mesh01": { "url": "wss://10.10.22.101/" },
      "mesh02": { "url": "wss://10.10.22.102/" },
      "mesh03": { "url": "wss://10.10.22.103/" }
    }
  },
  "domains": {
    "": {
      "title": "Mesh",
      "title2": "mesh01",
      "maxDeviceView": 300,
    }
  }
}

When attempting to connect to a desktop session from a peer, I either get “Connected” with a blank/black screen or it hangs at “Setup…” image image

And some relevant output from --debug peer:

PEER: FTunnel mesh03: Start connect to wss://10.10.22.103/meshrelay.ashx?p=2&nodeid=node//xdQB6262X3st75oKxBREOtA6alG@hqsXbDdPLRUnu74R0@J4m5f@n0CqjjHwArSu&id=jkrqs324t0c&rauth=ZtyPh@hUzKol7UNsh7t2zG71PRovZz2rGSIAnFjvTTRmkaTfLZt0P7AK1R7oRPoZkvgp0a9Xgav828vJiWJhMEUP999JWUM3j2Ha$5Ssc468OrM$b7q3zA==
PEER: FTunnel mesh03: Connected
PEER: FTunnel disconnect mesh03
PEER: FTunnel1: Soft disconnect
PEER: FTunnel2: Soft disconnect
PEER: FTunnel disconnect mesh03

Rolling the meshcentral version back to 1.1.1, I am able to connect normally. In a scenario where mesh01 is running 1.1.1 and mesh02 is running 1.1.27, I am able to connect to devices on either peer. I realize 1.1.1 is quite a jump, buts its the version I’m running on a legacy solo instance and I didn’t catch reference to the latest “stable” version.

what sticks out to me is these lines after first glimpes

MeshCentralServer2: COOKIE: ERR: Bad AESGCM cookie due to exception: Error: Unsupported state or unable to authenticate data
MeshCentralServer2: COOKIE: ERR: Bad AESSHA cookie due to exception: Error: error:1C80006B:Provider routines::wrong final block length

will look into it for you when i get chance