MeshCentral: Very high memory on customized agent after long time

Describe the bug I have encountered this on both Linux (Debian and Fedora) and Windows Server 2022. Right now I only see it on Windows, so I’m reporting this as a Windows issue. Agent is customized with my name

To Reproduce Steps to reproduce the behavior:

  1. Install MeshCentral agent
  2. Let the server run for a few days
  3. See Agent memory usage slowly creeping up. Starts at like 200MB, can go up to a gig
  4. After restarting the service it goes back down

Screenshots image

Server Software (please complete the following information):

  • OS: Debian 12
  • Virtualization: Docker
  • Network: LAN, MeshCentral in WAN mode behind NGINX Proxy Manager
  • Version: typhonragewind/meshcentral:mongodb-1.1.20
  • Node: Node.js v20.11.0

Remote Device (please complete the following information):

  • Device: KVM in Proxmox
  • OS: Windows Server 2022 Standard Eval
  • Network: LAN, MeshCentral in WAN mode
  • Current Core Version (if known): Current Core: Dec 9 2022, 1992526869

Additional context Add any other context about the problem here.

Your config.json file

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
    "_maintenanceMode": true,
    "cert": "[redacted]",
    "_SQLite3": true,
    "mongodb": "mongodb://mongodb:27017/mesh",
    "mongodbcol": "mesh",
    "WANonly": true,
    "sessionKey": "null",
    "port": 443,
    "aliasPort": 443,
    "redirPort": 80,
    "redirAliasPort": 80,
    "AgentPong": 300,
    "TLSOffload": "127.0.0.1",
    "_ignoreAgentHashCheck": true,
    "SelfUpdate": false,
    "AllowFraming": true,
    "WebRTC": false,
    "nice404": true,
    "allowHighQualityDesktop": true,
    "publicPushNotifications": false,
    "trustedProxy": "nginx-proxy-manager"
  },
  "domains": {
    "": {
      "_siteStyle": 2,
      "title": "MeshCentral",
      "title2": "[redacted]",
      "_titlePicture": "title-sample.png",
      "_loginPicture": "title-sample.png",
      "mobileSite": true,
      "maxDeviceView": 200,
      "_unknownUserRootRedirect": "https://www.youtube.com/watch?v=2Q_ZzBGPdqE",
      "nightMode": 0,
      "ipkvm": false,
      "minify": true,
      "newAccounts": false,
      "_welcomeText": "Sample Welcome Test.",
      "_welcomePicture": "mainwelcome.jpg",
      "_welcomePictureFullScreen": false,
      "meshMessengerTitle": "MeshMessenger",
      "_meshMessengerPicture": "messenger.png",
      "___hide__": "Sum of: 1 = Hide header, 2 = Hide tab, 4 = Hide footer, 8 = Hide title, 16 = Hide left bar, 32 = Hide back buttons",
      "hide": 4,
      "footer": "<a href='f'>Homepage</a>",
      "loginfooter": "This is not a public service.",
      "allowSavingDeviceCredentials": true,
      "guestDeviceSharing": true,
      "_AutoRemoveInactiveDevices": 37,
      "_DeviceSearchBarServerAndClientName": false,
      "_agentSelfGuestSharing": {
        "expire": 120
      },
      "certUrl": "https://meshcentral.[redacted]:443",
      "deviceMeshRouterLinks": {
        "rdp": true,
        "ssh": true,
        "scp": true
      },
      "_PreconfiguredScripts": [
        {
          "name": "Run NotePad as user",
          "file": "scripts/notepad.bat",
          "type": "bat",
          "runas": "user"
        },
        {
          "name": "Run NotePad as agent",
          "cmd": "notepad.exe",
          "type": "bat",
          "runas": "agent"
        },
        {
          "name": "Run echo",
          "cmd": "echo \"hello world\"",
          "type": "sh",
          "runas": "agent"
        },
        {
          "name": "Agent Update",
          "cmd": "agentupdate",
          "type": "agent"
        }
      ],
      "_PreconfiguredRemoteInput": [
        {
          "name": "CompanyUrl",
          "value": "[redacted]"
        },
        {
          "name": "Any Text",
          "value": "Any text\r"
        },
        {
          "name": "Welcome",
          "value": "Default welcome text"
        }
      ],
      "myServer": {
        "Backup": false,
        "Restore": false,
        "Upgrade": false,
        "ErrorLog": false,
        "Console": false,
        "Trace": false
      },
      "passwordRequirements": {
        "min": 8,
        "max": 128,
        "upper": 1,
        "lower": 1,
        "numeric": 1,
        "nonalpha": 1,
        "reset": 3650,
        "force2factor": false,
        "skip2factor": "127.0.0.1",
        "oldPasswordBan": 5,
        "banCommonPasswords": false,
        "twoFactorTimeout": 30
      },
      "twoFactorCookieDurationDays": 180,
      "agentInviteCodes": false,
      "_agentNoProxy": true,
      "geoLocation": false,
      "novnc": true,
      "mstsc": true,
      "ssh": true,
      "_WebEmailsPath": "/myserver/email-templates",
      "consentMessages": {
        "title": "MeshCentral Remote Access",
        "desktop": "{0} requesting remote desktop access. Grant access?",
        "terminal": "{0} requesting remote terminal access. Grant access?",
        "files": "{0} requesting remote files access. Grant access?",
        "consentTimeout": 60,
        "autoAcceptOnTimeout": false
      },
      "notificationMessages": {
        "title": "MeshCentral Remote Access",
        "desktop": "{0} started a remote desktop session.",
        "terminal": "{0} started a remote terminal session.",
        "files": "{0} started a remote files session."
      },
      "agentCustomization": {
        "displayName": "Tomsikr MeshCentral Agent",
        "description": "MeshCentral agent for remote monitoring, management and assistance.",
        "_companyName": "tomsikr llc",
        "serviceName": "tomsikrMeshAgent",
        "_image": "agent-logo.png",
        "_fileName": "_tomsikrMeshAgent",
        "filename": "MeshAgent"
      },
      "_agentFileInfo": {
        "_icon": "agent.ico",
        "_filedescription": "sample_filedescription",
        "fileversion": "1.0",
        "_internalname": "sample_internalname",
        "_legalcopyright": "sample_legalcopyright",
        "_originalfilename": "sample_originalfilename",
        "productname": "sample_productname",
        "productversion": "v0.1.2.3"
      },
      "assistantCustomization": {
        "title": "Tomsikr Mesh Assistant",
        "_image": "assistant-logo.png",
        "fileName": "tomsikrAssist"
      },
      "androidCustomization": {
        "title": "Tomsikr Mesh Android™",
        "_subtitle": "Product Subtitle™",
        "_image": "assistant-logo.png"
      },
      "_userAllowedIP": "127.0.0.1,192.168.1.0/24",
      "_userBlockedIP": "127.0.0.1,::1,192.168.0.100",
      "_agentAllowedIP": "192.168.0.100/24",
      "_agentBlockedIP": "127.0.0.1,::1",
      "___userSessionIdleTimeout__": "Number of user idle minutes before auto-disconnect",
      "_userSessionIdleTimeout": 60,
      "_userConsentFlags": {
        "desktopnotify": true,
        "terminalnotify": true,
        "filenotify": true,
        "desktopprompt": true,
        "terminalprompt": true,
        "fileprompt": true,
        "desktopprivacybar": true
      },
      "urlSwitching": true,
      "_desktopPrivacyBarText": "Privacy bar: {0}, {1}",
      "_limits": {
        "maxDevices": 3000,
        "maxUserAccounts": 100,
        "maxUserSessions": 100,
        "maxAgentSessions": 100,
        "maxSingleUserSessions": 10
      },
      "terminal": {
        "linuxshell": "login",
        "launchCommand": {
          "linux": "clear\necho \"Hello Linux\"\n",
          "darwin": "clear\necho \"Hello MacOS\"\n",
          "freebsd": "clear\necho \"Hello FreeBSD\"\n"
        }
      },
      "redirects": {
        "homepage": "[redacted]"
      },
      "_yubikey": {
        "id": "0000",
        "secret": "xxxxxxxxxxxxxxxxxxxxx",
        "_proxy": "http://myproxy.domain.com:80"
      },
      "_httpHeaders": {
        "Strict-Transport-Security": "max-age=360000",
        "x-frame-options": "SAMEORIGIN"
      },
      "_agentConfig": [ "webSocketMaskOverride=1", "coreDumpEnabled=1" ],
      "_assistantConfig": [ "disableUpdate=1" ],
      "_sessionRecording": {
        "_onlySelectedUsers": true,
        "_onlySelectedUserGroups": true,
        "_onlySelectedDeviceGroups": true,
        "_filepath": "C:\\temp",
        "_index": true,
        "_maxRecordings": 10,
        "_maxRecordingDays": 15,
        "_maxRecordingSizeMegabytes": 3,
        "__protocols__": "Is an array: 1 = Terminal, 2 = Desktop, 5 = Files, 100 = Intel AMT WSMAN, 101 = Intel AMT Redirection, 200 = Messenger",
        "protocols": [ 1, 2, 101 ]
      }
    }
  }
}

“Custom NGINX Configuration” in NGINX Proxy Manager

proxy_set_header CF-Connecting-IP $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 25 (6 by maintainers)

Most upvoted comments

so just tagging onto this (and other similar issues) ive just discovered this issue by mistake but i think i know the issue why its happening

i have a pi 3b+ in my office which has a touchscreen attached to it running nothing at all but node-red (im testing this for a client of meshcentral) i suddently notice it is freezing and locking up after a day for no reason? so i left open the task manager and could see my meshagent using up to 300MB+ of memory after a few hours (it started at just 13MB)

i suddenly had a brave wave, my meshcentral server isnt actually running as its my development server so i start/stop it whenever i want!

from the looks of it, the memory leak (increasing) is coming from the possible reconnection method we use in the meshagent, and the meshagent might be having problems connecting to your meshcentral server, or you are restarting your meshcentral server quite alot

im also not sure how to fix it either yet so it needs investigating more

I got same problem with high memory usage of meshagent on all platform: Linux, Windows, VPS… I haven’t heard problem from Mesh server hosting on Public IP. From the post of @krayon007 and Meshcentral documentation:

The “AgentPong” line instructs the server to send data to the agent each 300 seconds and the agent by default will send data to the server every 120 seconds. As long as NGINX timeouts are longer than this, connections should remain open.

So agent try to send data every 120 sec to server with only 60 sec (default value of NGINX) timeout of Nginx Proxy Manager (NPM), making connection still open, maybe a bug. Meshagent will create more new connections and memory leaked. My Meshcentral server put behind NPM, I tried to put this in advanced tab, greater than 120 sec:

  • proxy_read_timeout 200s;

  • proxy_connect_timeout 200s;

  • proxy_send_timeout 200s;

image Seem memory problem gone. image I haven’t tried on other reversed proxy but I think all are same problem. Hope it help.