aedes: [Bug] Aedes stops accepting subscribe/ publish

Hello

I have migrated from mosca few days back . I have a test server with ~10 clients connected and another production server with ~400 clients connected. The smaller instance works great but the bigger one stops responding to subscribe and publish requests after few days . Not sure what could be the cause. I checked the CPU/ memory usage. Its not high.

Aedes Code https://pastebin.com/raw/Zbf58pR1

Logs - I only see clients being connected but unable to publish / subscribe

authorize :- device-5e3c5054ca15e200c69dd8c6
client connected Tin-fb49
authorize :- device-5e3c5054ca15e200c69dd8c6
client connected Tin-fb49
authorize :- device-5e3csdfsdfsdfsdfsdfsdfsdfsdf
client connected Tin-fb49
authorize :- device-5e3c5054ca15esdfsdfsdfsdfds
client connected Tin-rewr

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 90 (38 by maintainers)

Commits related to this issue

Most upvoted comments

@robertsLando yeah , problem resolved. I will keep you posted if the problem reappears.

Thanks. Not sure this help. But i usually disconnect / close client on client error

@bhirave It’s useless to call client.close on clientError as it’s already called immediatly after that event is emitted: https://github.com/moscajs/aedes/blob/master/lib/client.js#L192

@robertsLando I would need at least a week to confirm if the problem is resolved. It happens spordiacly

Thanks @robertsLando . I am really looking forward to hear solution / fix for original thread @mogupta started. so i could take care of it before we perform migration from mosca to aedes. Let me run some benchmarking tools, see what happens with it.

@mogupta I suggest you to pass to aedes the db connection instead of the url and wait for persistence ready before starting the server, that could happen when aedes receives lot messages and the persistence is not ready or/and you could use mongoPersistence.setMaxListeners to increese max listeners

@bhirave I will try those tomorrow morning IST (when my users are sleeping) . The broker doesn’t seem to be flooded. No matter how I try and how many connections are alive, the broker stops accepting publish messages. no error is returned on the client. I tried with Mqttfx when the broker was misbehaving.

@mogupta i would recommend you to add following subscriptions, so you can get more logging, why subscription being stopped

aedes.on('connectionError', function (client, err) {
    console.log('client connection Error:', client.id, "==>", err.message);
    // disconnect client
    client.close();
});


aedes.on('clientError', function (client, err) {
    console.log('client error:', client.id, "==>", err.message);
    // disconnect client
    client.close();
});

That’s interesting! I would like to know if @mogupta is using clientsId with 23 or more charaters. There must be a memory leak in connect handler when return code is 2. I will dig into this tomorrow


Daniel

On 11 Jun 2020, at 21:19, Bapu Hirave notifications@github.com wrote:

@mogupta @robertsLando i also found that, if you dont set maxClientsIdLength, default value is 23 (?). So i am running benchmark tool right now, my client was generating random id (and length was more than 23).

three observations

if client id length is more than 23, and auth/authz hooks are async/await, with 150 clients, broker stopped responding without any error

if client id length is more than 23, and NO AUTH HOOKS, with 150 clients, broker stopped responding but throws error - identifier rejected

if client id is specified (lets say 50), and auth/authz hooks are still async/await, broker does continuously operate

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Try with something like

// In this case the client authorized as alice can publish to /users/alice taking
// the username from the topic and verifing it is the same of the authorized user
var authorizePublish = function(client, packet, callback) {
  var error = null
  validateRequest(client, packet).then(err => {
    error = err
    if (!err)   deviceToggled(client, packet.topic, packet.payload);
  }).catch(err => {
    error = err
  }).finally(()=> callback(error))
}
 
// In this case the client authorized as alice can subscribe to /users/alice taking
// the username from the topic and verifing it is the same of the authorized user
var authorizeSubscribe = async function(client, sub, callback) {
  var error = null
  validateRequest(client, packet).then(err => {
    error = err
  }).catch(err => {
    error = err
  }).finally(() => callback(error, sub))
}

In this way you are sure callback is always called

I started the broker directly using memory, and reached a certain amount of connections, but also appeared that I could not publish or subscribe, but only connect.