uptime-kuma: Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?

Same error, and it happens every night at a specific time. (3.17 AM)

I am getting the same running on Kubernetes. Found this while looking in the knex https://github.com/knex/knex/issues/2820

Users are strongly encouraged to update to 1.23 before reporting related issues. You can either try the beta now or wait for an official release soon.

The server runs the task clearing monitor history data beyond the defined period at 03:14am each day (server time). 1.23 includes PR #2800 and #3380, which improves database write to disk behavior and how deletes are handled. Database operations are still blocking, but it should now takes less time to process them.

If you are still having issues, pressing the “Settings” -> “Monitor History” -> “Shrink Database” button should also help in the short term (the description previously written is not entirely accurate). Finally, disk performance is important and if your server has poor IO performance and/or you are running a large number of monitors, the chance of this error occurring will increase.

Same error, and it happens every night at a specific time. (3.17 AM)

Same… not sure what uptime kuma does at that time, but multiple monitors go offline with this error at 3:14 for me then come back online like 4min later. Maybe it’s some DB cleanup process that hammers the DB and causes it I suppose.

From a very brief skim of Google some people are pointing towards sqlite being a limitation. @louislam support for MySQL still not likely to be considered?

I know it has been mentioned before that Kuma isn’t a production ready monitoring tool but in reality it’s not far off. Bar the above issues we have found it very useful.

It’s in my 2.0 roadmap. https://github.com/users/louislam/projects/4

What fixed this for me was Settings -> Monitor History -> Clear all Statistics. Then change Keep monitor history for 7 days.

This is likely not a cpu power issue but an issue of having too much data in sqlite which takes longer (and ultimately times out) to run queries with so much data. I believe the old default was 0 for keep monitor history (forever) which that default should be changed to something like 7 or 14. I probably had a years worth of data which is also pretty useless but since I cleared everything I haven’t had any issues.

I have this error from time to time, causing a burst of downtime notifications that are quickly resolved. It would be nice to get rid of those false-positives.

I had the same issue. As soon as I am trying to delete a specific monitor (which as a lot of events associated to it), I get:

Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:312:26)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processTimers (node:internal/timers:499:9)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:287:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.find (/app/node_modules/redbean-node/dist/redbean-node.js:464:24)
    at async Timeout.generateMaintenanceTimeslots [as _onTimeout] (/app/server/uptime-kuma-server.js:275:20) {
  sql: undefined,
  bindings: undefined
}
    at process.<anonymous> (/app/server/server.js:1794:13)
    at process.emit (node:events:513:28)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
    at runNextTicks (node:internal/process/task_queues:65:3)
    at processTimers (node:internal/timers:499:9)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
2023-02-14T10:51:52+01:00 [MONITOR] WARN: Monitor #1 'mescal-anr': Pending: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call? | Max retries: 3 | Retry: 1 | Retry Interval: 60 seconds | Type: http
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:312:26)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:287:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.normalizeRaw (/app/node_modules/redbean-node/dist/redbean-node.js:569:22)
    at async RedBeanNode.getRow (/app/node_modules/redbean-node/dist/redbean-node.js:555:22)
    at async RedBeanNode.getCell (/app/node_modules/redbean-node/dist/redbean-node.js:590:19)
    at async Function.get (/app/server/settings.js:54:21)
    at async exports.setting (/app/server/util-server.js:416:12)
    at async /app/server/server.js:186:13 {
  sql: 'SELECT `value` FROM setting WHERE `key` = ?  limit ?',
  bindings: [ 'trustProxy', 1 ]
}
    at process.<anonymous> (/app/server/server.js:1794:13)
    at process.emit (node:events:513:28)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues

Then, the DB is corrupted and I have to (force) stop the container and restore an old DB to get uptime kuma working. Other monitor deletion worked.

I think it is a timeout somewhere related to a big SQL query. I have no issue related to performances (it is a big VM).

docker version:

docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64
 Experimental: false

Uptime kuma is the last version.

Thanks,

EDIT : after stopping the container, and waiting for it to be stopped (…very long time), removing it and restarting it back (long period to wait before it becomes healthy and is available again), it worked.

I got it one or two times. At least not daily. Op 10 sep 2023 om 10:14 heeft Uthpal P @.***> het volgende geschreven: @toineenzo Did upgrading to 1.23 work? I’m still facing this error after the upgrade.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

I am having this error quite often lately, is there any resolution on the horizon? Working with docker and latest version of docker and uptime kuma

I have discovered the problem for me at least. I am running uptime inside docker on a NAS. When the disk activity was high i would get this error message.

Once i addressed the continues high disk read/write actions the messages stayed away.

Hopefully someone else can benefit from this respons.

I had the same issue. As soon as I am trying to delete a specific monitor (which as a lot of events associated to it), I get:

Happening to me as well, editing a monitor kills uptime Kuma

For ref currently seem to have ameliorated this issue by changing the connection pool settings to:

                min: 0,
                max: 10,
                reapIntervalMillis: 1000,
                createRetryIntervalMillis: 200,
                idleTimeoutMillis: 30 * 1000,
                propagateCreateError: false,
                acquireTimeoutMillis: acquireConnectionTimeout,

From a very brief skim of Google some people are pointing towards sqlite being a limitation. @louislam support for MySQL still not likely to be considered?

I can’t remember which issue it was, but there was a suggestion about splitting up the config and results into two separate databases, something that would make sense. I think for the results database, a time series one would be an appropriate choice, then we could just stick to sqlite for config

I have the same issue and its not due to underpowered machine.

$ docker version
Client:
 Version:           20.10.17-ce
 API version:       1.41
 Go version:        go1.17.13
 Git commit:        a89b84221c85
 Built:             Wed Jun 29 12:00:00 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.17-ce
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.13
  Git commit:       a89b84221c85
  Built:            Wed Jun 29 12:00:00 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-ga916309fff0f
 docker-init:
  Version:          0.1.5_catatonit
  GitCommit:        

$ inxi -F
System:    Kernel: 5.14.21-150400.22-default x86_64 bits: 64 Console: pty pts/2 Distro: openSUSE Leap 15.4
CPU:       Info: 8-Core model: AMD Ryzen 7 2700 bits: 64 type: MT MCP cache: L2: 4 MiB
Info:      Processes: 1450 Uptime: 29d 0h 34m Memory: 62.69 GiB used: 41.1 GiB (65.6%) Init: systemd runlevel: 3 Shell: Zsh
           inxi: 3.3.07

$ docker image ls | grep uptime
louislam/uptime-kuma                                                1                   930a7e08142f   8 weeks ago     350MB