river: Error during reelection
The single process that processes River jobs generate this error multiple times a day:
{
"_sysTime": "2024-12-16T19:04:31.633026767Z",
"_time": "2024-12-16T19:04:31Z",
"client_id": "magic-api-worker-1_2024_12_12T19_27_07_561681",
"err": {
"error": "error beginning transaction: context deadline exceeded",
"kind": "*fmt.wrapError"
},
"level": "error",
"message": "Elector: Error attempting reelection",
"process": "worker",
"sleep_duration": 1079.326989
}
Jobs are inserted by an insert-only client on another process. The few jobs (maximum 2 per day) are still correctly handled. Postgres is not overused and is in version 15.8.0. Driver is pgx v5.7.1. Connection is quite standard (no pgbouncer). River is v0.14.2.
How could I investigate the root cause of this issue?
The code is almost a duplication of your tutorial:
workers := river.NewWorkers()
river.AddWorker(workers, myWorker)
logger := slog.New(slogzerolog.Option{Level: slog.LevelInfo, Logger: &log.Logger}.NewZerologHandler())
riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
Logger: logger,
Queues: map[string]river.QueueConfig{river.QueueDefault: {MaxWorkers: 10},},
Workers: workers,
})
Thanks
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 18 (7 by maintainers)
Commits related to this issue
- Log sleep durations more definitively While looking into #695 the other day, I was reminded that the handling of `time.Duration` in things like logging is potentially very good. `time.Duration` has ... — committed to riverqueue/river by brandur 2 years ago
- Log sleep durations more definitively While looking into #695 the other day, I was reminded that the handling of `time.Duration` in things like logging is potentially very good. `time.Duration` has ... — committed to riverqueue/river by brandur 2 years ago
- Log sleep durations more definitively While looking into #695 the other day, I was reminded that the handling of `time.Duration` in things like logging is potentially not very good. `time.Duration` ... — committed to riverqueue/river by brandur 2 years ago
- Log sleep durations more definitively While looking into #695 the other day, I was reminded that the handling of `time.Duration` in things like logging is potentially not very good. `time.Duration` ... — committed to riverqueue/river by brandur 2 years ago
- Log sleep durations more definitively (#699) While looking into #695 the other day, I was reminded that the handling of `time.Duration` in things like logging is potentially not very good. `time.... — committed to riverqueue/river by brandur 2 years ago
Hello, it appears we had this issue because we were using the same database pool for River internal operations and our business code. We reached the max connections in our business code, which led to missing connections for River internal operations. We fixed it by using a separate database pool for River and refactoring some transactions in our code. Thank you for helping us identify this problem. 🙏
It’s definitely tough to debug this based on the available info and without a reproduction. Trying to compile a list of what we do know:
error beginning transaction: context deadline exceededindicates it was theBeginwhich failed on thisWithTxVcall, i.e. the context was either already expired when the operation was started, or the presently-hardcoded 5 seconddeadlineTimeoutwas exceeded while simply attempting to executeBEGINon the database connection.Acquire. In pgx, a conn must be acquired from the pool (already connected) before the actual operation (Begin) is run on it. The context timeout is inclusive of acquire time.Begin, it disposes the connection which calls through topgconn.Close. This means the conn will not be returned to the pool, so any subsequent attempts must be on a different or new conn.Pool.Acquirecall is made, the context may be cancelled prior to the new conn being established. However in that case even though theAcquirewill return an error, the underlying resource creation will not be cancelled, so in theory if it were merely slow to establish new conns you would still see some of them get picked up eventually. Unless of course the pool is already full, in which case no new resources would be acquired, and this operation would merely timeout without any real attempt being made to acquire a connection.Acquire+Beginis taking longer than 5 seconds indefinitely under these circumstances.This all leads me toward a conclusion that your pgxpool is overloaded and/or at capacity and simply unable to consistently (or at all?) acquire a connection within 5 seconds. Potentially you have other operations that are acquiring and never releasing conns on the same pool, or are simply consuming all conns with very slow operations. This seems most likely to me given that a restart fixes it; essentially you are likely leaking pool conns or tying them up somehow.
The simplest way to validate this would be to set up a
ConnConfig.Tracerthat implementsAcquireTracerso that you can see how long your acquire calls are taking. You can also periodically grab stats from your pool so that you can see how long acquires are taking, how many conns are free, etc. Could you please try one of those and report back what you learn?It’s also a good moment to highlight that you don’t necessarily need River to use the same conn pool as the rest of your app, though this is probably the right thing to do most of the time. Fundamentally a running
river.Clientneeds to be able to periodically run queries against the DB pool you give it, and if it isn’t able to something will break.@brandur I’m not sure that any of the above would be helped by removing context timeouts as we’re probably not actually seeing a Postgres error here—in fact you’d probably see the client waiting even longer for operations that aren’t ever going to succeed.