river: Error during reelection

The single process that processes River jobs generate this error multiple times a day:

{
    "_sysTime": "2024-12-16T19:04:31.633026767Z",
    "_time": "2024-12-16T19:04:31Z",
    "client_id": "magic-api-worker-1_2024_12_12T19_27_07_561681",
    "err": {
        "error": "error beginning transaction: context deadline exceeded",
        "kind": "*fmt.wrapError"
    },
    "level": "error",
    "message": "Elector: Error attempting reelection",
    "process": "worker",
    "sleep_duration": 1079.326989
}

Jobs are inserted by an insert-only client on another process. The few jobs (maximum 2 per day) are still correctly handled. Postgres is not overused and is in version 15.8.0. Driver is pgx v5.7.1. Connection is quite standard (no pgbouncer). River is v0.14.2.

How could I investigate the root cause of this issue?

The code is almost a duplication of your tutorial:

workers := river.NewWorkers()
river.AddWorker(workers, myWorker)

logger := slog.New(slogzerolog.Option{Level: slog.LevelInfo, Logger: &log.Logger}.NewZerologHandler())
riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
    Logger: logger,
    Queues: map[string]river.QueueConfig{river.QueueDefault: {MaxWorkers: 10},},
    Workers: workers,
})

Thanks

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 18 (7 by maintainers)

Commits related to this issue

Most upvoted comments

Hello, it appears we had this issue because we were using the same database pool for River internal operations and our business code. We reached the max connections in our business code, which led to missing connections for River internal operations. We fixed it by using a separate database pool for River and refactoring some transactions in our code. Thank you for helping us identify this problem. 🙏

It’s definitely tough to debug this based on the available info and without a reproduction. Trying to compile a list of what we do know:

  1. As far as we can tell, the leader election query itself is always fast, or at least we have no evidence to suggest otherwise. Makes sense, it’s a tiny unlogged table that should always be in memory or quick to load, so unless there’s a stuck transaction blocking access this query should be ~immediate.
  2. The error message error beginning transaction: context deadline exceeded indicates it was the Begin which failed on this WithTxV call, i.e. the context was either already expired when the operation was started, or the presently-hardcoded 5 second deadlineTimeout was exceeded while simply attempting to execute BEGIN on the database connection.
  3. Alternatively, the timeout may have occurred during Acquire. In pgx, a conn must be acquired from the pool (already connected) before the actual operation (Begin) is run on it. The context timeout is inclusive of acquire time.
  4. When pgx encounters a context timeout on Begin, it disposes the connection which calls through to pgconn.Close. This means the conn will not be returned to the pool, so any subsequent attempts must be on a different or new conn.
  5. When the underlying puddle Pool.Acquire call is made, the context may be cancelled prior to the new conn being established. However in that case even though the Acquire will return an error, the underlying resource creation will not be cancelled, so in theory if it were merely slow to establish new conns you would still see some of them get picked up eventually. Unless of course the pool is already full, in which case no new resources would be acquired, and this operation would merely timeout without any real attempt being made to acquire a connection.
  6. When this situation occurs as outlined by @almottier, it appears to occur endlessly until a restart. This means that the sum of Acquire + Begin is taking longer than 5 seconds indefinitely under these circumstances.
  7. Fetching of jobs also stopped per @almottier, again indicating the client is unable to successfully execute queries periodically.

This all leads me toward a conclusion that your pgxpool is overloaded and/or at capacity and simply unable to consistently (or at all?) acquire a connection within 5 seconds. Potentially you have other operations that are acquiring and never releasing conns on the same pool, or are simply consuming all conns with very slow operations. This seems most likely to me given that a restart fixes it; essentially you are likely leaking pool conns or tying them up somehow.

The simplest way to validate this would be to set up a ConnConfig.Tracer that implements AcquireTracer so that you can see how long your acquire calls are taking. You can also periodically grab stats from your pool so that you can see how long acquires are taking, how many conns are free, etc. Could you please try one of those and report back what you learn?

It’s also a good moment to highlight that you don’t necessarily need River to use the same conn pool as the rest of your app, though this is probably the right thing to do most of the time. Fundamentally a running river.Client needs to be able to periodically run queries against the DB pool you give it, and if it isn’t able to something will break.

@brandur I’m not sure that any of the above would be helped by removing context timeouts as we’re probably not actually seeing a Postgres error here—in fact you’d probably see the client waiting even longer for operations that aren’t ever going to succeed.