patroni: Replication connection error: could not connect to server: FATAL: password authentication failed for user "replicator"

First make sure correct pg_hba.conf is in use:

postgres@18aa43abe82c:/$ psql
psql (12.2 (Debian 12.2-2.pgdg100+1))
Type "help" for help.

postgres=# select current_setting('hba_file');
                 current_setting
-------------------------------------------------
 /var/lib/postgresql/data/postgres-3/pg_hba.conf
(1 row)

postgres=# \q
postgres@18aa43abe82c:/$ cat /var/lib/postgresql/data/postgres-3/pg_hba.conf
# Do not edit this file manually!
# It will be overwritten by Patroni!
host replication replicator 127.0.0.1/32 md5
host replication replicator 10.0.0.0/24 md5
host all all 0.0.0.0/0 md5

Then lets try to connect to current leader manually with login and password from 10.0.0.5 to 10.0.0.7:

# Running on 10.0.0.5
postgres@93873aaf2ab3:/$ psql -h 10.0.0.7 postgres -U replicator -W
Password:
psql (12.2 (Debian 12.2-2.pgdg100+1))
Type "help" for help.

postgres=>

Looks correct. However patroni fails to connect. Current leader logs are full of:

2020-04-13 14:01:27,905 INFO: Lock owner: node-postgres-3; I am node-postgres-3
2020-04-13 14:01:27,917 INFO: no action.  i am the leader with the lock
2020-04-13 14:01:30 UTC [691]: [1-1] 5e9470ba.2b3 0 [unknown] [unknown] [unknown] 10.0.0.7 LOG:  connection received: host=10.0.0.7 port=34718
2020-04-13 14:01:31 UTC [692]: [1-1] 5e9470bb.2b4 0 [unknown] [unknown] [unknown] 10.0.0.5 LOG:  connection received: host=10.0.0.5 port=48302
2020-04-13 14:01:31 UTC [692]: [2-1] 5e9470bb.2b4 0 [unknown] replicator [unknown] 10.0.0.5 FATAL:  password authentication failed for user "replicator"
2020-04-13 14:01:31 UTC [692]: [3-1] 5e9470bb.2b4 0 [unknown] replicator [unknown] 10.0.0.5 DETAIL:  Password does not match for user "replicator".
 Connection matched pg_hba.conf line 5: "host  replication replicator 10.0.0.0/24    md5”

Looks like a bug?

patroni.yml config:

---
scope: postgres
name: node-postgres-3
namespace: /patroni

consul:
  host: 10.0.0.7
  port: 8500
  token: <my-token-here>
  register_service: true

restapi:
  listen: ':8008'
  connect_address: 10.0.0.7:8008

bootstrap:
  method: initdb
  intidb:
    - data-checksums
    - encoding: UTF8
    - locale: en_US.UTF-8
  walg:
    command: wal-g backup-fetch /var/lib/postgresql/data/postgres-3 LATEST
    keep_existing_recovery_conf: false
    recovery_conf:
      recovery_target_action: promote
      recovery_target_timeline: latest
      recovery_target_inclusive: true
      restore_command: wal-g wal-fetch "%f" "%p"
  pg_hba:
    - host replication replicator 127.0.0.1/32 md5
    - host replication replicator 10.0.0.0/24 md5
    - host all all 0.0.0.0/0 md5
  dcs:
    loop_wait: 10
    ttl: 30
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    master_start_timeout: 300
    synchronous_mode: false
    synchronous_mode_strict: false
    postgresql:
      use_pg_rewind: true
      use_slots: true
      parameters:
        archive_mode: "on"
        archive_timeout: 300s
        archive_command: wal-g wal-push "%p"
        wal_level: replica
        wal_keep_segments: 8
        wal_log_hints: "on"
        max_wal_senders: 10
        max_replication_slots: 10
        hot_standby: "on"

postgresql:
  authentication:
    superuser:
      username: postgres
      password: mypassword
    replication:
      username: replicator
      password: mypassword
    rewind:
      username: rewind
      password: mypassword
  listen: '*:5432'
  connect_address: 10.0.0.7:5432
  data_dir: /var/lib/postgresql/data/postgres-3
  pgpass: /var/run/postgresql/.pgpass
  #recovery_conf:
  #  restore_command: wal-g wal-fetch "%f" "%p"
  #  recovery_target_timeline: latest
  #  recovery_target_action: promote
  #  recovery_target_inclusive: true
  pg_hba:
    - host replication replicator 127.0.0.1/32 md5
    - host replication replicator 10.0.0.0/24 md5
    - host all all 0.0.0.0/0 md5
  parameters:
    shared_preload_libraries: 'pg_stat_statements'
    tcp_keepalives_idle: 900
    tcp_keepalives_interval: 100
    log_destination: "stderr"
    log_line_prefix: '%t [%p]: [%l-1] %c %x %d %u %a %h '
    log_checkpoints: 'on'
    log_lock_waits: 'on'
    log_min_duration_statement: 500
    log_autovacuum_min_duration: 0
    log_connections: 'on'
    log_disconnections: 'on'
    log_statement: 'ddl'
    log_temp_files: 0
    track_functions: all
  create_replica_methods:
    #- wal_g
    - basebackup
  wal_g:
    command: wal-g backup-fetch /var/lib/postgresql/data/postgres-3 LATEST
    no_master: 1
    no_params: true
  basebackup:
    max-rate: "100M"
    checkpoint: fast

watchdog:
  mode: off

tags:
  nofailover: false
  noloadbalance: false
  clonefrom: false
  nosync: false

Patroni: patronictl version 1.6.4 PostgreSQL: postgres (PostgreSQL) 12.2 (Debian 12.2-2.pgdg100+1)

Similar issue: #951