cloudnative-pg: Barman failed ot run wal-archive command - storing backups in minIO
I am trying to set up a cloudnative pg with s3 backup on minIO. Unfortunately fails the barman backup while running wal-archive command.
{"level":"info","ts":"2023-08-23T07:51:39Z","logger":"postgres","msg":"record","logging_pod":"postgres-db-1","record":{"log_time":"2023-08-23 07:51:39.617 UTC","process_id":"29","session_id":"64e51ab8.1d","session_line_num":"2603","session_start_time":"2023-08-22 20:29:44 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"archive command failed with exit code 1","detail":"The failed archive command was: /controller/manager wal-archive --log-destination /controller/log/postgres.json pg_wal/000000010000000000000001","backend_type":"archiver","query_id":"0"}}
{"level":"info","ts":"2023-08-23T07:51:39Z","logger":"postgres","msg":"record","logging_pod":"postgres-db-1","record":{"log_time":"2023-08-23 07:51:39.617 UTC","process_id":"29","session_id":"64e51ab8.1d","session_line_num":"2604","session_start_time":"2023-08-22 20:29:44 UTC","transaction_id":"0","error_severity":"WARNING","sql_state_code":"01000","message":"archiving write-ahead log file \"000000010000000000000001\" failed too many times, will try again later","backend_type":"archiver","query_id":"0"}}
This is what I get from the pod logs and the backups.postgresql.cnpg.io results in PHASE: walArchivingFailing.
This is my cloudnativePG cluster:
kind: Secret
metadata:
name: aws-creds
namespace: postgres-db
labels:
cnpg.io/reload: "true"
type: Opaque
stringData:
MINIO_ACCESS_KEY: <secret>
MINIO_SECRET_KEY: <secret>
---
apiVersion: v1
data:
password: <secret>
username: <secret>
kind: Secret
metadata:
namespace: postgres-db
name: postgres-root
type: kubernetes.io/basic-auth
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
namespace: postgres-db
name: postgres-db
spec:
instances: 3
backup:
barmanObjectStore:
destinationPath: s3://backup/
endpointURL: https://my.minio.url
s3Credentials:
accessKeyId:
name: aws-creds
key: MINIO_ACCESS_KEY
secretAccessKey:
name: aws-creds
key: MINIO_SECRET_KEY
wal:
compression: gzip
# Parameters and pg_hba configuration will be append
# to the default ones to make the cluster work
postgresql:
parameters:
max_worker_processes: "60"
pg_hba:
# To access through TCP/IP you will need to get username
# and password from the secret cluster-example-custom-app
- host all all all md5
superuserSecret:
name: postgres-root
# Example of rolling update strategy:
# - unsupervised: automated update of the primary once all
# replicas have been upgraded (default)
# - supervised: requires manual supervision to perform
# the switchover of the primary
primaryUpdateStrategy: unsupervised
# Require 1Gi of space per instance using default storage class
storage:
size: 1Gi
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
namespace: postgres-db
name: pg-backup
spec:
immediate: true # Backup starts immediately after ScheduledBackup has been created
schedule: "0 0 * * * *"
cluster:
name: postgres-db
this is the operator values.yaml
# Copyright The CloudNativePG Contributors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Default values for CloudNativePG.
# This is a YAML-formatted file.
# Please declare variables to be passed to your templates.
replicaCount: 1
image:
repository: ghcr.io/cloudnative-pg/cloudnative-pg
pullPolicy: IfNotPresent
# -- Overrides the image tag whose default is the chart appVersion.
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
crds:
# -- Specifies whether the CRDs should be created when installing the chart.
create: true
# -- The webhook configuration.
webhook:
port: 9443
mutating:
create: true
failurePolicy: Fail
validating:
create: true
failurePolicy: Fail
livenessProbe:
initialDelaySeconds: 3
readinessProbe:
initialDelaySeconds: 3
# -- Operator configuration.
config:
# -- Specifies whether the secret should be created.
create: true
# -- The name of the configmap/secret to use.
name: cnpg-controller-manager-config
# -- Specifies whether it should be stored in a secret, instead of a configmap.
secret: false
# -- The content of the configmap/secret, see
# https://cloudnative-pg.io/documentation/current/operator_conf/#available-options
# for all the available options.
data: {}
# INHERITED_ANNOTATIONS: categories
# INHERITED_LABELS: environment, workload, app
# WATCH_NAMESPACE: namespace-a,namespace-b
WATCH_NAMESPACE: postgres-db
# -- Additinal arguments to be added to the operator's args list.
additionalArgs: []
serviceAccount:
# -- Specifies whether the service account should be created.
create: true
# -- The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template.
name: ""
rbac:
# -- Specifies whether ClusterRole and ClusterRoleBinding should be created.
create: true
# -- Annotations to be added to all other resources.
commonAnnotations: {}
# -- Annotations to be added to the pod.
podAnnotations: {}
# -- Labels to be added to the pod.
podLabels: {}
# -- Container Security Context.
containerSecurityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1000
runAsGroup: 1000
capabilities:
drop:
- "ALL"
# -- Security Context for the whole pod.
podSecurityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
# fsGroup: 2000
# -- Priority indicates the importance of a Pod relative to other Pods.
priorityClassName: ""
service:
type: ClusterIP
# -- DO NOT CHANGE THE SERVICE NAME as it is currently used to generate the certificate
# and can not be configured
name: cnpg-webhook-service
port: 443
resources: {}
# If you want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
#
# limits:
# cpu: 100m
# memory: 200Mi
# requests:
# cpu: 100m
# memory: 100Mi
# -- Nodeselector for the operator to be installed.
nodeSelector: {}
# -- Tolerations for the operator to be installed.
tolerations: []
# -- Affinity for the operator to be installed.
affinity: {}
monitoring:
# -- Specifies whether the monitoring should be enabled. Requires Prometheus Operator CRDs.
podMonitorEnabled: false
# Default monitoring queries
monitoringQueriesConfigMap:
# -- The name of the default monitoring configmap.
name: cnpg-default-monitoring
# -- A string representation of a YAML defining monitoring queries.
queries: |
backends:
query: |
SELECT sa.datname
, sa.usename
, sa.application_name
, states.state
, COALESCE(sa.count, 0) AS total
, COALESCE(sa.max_tx_secs, 0) AS max_tx_duration_seconds
FROM ( VALUES ('active')
, ('idle')
, ('idle in transaction')
, ('idle in transaction (aborted)')
, ('fastpath function call')
, ('disabled')
) AS states(state)
LEFT JOIN (
SELECT datname
, state
, usename
, COALESCE(application_name, '') AS application_name
, COUNT(*)
, COALESCE(EXTRACT (EPOCH FROM (max(now() - xact_start))), 0) AS max_tx_secs
FROM pg_catalog.pg_stat_activity
GROUP BY datname, state, usename, application_name
) sa ON states.state = sa.state
WHERE sa.usename IS NOT NULL
metrics:
- datname:
usage: "LABEL"
description: "Name of the database"
- usename:
usage: "LABEL"
description: "Name of the user"
- application_name:
usage: "LABEL"
description: "Name of the application"
- state:
usage: "LABEL"
description: "State of the backend"
- total:
usage: "GAUGE"
description: "Number of backends"
- max_tx_duration_seconds:
usage: "GAUGE"
description: "Maximum duration of a transaction in seconds"
backends_waiting:
query: |
SELECT count(*) AS total
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_locks blocking_locks
ON blocking_locks.locktype = blocked_locks.locktype
AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
AND blocking_locks.pid != blocked_locks.pid
JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted
metrics:
- total:
usage: "GAUGE"
description: "Total number of backends that are currently waiting on other queries"
pg_database:
query: |
SELECT datname
, pg_catalog.pg_database_size(datname) AS size_bytes
, pg_catalog.age(datfrozenxid) AS xid_age
, pg_catalog.mxid_age(datminmxid) AS mxid_age
FROM pg_catalog.pg_database
metrics:
- datname:
usage: "LABEL"
description: "Name of the database"
- size_bytes:
usage: "GAUGE"
description: "Disk space used by the database"
- xid_age:
usage: "GAUGE"
description: "Number of transactions from the frozen XID to the current one"
- mxid_age:
usage: "GAUGE"
description: "Number of multiple transactions (Multixact) from the frozen XID to the current one"
pg_postmaster:
query: |
SELECT EXTRACT(EPOCH FROM pg_postmaster_start_time) AS start_time
FROM pg_catalog.pg_postmaster_start_time()
metrics:
- start_time:
usage: "GAUGE"
description: "Time at which postgres started (based on epoch)"
pg_replication:
query: "SELECT CASE WHEN NOT pg_catalog.pg_is_in_recovery()
THEN 0
ELSE GREATEST (0,
EXTRACT(EPOCH FROM (now() - pg_catalog.pg_last_xact_replay_timestamp())))
END AS lag,
pg_catalog.pg_is_in_recovery() AS in_recovery,
EXISTS (TABLE pg_stat_wal_receiver) AS is_wal_receiver_up,
(SELECT count(*) FROM pg_stat_replication) AS streaming_replicas"
metrics:
- lag:
usage: "GAUGE"
description: "Replication lag behind primary in seconds"
- in_recovery:
usage: "GAUGE"
description: "Whether the instance is in recovery"
- is_wal_receiver_up:
usage: "GAUGE"
description: "Whether the instance wal_receiver is up"
- streaming_replicas:
usage: "GAUGE"
description: "Number of streaming replicas connected to the instance"
pg_replication_slots:
query: |
SELECT slot_name,
slot_type,
database,
active,
pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), restart_lsn)
FROM pg_catalog.pg_replication_slots
WHERE NOT temporary
metrics:
- slot_name:
usage: "LABEL"
description: "Name of the replication slot"
- slot_type:
usage: "LABEL"
description: "Type of the replication slot"
- database:
usage: "LABEL"
description: "Name of the database"
- active:
usage: "GAUGE"
description: "Flag indicating whether the slot is active"
- pg_wal_lsn_diff:
usage: "GAUGE"
description: "Replication lag in bytes"
pg_stat_archiver:
query: |
SELECT archived_count
, failed_count
, COALESCE(EXTRACT(EPOCH FROM (now() - last_archived_time)), -1) AS seconds_since_last_archival
, COALESCE(EXTRACT(EPOCH FROM (now() - last_failed_time)), -1) AS seconds_since_last_failure
, COALESCE(EXTRACT(EPOCH FROM last_archived_time), -1) AS last_archived_time
, COALESCE(EXTRACT(EPOCH FROM last_failed_time), -1) AS last_failed_time
, COALESCE(CAST(CAST('x'||pg_catalog.right(pg_catalog.split_part(last_archived_wal, '.', 1), 16) AS pg_catalog.bit(64)) AS pg_catalog.int8), -1) AS last_archived_wal_start_lsn
, COALESCE(CAST(CAST('x'||pg_catalog.right(pg_catalog.split_part(last_failed_wal, '.', 1), 16) AS pg_catalog.bit(64)) AS pg_catalog.int8), -1) AS last_failed_wal_start_lsn
, EXTRACT(EPOCH FROM stats_reset) AS stats_reset_time
FROM pg_catalog.pg_stat_archiver
metrics:
- archived_count:
usage: "COUNTER"
description: "Number of WAL files that have been successfully archived"
- failed_count:
usage: "COUNTER"
description: "Number of failed attempts for archiving WAL files"
- seconds_since_last_archival:
usage: "GAUGE"
description: "Seconds since the last successful archival operation"
- seconds_since_last_failure:
usage: "GAUGE"
description: "Seconds since the last failed archival operation"
- last_archived_time:
usage: "GAUGE"
description: "Epoch of the last time WAL archiving succeeded"
- last_failed_time:
usage: "GAUGE"
description: "Epoch of the last time WAL archiving failed"
- last_archived_wal_start_lsn:
usage: "GAUGE"
description: "Archived WAL start LSN"
- last_failed_wal_start_lsn:
usage: "GAUGE"
description: "Last failed WAL LSN"
- stats_reset_time:
usage: "GAUGE"
description: "Time at which these statistics were last reset"
pg_stat_bgwriter:
query: |
SELECT checkpoints_timed
, checkpoints_req
, checkpoint_write_time
, checkpoint_sync_time
, buffers_checkpoint
, buffers_clean
, maxwritten_clean
, buffers_backend
, buffers_backend_fsync
, buffers_alloc
FROM pg_catalog.pg_stat_bgwriter
metrics:
- checkpoints_timed:
usage: "COUNTER"
description: "Number of scheduled checkpoints that have been performed"
- checkpoints_req:
usage: "COUNTER"
description: "Number of requested checkpoints that have been performed"
- checkpoint_write_time:
usage: "COUNTER"
description: "Total amount of time that has been spent in the portion of checkpoint processing where files are written to disk, in milliseconds"
- checkpoint_sync_time:
usage: "COUNTER"
description: "Total amount of time that has been spent in the portion of checkpoint processing where files are synchronized to disk, in milliseconds"
- buffers_checkpoint:
usage: "COUNTER"
description: "Number of buffers written during checkpoints"
- buffers_clean:
usage: "COUNTER"
description: "Number of buffers written by the background writer"
- maxwritten_clean:
usage: "COUNTER"
description: "Number of times the background writer stopped a cleaning scan because it had written too many buffers"
- buffers_backend:
usage: "COUNTER"
description: "Number of buffers written directly by a backend"
- buffers_backend_fsync:
usage: "COUNTER"
description: "Number of times a backend had to execute its own fsync call (normally the background writer handles those even when the backend does its own write)"
- buffers_alloc:
usage: "COUNTER"
description: "Number of buffers allocated"
pg_stat_database:
query: |
SELECT datname
, xact_commit
, xact_rollback
, blks_read
, blks_hit
, tup_returned
, tup_fetched
, tup_inserted
, tup_updated
, tup_deleted
, conflicts
, temp_files
, temp_bytes
, deadlocks
, blk_read_time
, blk_write_time
FROM pg_catalog.pg_stat_database
metrics:
- datname:
usage: "LABEL"
description: "Name of this database"
- xact_commit:
usage: "COUNTER"
description: "Number of transactions in this database that have been committed"
- xact_rollback:
usage: "COUNTER"
description: "Number of transactions in this database that have been rolled back"
- blks_read:
usage: "COUNTER"
description: "Number of disk blocks read in this database"
- blks_hit:
usage: "COUNTER"
description: "Number of times disk blocks were found already in the buffer cache, so that a read was not necessary (this only includes hits in the PostgreSQL buffer cache, not the operating system's file system cache)"
- tup_returned:
usage: "COUNTER"
description: "Number of rows returned by queries in this database"
- tup_fetched:
usage: "COUNTER"
description: "Number of rows fetched by queries in this database"
- tup_inserted:
usage: "COUNTER"
description: "Number of rows inserted by queries in this database"
- tup_updated:
usage: "COUNTER"
description: "Number of rows updated by queries in this database"
- tup_deleted:
usage: "COUNTER"
description: "Number of rows deleted by queries in this database"
- conflicts:
usage: "COUNTER"
description: "Number of queries canceled due to conflicts with recovery in this database"
- temp_files:
usage: "COUNTER"
description: "Number of temporary files created by queries in this database"
- temp_bytes:
usage: "COUNTER"
description: "Total amount of data written to temporary files by queries in this database"
- deadlocks:
usage: "COUNTER"
description: "Number of deadlocks detected in this database"
- blk_read_time:
usage: "COUNTER"
description: "Time spent reading data file blocks by backends in this database, in milliseconds"
- blk_write_time:
usage: "COUNTER"
description: "Time spent writing data file blocks by backends in this database, in milliseconds"
pg_stat_replication:
primary: true
query: |
SELECT usename
, COALESCE(application_name, '') AS application_name
, COALESCE(client_addr::text, '') AS client_addr
, EXTRACT(EPOCH FROM backend_start) AS backend_start
, COALESCE(pg_catalog.age(backend_xmin), 0) AS backend_xmin_age
, pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), sent_lsn) AS sent_diff_bytes
, pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), write_lsn) AS write_diff_bytes
, pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), flush_lsn) AS flush_diff_bytes
, COALESCE(pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), replay_lsn),0) AS replay_diff_bytes
, COALESCE((EXTRACT(EPOCH FROM write_lag)),0)::float AS write_lag_seconds
, COALESCE((EXTRACT(EPOCH FROM flush_lag)),0)::float AS flush_lag_seconds
, COALESCE((EXTRACT(EPOCH FROM replay_lag)),0)::float AS replay_lag_seconds
FROM pg_catalog.pg_stat_replication
metrics:
- usename:
usage: "LABEL"
description: "Name of the replication user"
- application_name:
usage: "LABEL"
description: "Name of the application"
- client_addr:
usage: "LABEL"
description: "Client IP address"
- backend_start:
usage: "COUNTER"
description: "Time when this process was started"
- backend_xmin_age:
usage: "COUNTER"
description: "The age of this standby's xmin horizon"
- sent_diff_bytes:
usage: "GAUGE"
description: "Difference in bytes from the last write-ahead log location sent on this connection"
- write_diff_bytes:
usage: "GAUGE"
description: "Difference in bytes from the last write-ahead log location written to disk by this standby server"
- flush_diff_bytes:
usage: "GAUGE"
description: "Difference in bytes from the last write-ahead log location flushed to disk by this standby server"
- replay_diff_bytes:
usage: "GAUGE"
description: "Difference in bytes from the last write-ahead log location replayed into the database on this standby server"
- write_lag_seconds:
usage: "GAUGE"
description: "Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written it"
- flush_lag_seconds:
usage: "GAUGE"
description: "Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written and flushed it"
- replay_lag_seconds:
usage: "GAUGE"
description: "Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written, flushed and applied it"
pg_settings:
query: |
SELECT name,
CASE setting WHEN 'on' THEN '1' WHEN 'off' THEN '0' ELSE setting END AS setting
FROM pg_catalog.pg_settings
WHERE vartype IN ('integer', 'real', 'bool')
ORDER BY 1
metrics:
- name:
usage: "LABEL"
description: "Name of the setting"
- setting:
usage: "GAUGE"
description: "Setting value"
About this issue
- Original URL
- State: open
- Created 10 months ago
- Comments: 23 (12 by maintainers)
OK, so the check generally succeeded. Right now it looks like in both cases there should be exactly that same barman command run, but the outcome is different. Strange … The only thing I can think of right now is some problem with " ", so strings are read differently somehow (try mess around with different configuration strings definition …) You could also try one more thing - copy ps command to the container (it requires also some additional libs), or prepare an image containing the ps command and try to catch what is exactly being run by the manager. There has to be some difference. There is no way two identical commands with two identical sets of parameters produce a different outcome.
Controller manager, as U can see, first runs barman-cloud-check-wal-archive command, and it gets quite straight forward response message: “An error occurred (403) when calling the HeadBucket operation: Forbidden”. Looks like a permission problem running HeadBucket command. Try to manually test Your bucket against HeadBucket command (using any S3 tool). If You make it succeed, backups should also start running.
As long as this topic is still open I would use the opportunity to get your support. I still wasn’t able to write into my minio via the manager. Not quite sure why. I think, I’ve checked everything twice and I can’t explain to me, what I am doing wrong. As long using the barman from the cli works fine, so the automatic backup is just failing with not so clear error.
I’ve checked the asnwer from @MichaluxPL on Sep 1 and I’ll paste now exactly what I did without masking it:
This is how I use the barman Cli command:
export AWS_SECRET_ACCESS_KEY=minio123 export AWS_ACCESS_KEY_ID=minio barman-cloud-wal-archive -vv --cloud-provider aws-s3 --endpoint-url http://minio.minio.svc.cluster.local s3://test postgres-db /var/lib/postgresql/data/pgdata/pg_wal/000000010000000000000001And this is the log:2023-11-12 11:18:19,142 [27954] DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2023-11-12 11:18:19,144 [27954] DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway 2023-11-12 11:18:19,144 [27954] DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2023-11-12 11:18:19,146 [27954] DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2023-11-12 11:18:19,146 [27954] DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2023-11-12 11:18:19,146 [27954] DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2023-11-12 11:18:19,147 [27954] DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section 2023-11-12 11:18:19,149 [27954] DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2023-11-12 11:18:19,149 [27954] DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section 2023-11-12 11:18:19,149 [27954] DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2023-11-12 11:18:19,149 [27954] DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section 2023-11-12 11:18:19,161 [27954] DEBUG: Loading JSON file: /usr/local/lib/python3.9/dist-packages/boto3/data/s3/2006-03-01/resources-1.json 2023-11-12 11:18:19,164 [27954] DEBUG: IMDS ENDPOINT: http://169.254.169.254/ 2023-11-12 11:18:19,166 [27954] DEBUG: Looking for credentials via: env 2023-11-12 11:18:19,166 [27954] INFO: Found credentials in environment variables. 2023-11-12 11:18:19,167 [27954] DEBUG: Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/endpoints.json 2023-11-12 11:18:19,192 [27954] DEBUG: Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/sdk-default-configuration.json 2023-11-12 11:18:19,192 [27954] DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fcd22aff310> 2023-11-12 11:18:19,204 [27954] DEBUG: Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/s3/2006-03-01/service-2.json 2023-11-12 11:18:19,224 [27954] DEBUG: Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz 2023-11-12 11:18:19,227 [27954] DEBUG: Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/partitions.json 2023-11-12 11:18:19,230 [27954] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fcd22bf1160> 2023-11-12 11:18:19,230 [27954] DEBUG: Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7fcd234acdc0> 2023-11-12 11:18:19,242 [27954] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fcd22bebee0> 2023-11-12 11:18:19,243 [27954] DEBUG: Setting s3 timeout as (60, 60) 2023-11-12 11:18:19,245 [27954] DEBUG: Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/_retry.json 2023-11-12 11:18:19,246 [27954] DEBUG: Registering retry handlers for service: s3 2023-11-12 11:18:19,246 [27954] DEBUG: Registering S3 region redirector handler 2023-11-12 11:18:19,246 [27954] DEBUG: Loading s3:s3 2023-11-12 11:18:19,248 [27954] DEBUG: Event before-parameter-build.s3.HeadBucket: calling handler <function validate_bucket_name at 0x7fcd22b16e50> 2023-11-12 11:18:19,248 [27954] DEBUG: Event before-parameter-build.s3.HeadBucket: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fcd22b1dca0> 2023-11-12 11:18:19,248 [27954] DEBUG: Event before-parameter-build.s3.HeadBucket: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,248 [27954] DEBUG: Event before-parameter-build.s3.HeadBucket: calling handler <function generate_idempotent_uuid at 0x7fcd22b16ca0> 2023-11-12 11:18:19,248 [27954] DEBUG: Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fcd22b1de50> 2023-11-12 11:18:19,248 [27954] DEBUG: Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,248 [27954] DEBUG: Calling endpoint provider with parameters: {'Bucket': 'test', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://minio.minio.svc.cluster.local', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} 2023-11-12 11:18:19,249 [27954] DEBUG: Endpoint provider result: http://minio.minio.svc.cluster.local/test 2023-11-12 11:18:19,249 [27954] DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2023-11-12 11:18:19,249 [27954] DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} 2023-11-12 11:18:19,250 [27954] DEBUG: Event before-call.s3.HeadBucket: calling handler <function add_expect_header at 0x7fcd22b1a1f0> 2023-11-12 11:18:19,250 [27954] DEBUG: Event before-call.s3.HeadBucket: calling handler <function add_recursion_detection_header at 0x7fcd22b16940> 2023-11-12 11:18:19,250 [27954] DEBUG: Event before-call.s3.HeadBucket: calling handler <function inject_api_version_header_if_needed at 0x7fcd22b1d550> 2023-11-12 11:18:19,250 [27954] DEBUG: Making request for OperationModel(name=HeadBucket) with params: {'url_path': '', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.28.24 md/Botocore#1.31.24 ua/2.0 os/linux#6.1.46-0-lts md/arch#x86_64 lang/python#3.9.2 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.24 Resource'}, 'body': b'', 'auth_path': '/test/', 'url': 'http://minio.minio.svc.cluster.local/test', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fcd22220160>, 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'test', 'params': {'Bucket': 'test'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} 2023-11-12 11:18:19,250 [27954] DEBUG: Event request-created.s3.HeadBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fcd22219d30>> 2023-11-12 11:18:19,250 [27954] DEBUG: Event choose-signer.s3.HeadBucket: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fcd22a1cdc0>> 2023-11-12 11:18:19,250 [27954] DEBUG: Event choose-signer.s3.HeadBucket: calling handler <function set_operation_specific_signer at 0x7fcd22b16b80> 2023-11-12 11:18:19,250 [27954] DEBUG: Event before-sign.s3.HeadBucket: calling handler <function remove_arn_from_signing_path at 0x7fcd22b1ddc0> 2023-11-12 11:18:19,251 [27954] DEBUG: Calculating signature using v4 auth. 2023-11-12 11:18:19,251 [27954] DEBUG: CanonicalRequest: HEAD /test host:minio.minio.svc.cluster.local x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20231112T111819Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-11-12 11:18:19,251 [27954] DEBUG: StringToSign: AWS4-HMAC-SHA256 20231112T111819Z 20231112/us-east-1/s3/aws4_request d08b0b57a4ed2b5f3560d54d9e90bfcc9b039075143224b72468b73b589f5230 2023-11-12 11:18:19,251 [27954] DEBUG: Signature: 8191b3cb07f07b73365fac0ae3700b79b2d1b5d01c3b8db425068facd47faa0e 2023-11-12 11:18:19,251 [27954] DEBUG: Event request-created.s3.HeadBucket: calling handler <function add_retry_headers at 0x7fcd22b1dc10> 2023-11-12 11:18:19,251 [27954] DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=http://minio.minio.svc.cluster.local/test, headers={'User-Agent': b'Boto3/1.28.24 md/Botocore#1.31.24 ua/2.0 os/linux#6.1.46-0-lts md/arch#x86_64 lang/python#3.9.2 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.24 Resource', 'X-Amz-Date': b'20231112T111819Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=minio/20231112/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8191b3cb07f07b73365fac0ae3700b79b2d1b5d01c3b8db425068facd47faa0e', 'amz-sdk-invocation-id': b'fcd48e53-97aa-419c-886c-aa9ed4de63dd', 'amz-sdk-request': b'attempt=1'}> 2023-11-12 11:18:19,252 [27954] DEBUG: Starting new HTTP connection (1): minio.minio.svc.cluster.local:80 2023-11-12 11:18:19,260 [27954] DEBUG: http://minio.minio.svc.cluster.local:80 "HEAD /test HTTP/1.1" 200 0 2023-11-12 11:18:19,260 [27954] DEBUG: Response headers: {'Accept-Ranges': 'bytes', 'Content-Length': '0', 'Content-Type': 'application/xml', 'Server': 'MinIO', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Vary': 'Origin, Accept-Encoding', 'X-Amz-Id-2': 'ad095a5437a80fddc583950b138ef69caabc97ee27d91c44a2614884173128a9', 'X-Amz-Request-Id': '1796DC16A8BBF2FB', 'X-Content-Type-Options': 'nosniff', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 12 Nov 2023 11:18:19 GMT'} 2023-11-12 11:18:19,261 [27954] DEBUG: Response body: b'' 2023-11-12 11:18:19,261 [27954] DEBUG: Event needs-retry.s3.HeadBucket: calling handler <botocore.retryhandler.RetryHandler object at 0x7fcd23264b80> 2023-11-12 11:18:19,261 [27954] DEBUG: No retry needed. 2023-11-12 11:18:19,261 [27954] DEBUG: Event needs-retry.s3.HeadBucket: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,261 [27954] DEBUG: Acquiring 0 2023-11-12 11:18:19,262 [27954] DEBUG: UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fcd22009310>}) about to wait for the following futures [] 2023-11-12 11:18:19,262 [27954] DEBUG: UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fcd22009310>}) done waiting for dependent futures 2023-11-12 11:18:19,262 [27954] DEBUG: Executing task UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fcd22009310>}) with kwargs {'client': <botocore.client.S3 object at 0x7fcd22219e50>, 'config': <boto3.s3.transfer.TransferConfig object at 0x7fcd23282c70>, 'osutil': <s3transfer.utils.OSUtils object at 0x7fcd23282a30>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7fcd23282be0>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fcd22009310>} 2023-11-12 11:18:19,262 [27954] DEBUG: Submitting task CreateMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7fcd23282be0> for transfer request: 0. 2023-11-12 11:18:19,262 [27954] DEBUG: Acquiring 0 2023-11-12 11:18:19,264 [27954] DEBUG: CreateMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) about to wait for the following futures [] 2023-11-12 11:18:19,264 [27954] DEBUG: CreateMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) done waiting for dependent futures 2023-11-12 11:18:19,264 [27954] DEBUG: Executing task CreateMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7fcd22219e50>, 'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}} 2023-11-12 11:18:19,265 [27954] DEBUG: Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function validate_ascii_metadata at 0x7fcd22b1aaf0> 2023-11-12 11:18:19,265 [27954] DEBUG: Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function sse_md5 at 0x7fcd22b16ee0> 2023-11-12 11:18:19,265 [27954] DEBUG: Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function validate_bucket_name at 0x7fcd22b16e50> 2023-11-12 11:18:19,265 [27954] DEBUG: Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fcd22b1dca0> 2023-11-12 11:18:19,265 [27954] DEBUG: Event before-parameter-build.s3.CreateMultipartUpload: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,265 [27954] DEBUG: Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function generate_idempotent_uuid at 0x7fcd22b16ca0> 2023-11-12 11:18:19,266 [27954] DEBUG: Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fcd22b1de50> 2023-11-12 11:18:19,266 [27954] DEBUG: Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,267 [27954] DEBUG: Calling endpoint provider with parameters: {'Bucket': 'test', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://minio.minio.svc.cluster.local', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} 2023-11-12 11:18:19,267 [27954] DEBUG: Endpoint provider result: http://minio.minio.svc.cluster.local/test 2023-11-12 11:18:19,267 [27954] DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2023-11-12 11:18:19,267 [27954] DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} 2023-11-12 11:18:19,268 [27954] DEBUG: Event before-call.s3.CreateMultipartUpload: calling handler <function add_expect_header at 0x7fcd22b1a1f0> 2023-11-12 11:18:19,268 [27954] DEBUG: Event before-call.s3.CreateMultipartUpload: calling handler <function add_recursion_detection_header at 0x7fcd22b16940> 2023-11-12 11:18:19,268 [27954] DEBUG: Event before-call.s3.CreateMultipartUpload: calling handler <function inject_api_version_header_if_needed at 0x7fcd22b1d550> 2023-11-12 11:18:19,268 [27954] DEBUG: Making request for OperationModel(name=CreateMultipartUpload) with params: {'url_path': '/postgres-db/wals/0000000100000000/000000010000000000000001?uploads', 'query_string': {}, 'method': 'POST', 'headers': {'User-Agent': 'Boto3/1.28.24 md/Botocore#1.31.24 ua/2.0 os/linux#6.1.46-0-lts md/arch#x86_64 lang/python#3.9.2 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.24 Resource'}, 'body': b'', 'auth_path': '/test/postgres-db/wals/0000000100000000/000000010000000000000001', 'url': 'http://minio.minio.svc.cluster.local/test/postgres-db/wals/0000000100000000/000000010000000000000001?uploads', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fcd22220160>, 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'test', 'params': {'Bucket': 'test', 'Key': 'postgres-db/wals/0000000100000000/000000010000000000000001'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} 2023-11-12 11:18:19,269 [27954] DEBUG: Submitting task UploadPartTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 1, 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7fcd23282be0> for transfer request: 0. 2023-11-12 11:18:19,269 [27954] DEBUG: Event request-created.s3.CreateMultipartUpload: calling handler <function signal_not_transferring at 0x7fcd2209c0d0> 2023-11-12 11:18:19,270 [27954] DEBUG: Acquiring 0 2023-11-12 11:18:19,270 [27954] DEBUG: Event request-created.s3.CreateMultipartUpload: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fcd22219d30>> 2023-11-12 11:18:19,271 [27954] DEBUG: UploadPartTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 1, 'extra_args': {}}) about to wait for the following futures [<s3transfer.futures.ExecutorFuture object at 0x7fcd22009be0>] 2023-11-12 11:18:19,271 [27954] DEBUG: Event choose-signer.s3.CreateMultipartUpload: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fcd22a1cdc0>> 2023-11-12 11:18:19,272 [27954] DEBUG: Event choose-signer.s3.CreateMultipartUpload: calling handler <function set_operation_specific_signer at 0x7fcd22b16b80> 2023-11-12 11:18:19,273 [27954] DEBUG: Event before-sign.s3.CreateMultipartUpload: calling handler <function remove_arn_from_signing_path at 0x7fcd22b1ddc0> 2023-11-12 11:18:19,273 [27954] DEBUG: Calculating signature using v4 auth. 2023-11-12 11:18:19,273 [27954] DEBUG: CanonicalRequest: POST /test/postgres-db/wals/0000000100000000/000000010000000000000001 uploads= host:minio.minio.svc.cluster.local x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20231112T111819Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-11-12 11:18:19,274 [27954] DEBUG: StringToSign: AWS4-HMAC-SHA256 20231112T111819Z 20231112/us-east-1/s3/aws4_request 7c1c2cd4e59c72042907da64b9855eba3fa60c0981295ee02a94f3c42c5ddf09 2023-11-12 11:18:19,274 [27954] DEBUG: Signature: 3168659840611e3dc7227cc5334e6b1f89c96e7eaaf1c9d3d5680320de897556 2023-11-12 11:18:19,271 [27954] DEBUG: UploadPartTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 1, 'extra_args': {}}) about to wait for <s3transfer.futures.ExecutorFuture object at 0x7fcd22009be0> 2023-11-12 11:18:19,274 [27954] DEBUG: Submitting task UploadPartTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 2, 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7fcd23282be0> for transfer request: 0. 2023-11-12 11:18:19,274 [27954] DEBUG: Event request-created.s3.CreateMultipartUpload: calling handler <function signal_transferring at 0x7fcd2209c160> 2023-11-12 11:18:19,289 [27954] DEBUG: Acquiring 0 2023-11-12 11:18:19,290 [27954] DEBUG: Event request-created.s3.CreateMultipartUpload: calling handler <function add_retry_headers at 0x7fcd22b1dc10> 2023-11-12 11:18:19,290 [27954] DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=http://minio.minio.svc.cluster.local/test/postgres-db/wals/0000000100000000/000000010000000000000001?uploads, headers={'User-Agent': b'Boto3/1.28.24 md/Botocore#1.31.24 ua/2.0 os/linux#6.1.46-0-lts md/arch#x86_64 lang/python#3.9.2 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.24 Resource', 'X-Amz-Date': b'20231112T111819Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=minio/20231112/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3168659840611e3dc7227cc5334e6b1f89c96e7eaaf1c9d3d5680320de897556', 'amz-sdk-invocation-id': b'2c98e0c9-4051-46ca-b027-7504ed759585', 'amz-sdk-request': b'attempt=1', 'Content-Length': '0'}> 2023-11-12 11:18:19,291 [27954] DEBUG: UploadPartTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 2, 'extra_args': {}}) about to wait for the following futures [<s3transfer.futures.ExecutorFuture object at 0x7fcd22009be0>] 2023-11-12 11:18:19,292 [27954] DEBUG: UploadPartTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 2, 'extra_args': {}}) about to wait for <s3transfer.futures.ExecutorFuture object at 0x7fcd22009be0> 2023-11-12 11:18:19,292 [27954] DEBUG: Submitting task CompleteMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7fcd23282be0> for transfer request: 0. 2023-11-12 11:18:19,292 [27954] DEBUG: Acquiring 0 2023-11-12 11:18:19,292 [27954] DEBUG: CompleteMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) about to wait for the following futures [<s3transfer.futures.ExecutorFuture object at 0x7fcd22009be0>, <s3transfer.futures.ExecutorFuture object at 0x7fcd2201cd30>, <s3transfer.futures.ExecutorFuture object at 0x7fcd22028430>] 2023-11-12 11:18:19,314 [27954] DEBUG: http://minio.minio.svc.cluster.local:80 "POST /test/postgres-db/wals/0000000100000000/000000010000000000000001?uploads HTTP/1.1" 200 359 2023-11-12 11:18:19,314 [27954] DEBUG: Releasing acquire 0/None 2023-11-12 11:18:19,314 [27954] DEBUG: CompleteMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) about to wait for <s3transfer.futures.ExecutorFuture object at 0x7fcd22009be0> 2023-11-12 11:18:19,315 [27954] DEBUG: Response headers: {'Accept-Ranges': 'bytes', 'Content-Length': '359', 'Content-Type': 'application/xml', 'Server': 'MinIO', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Vary': 'Origin, Accept-Encoding', 'X-Amz-Id-2': 'ad095a5437a80fddc583950b138ef69caabc97ee27d91c44a2614884173128a9', 'X-Amz-Request-Id': '1796DC16AAD26129', 'X-Content-Type-Options': 'nosniff', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 12 Nov 2023 11:18:19 GMT'} 2023-11-12 11:18:19,315 [27954] DEBUG: Response body: b'<?xml version="1.0" encoding="UTF-8"?>\n<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>test</Bucket><Key>postgres-db/wals/0000000100000000/000000010000000000000001</Key><UploadId>OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ</UploadId></InitiateMultipartUploadResult>' 2023-11-12 11:18:19,316 [27954] DEBUG: Event needs-retry.s3.CreateMultipartUpload: calling handler <botocore.retryhandler.RetryHandler object at 0x7fcd23264b80> 2023-11-12 11:18:19,316 [27954] DEBUG: No retry needed. 2023-11-12 11:18:19,316 [27954] DEBUG: Event needs-retry.s3.CreateMultipartUpload: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,316 [27954] DEBUG: Releasing acquire 0/None 2023-11-12 11:18:19,316 [27954] DEBUG: UploadPartTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 2, 'extra_args': {}}) done waiting for dependent futures 2023-11-12 11:18:19,316 [27954] DEBUG: UploadPartTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 1, 'extra_args': {}}) done waiting for dependent futures 2023-11-12 11:18:19,316 [27954] DEBUG: Executing task UploadPartTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 2, 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7fcd22219e50>, 'fileobj': <s3transfer.utils.ReadFileChunk object at 0x7fcd22028070>, 'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 2, 'extra_args': {}, 'upload_id': 'OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ'} 2023-11-12 11:18:19,316 [27954] DEBUG: CompleteMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) about to wait for <s3transfer.futures.ExecutorFuture object at 0x7fcd2201cd30> 2023-11-12 11:18:19,316 [27954] DEBUG: Executing task UploadPartTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 1, 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7fcd22219e50>, 'fileobj': <s3transfer.utils.ReadFileChunk object at 0x7fcd2201c5e0>, 'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'part_number': 1, 'extra_args': {}, 'upload_id': 'OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ'} 2023-11-12 11:18:19,317 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <function sse_md5 at 0x7fcd22b16ee0> 2023-11-12 11:18:19,318 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <function sse_md5 at 0x7fcd22b16ee0> 2023-11-12 11:18:19,318 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <function convert_body_to_file_like_object at 0x7fcd22b1d430> 2023-11-12 11:18:19,318 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <function validate_bucket_name at 0x7fcd22b16e50> 2023-11-12 11:18:19,318 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <function convert_body_to_file_like_object at 0x7fcd22b1d430> 2023-11-12 11:18:19,318 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fcd22b1dca0> 2023-11-12 11:18:19,319 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,319 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <function generate_idempotent_uuid at 0x7fcd22b16ca0> 2023-11-12 11:18:19,318 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <function validate_bucket_name at 0x7fcd22b16e50> 2023-11-12 11:18:19,319 [27954] DEBUG: Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fcd22b1de50> 2023-11-12 11:18:19,319 [27954] DEBUG: Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,319 [27954] DEBUG: Calling endpoint provider with parameters: {'Bucket': 'test', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://minio.minio.svc.cluster.local', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} 2023-11-12 11:18:19,319 [27954] DEBUG: Endpoint provider result: http://minio.minio.svc.cluster.local/test 2023-11-12 11:18:19,320 [27954] DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2023-11-12 11:18:19,320 [27954] DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} 2023-11-12 11:18:19,320 [27954] DEBUG: Event before-call.s3.UploadPart: calling handler <function conditionally_calculate_md5 at 0x7fcd22cb2e50> 2023-11-12 11:18:19,319 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fcd22b1dca0> 2023-11-12 11:18:19,321 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,321 [27954] DEBUG: Event before-parameter-build.s3.UploadPart: calling handler <function generate_idempotent_uuid at 0x7fcd22b16ca0> 2023-11-12 11:18:19,321 [27954] DEBUG: Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fcd22b1de50> 2023-11-12 11:18:19,321 [27954] DEBUG: Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,322 [27954] DEBUG: Calling endpoint provider with parameters: {'Bucket': 'test', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://minio.minio.svc.cluster.local', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} 2023-11-12 11:18:19,322 [27954] DEBUG: Endpoint provider result: http://minio.minio.svc.cluster.local/test 2023-11-12 11:18:19,322 [27954] DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2023-11-12 11:18:19,322 [27954] DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} 2023-11-12 11:18:19,322 [27954] DEBUG: Event before-call.s3.UploadPart: calling handler <function conditionally_calculate_md5 at 0x7fcd22cb2e50> 2023-11-12 11:18:19,337 [27954] DEBUG: Event before-call.s3.UploadPart: calling handler <function add_expect_header at 0x7fcd22b1a1f0> 2023-11-12 11:18:19,337 [27954] DEBUG: Adding expect 100 continue header to request. 2023-11-12 11:18:19,337 [27954] DEBUG: Event before-call.s3.UploadPart: calling handler <function add_recursion_detection_header at 0x7fcd22b16940> 2023-11-12 11:18:19,337 [27954] DEBUG: Event before-call.s3.UploadPart: calling handler <function inject_api_version_header_if_needed at 0x7fcd22b1d550> 2023-11-12 11:18:19,338 [27954] DEBUG: Making request for OperationModel(name=UploadPart) with params: {'url_path': '/postgres-db/wals/0000000100000000/000000010000000000000001', 'query_string': {'uploadId': 'OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ', 'partNumber': 2}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.28.24 md/Botocore#1.31.24 ua/2.0 os/linux#6.1.46-0-lts md/arch#x86_64 lang/python#3.9.2 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.24 Resource', 'Content-MD5': 'Q+sCWFJ5g6KOfXwLrl5oBw==', 'Expect': '100-continue'}, 'body': <s3transfer.utils.ReadFileChunk object at 0x7fcd22028070>, 'auth_path': '/test/postgres-db/wals/0000000100000000/000000010000000000000001', 'url': 'http://minio.minio.svc.cluster.local/test/postgres-db/wals/0000000100000000/000000010000000000000001?uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ&partNumber=2', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fcd22220160>, 'has_streaming_input': True, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'test', 'params': {'Bucket': 'test', 'Key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'UploadId': 'OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ', 'PartNumber': 2, 'Body': <s3transfer.utils.ReadFileChunk object at 0x7fcd22028070>}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} 2023-11-12 11:18:19,338 [27954] DEBUG: Event request-created.s3.UploadPart: calling handler <function signal_not_transferring at 0x7fcd2209c0d0> 2023-11-12 11:18:19,339 [27954] DEBUG: Event request-created.s3.UploadPart: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fcd22219d30>> 2023-11-12 11:18:19,339 [27954] DEBUG: Event choose-signer.s3.UploadPart: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fcd22a1cdc0>> 2023-11-12 11:18:19,339 [27954] DEBUG: Event choose-signer.s3.UploadPart: calling handler <function set_operation_specific_signer at 0x7fcd22b16b80> 2023-11-12 11:18:19,339 [27954] DEBUG: Event before-sign.s3.UploadPart: calling handler <function remove_arn_from_signing_path at 0x7fcd22b1ddc0> 2023-11-12 11:18:19,349 [27954] DEBUG: Event before-call.s3.UploadPart: calling handler <function add_expect_header at 0x7fcd22b1a1f0> 2023-11-12 11:18:19,349 [27954] DEBUG: Adding expect 100 continue header to request. 2023-11-12 11:18:19,349 [27954] DEBUG: Event before-call.s3.UploadPart: calling handler <function add_recursion_detection_header at 0x7fcd22b16940> 2023-11-12 11:18:19,349 [27954] DEBUG: Event before-call.s3.UploadPart: calling handler <function inject_api_version_header_if_needed at 0x7fcd22b1d550> 2023-11-12 11:18:19,349 [27954] DEBUG: Making request for OperationModel(name=UploadPart) with params: {'url_path': '/postgres-db/wals/0000000100000000/000000010000000000000001', 'query_string': {'uploadId': 'OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ', 'partNumber': 1}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.28.24 md/Botocore#1.31.24 ua/2.0 os/linux#6.1.46-0-lts md/arch#x86_64 lang/python#3.9.2 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.24 Resource', 'Content-MD5': 'qVQ55Mt9tu4dMdz3CJMJxA==', 'Expect': '100-continue'}, 'body': <s3transfer.utils.ReadFileChunk object at 0x7fcd2201c5e0>, 'auth_path': '/test/postgres-db/wals/0000000100000000/000000010000000000000001', 'url': 'http://minio.minio.svc.cluster.local/test/postgres-db/wals/0000000100000000/000000010000000000000001?uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ&partNumber=1', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fcd22220160>, 'has_streaming_input': True, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'test', 'params': {'Bucket': 'test', 'Key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'UploadId': 'OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ', 'PartNumber': 1, 'Body': <s3transfer.utils.ReadFileChunk object at 0x7fcd2201c5e0>}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} 2023-11-12 11:18:19,350 [27954] DEBUG: Event request-created.s3.UploadPart: calling handler <function signal_not_transferring at 0x7fcd2209c0d0> 2023-11-12 11:18:19,350 [27954] DEBUG: Event request-created.s3.UploadPart: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fcd22219d30>> 2023-11-12 11:18:19,350 [27954] DEBUG: Event choose-signer.s3.UploadPart: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fcd22a1cdc0>> 2023-11-12 11:18:19,350 [27954] DEBUG: Event choose-signer.s3.UploadPart: calling handler <function set_operation_specific_signer at 0x7fcd22b16b80> 2023-11-12 11:18:19,350 [27954] DEBUG: Event before-sign.s3.UploadPart: calling handler <function remove_arn_from_signing_path at 0x7fcd22b1ddc0> 2023-11-12 11:18:19,391 [27954] DEBUG: Calculating signature using v4 auth. 2023-11-12 11:18:19,392 [27954] DEBUG: CanonicalRequest: PUT /test/postgres-db/wals/0000000100000000/000000010000000000000001 partNumber=2&uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ content-md5:Q+sCWFJ5g6KOfXwLrl5oBw== host:minio.minio.svc.cluster.local x-amz-content-sha256:0f79b750821ed35378d7e908888b620012bb33d408622e50833d0f64c34a1a3c x-amz-date:20231112T111819Z content-md5;host;x-amz-content-sha256;x-amz-date 0f79b750821ed35378d7e908888b620012bb33d408622e50833d0f64c34a1a3c 2023-11-12 11:18:19,392 [27954] DEBUG: StringToSign: AWS4-HMAC-SHA256 20231112T111819Z 20231112/us-east-1/s3/aws4_request 23b523fde1c54efaa54a079bd08032302cb35a691515d8d2764806cd7b41cd8c 2023-11-12 11:18:19,392 [27954] DEBUG: Signature: 648ae85888e2b12fc6b35c03bbbedc71d9f2d1f7b0c6feccdf241061faceefe1 2023-11-12 11:18:19,392 [27954] DEBUG: Event request-created.s3.UploadPart: calling handler <function signal_transferring at 0x7fcd2209c160> 2023-11-12 11:18:19,392 [27954] DEBUG: Event request-created.s3.UploadPart: calling handler <function add_retry_headers at 0x7fcd22b1dc10> 2023-11-12 11:18:19,392 [27954] DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://minio.minio.svc.cluster.local/test/postgres-db/wals/0000000100000000/000000010000000000000001?uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ&partNumber=2, headers={'User-Agent': b'Boto3/1.28.24 md/Botocore#1.31.24 ua/2.0 os/linux#6.1.46-0-lts md/arch#x86_64 lang/python#3.9.2 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.24 Resource', 'Content-MD5': b'Q+sCWFJ5g6KOfXwLrl5oBw==', 'Expect': b'100-continue', 'X-Amz-Date': b'20231112T111819Z', 'X-Amz-Content-SHA256': b'0f79b750821ed35378d7e908888b620012bb33d408622e50833d0f64c34a1a3c', 'Authorization': b'AWS4-HMAC-SHA256 Credential=minio/20231112/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=648ae85888e2b12fc6b35c03bbbedc71d9f2d1f7b0c6feccdf241061faceefe1', 'amz-sdk-invocation-id': b'31eb46cd-6c55-44cd-9660-7df2840aefe7', 'amz-sdk-request': b'attempt=1', 'Content-Length': '8388608'}> 2023-11-12 11:18:19,394 [27954] DEBUG: Waiting for 100 Continue response. 2023-11-12 11:18:19,403 [27954] DEBUG: Calculating signature using v4 auth. 2023-11-12 11:18:19,403 [27954] DEBUG: CanonicalRequest: PUT /test/postgres-db/wals/0000000100000000/000000010000000000000001 partNumber=1&uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ content-md5:qVQ55Mt9tu4dMdz3CJMJxA== host:minio.minio.svc.cluster.local x-amz-content-sha256:ac00796981635fed3fe8b787635640a532faeeaeb49e871c98b08001fbea3286 x-amz-date:20231112T111819Z content-md5;host;x-amz-content-sha256;x-amz-date ac00796981635fed3fe8b787635640a532faeeaeb49e871c98b08001fbea3286 2023-11-12 11:18:19,403 [27954] DEBUG: StringToSign: AWS4-HMAC-SHA256 20231112T111819Z 20231112/us-east-1/s3/aws4_request 04c54e15f6c9ef3880a44d5e1b0af774426a72af0152e9d92ce22db5f421c55c 2023-11-12 11:18:19,403 [27954] DEBUG: Signature: 3e9323c81c5daad4b04f06866e1d13172e6e904aa1ef0057e32126cf96a66688 2023-11-12 11:18:19,403 [27954] DEBUG: Event request-created.s3.UploadPart: calling handler <function signal_transferring at 0x7fcd2209c160> 2023-11-12 11:18:19,403 [27954] DEBUG: Event request-created.s3.UploadPart: calling handler <function add_retry_headers at 0x7fcd22b1dc10> 2023-11-12 11:18:19,403 [27954] DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://minio.minio.svc.cluster.local/test/postgres-db/wals/0000000100000000/000000010000000000000001?uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ&partNumber=1, headers={'User-Agent': b'Boto3/1.28.24 md/Botocore#1.31.24 ua/2.0 os/linux#6.1.46-0-lts md/arch#x86_64 lang/python#3.9.2 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.24 Resource', 'Content-MD5': b'qVQ55Mt9tu4dMdz3CJMJxA==', 'Expect': b'100-continue', 'X-Amz-Date': b'20231112T111819Z', 'X-Amz-Content-SHA256': b'ac00796981635fed3fe8b787635640a532faeeaeb49e871c98b08001fbea3286', 'Authorization': b'AWS4-HMAC-SHA256 Credential=minio/20231112/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=3e9323c81c5daad4b04f06866e1d13172e6e904aa1ef0057e32126cf96a66688', 'amz-sdk-invocation-id': b'15313c4f-24ee-4fc6-87d2-31910c7a5b6a', 'amz-sdk-request': b'attempt=1', 'Content-Length': '8388608'}> 2023-11-12 11:18:19,404 [27954] DEBUG: Starting new HTTP connection (2): minio.minio.svc.cluster.local:80 2023-11-12 11:18:19,405 [27954] DEBUG: 100 Continue response seen, now sending request body. 2023-11-12 11:18:19,407 [27954] DEBUG: Waiting for 100 Continue response. 2023-11-12 11:18:19,428 [27954] DEBUG: 100 Continue response seen, now sending request body. 2023-11-12 11:18:19,592 [27954] DEBUG: http://minio.minio.svc.cluster.local:80 "PUT /test/postgres-db/wals/0000000100000000/000000010000000000000001?uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ&partNumber=2 HTTP/1.1" 200 0 2023-11-12 11:18:19,592 [27954] DEBUG: Response headers: {'Accept-Ranges': 'bytes', 'Content-Length': '0', 'ETag': '"43eb0258527983a28e7d7c0bae5e6807"', 'Server': 'MinIO', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Vary': 'Origin, Accept-Encoding', 'X-Amz-Id-2': 'ad095a5437a80fddc583950b138ef69caabc97ee27d91c44a2614884173128a9', 'X-Amz-Request-Id': '1796DC16B0EC4EEB', 'X-Content-Type-Options': 'nosniff', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 12 Nov 2023 11:18:19 GMT'} 2023-11-12 11:18:19,593 [27954] DEBUG: Response body: b'' 2023-11-12 11:18:19,593 [27954] DEBUG: Event needs-retry.s3.UploadPart: calling handler <botocore.retryhandler.RetryHandler object at 0x7fcd23264b80> 2023-11-12 11:18:19,593 [27954] DEBUG: No retry needed. 2023-11-12 11:18:19,594 [27954] DEBUG: Event needs-retry.s3.UploadPart: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,594 [27954] DEBUG: Releasing acquire 0/None 2023-11-12 11:18:19,607 [27954] DEBUG: http://minio.minio.svc.cluster.local:80 "PUT /test/postgres-db/wals/0000000100000000/000000010000000000000001?uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ&partNumber=1 HTTP/1.1" 200 0 2023-11-12 11:18:19,608 [27954] DEBUG: Response headers: {'Accept-Ranges': 'bytes', 'Content-Length': '0', 'ETag': '"a95439e4cb7db6ee1d31dcf7089309c4"', 'Server': 'MinIO', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Vary': 'Origin, Accept-Encoding', 'X-Amz-Id-2': 'ad095a5437a80fddc583950b138ef69caabc97ee27d91c44a2614884173128a9', 'X-Amz-Request-Id': '1796DC16B21152F1', 'X-Content-Type-Options': 'nosniff', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 12 Nov 2023 11:18:19 GMT'} 2023-11-12 11:18:19,608 [27954] DEBUG: Response body: b'' 2023-11-12 11:18:19,608 [27954] DEBUG: Event needs-retry.s3.UploadPart: calling handler <botocore.retryhandler.RetryHandler object at 0x7fcd23264b80> 2023-11-12 11:18:19,608 [27954] DEBUG: No retry needed. 2023-11-12 11:18:19,608 [27954] DEBUG: Event needs-retry.s3.UploadPart: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,609 [27954] DEBUG: Releasing acquire 0/None 2023-11-12 11:18:19,609 [27954] DEBUG: CompleteMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) about to wait for <s3transfer.futures.ExecutorFuture object at 0x7fcd22028430> 2023-11-12 11:18:19,609 [27954] DEBUG: CompleteMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) done waiting for dependent futures 2023-11-12 11:18:19,609 [27954] DEBUG: Executing task CompleteMultipartUploadTask(transfer_id=0, {'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7fcd22219e50>, 'bucket': 'test', 'key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'extra_args': {}, 'upload_id': 'OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ', 'parts': [{'ETag': '"a95439e4cb7db6ee1d31dcf7089309c4"', 'PartNumber': 1}, {'ETag': '"43eb0258527983a28e7d7c0bae5e6807"', 'PartNumber': 2}]} 2023-11-12 11:18:19,610 [27954] DEBUG: Event before-parameter-build.s3.CompleteMultipartUpload: calling handler <function validate_bucket_name at 0x7fcd22b16e50> 2023-11-12 11:18:19,610 [27954] DEBUG: Event before-parameter-build.s3.CompleteMultipartUpload: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fcd22b1dca0> 2023-11-12 11:18:19,610 [27954] DEBUG: Event before-parameter-build.s3.CompleteMultipartUpload: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,610 [27954] DEBUG: Event before-parameter-build.s3.CompleteMultipartUpload: calling handler <function generate_idempotent_uuid at 0x7fcd22b16ca0> 2023-11-12 11:18:19,610 [27954] DEBUG: Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fcd22b1de50> 2023-11-12 11:18:19,610 [27954] DEBUG: Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,610 [27954] DEBUG: Calling endpoint provider with parameters: {'Bucket': 'test', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://minio.minio.svc.cluster.local', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} 2023-11-12 11:18:19,610 [27954] DEBUG: Endpoint provider result: http://minio.minio.svc.cluster.local/test 2023-11-12 11:18:19,610 [27954] DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2023-11-12 11:18:19,610 [27954] DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} 2023-11-12 11:18:19,611 [27954] DEBUG: Event before-call.s3.CompleteMultipartUpload: calling handler <function add_expect_header at 0x7fcd22b1a1f0> 2023-11-12 11:18:19,611 [27954] DEBUG: Event before-call.s3.CompleteMultipartUpload: calling handler <function add_recursion_detection_header at 0x7fcd22b16940> 2023-11-12 11:18:19,611 [27954] DEBUG: Event before-call.s3.CompleteMultipartUpload: calling handler <function inject_api_version_header_if_needed at 0x7fcd22b1d550> 2023-11-12 11:18:19,611 [27954] DEBUG: Making request for OperationModel(name=CompleteMultipartUpload) with params: {'url_path': '/postgres-db/wals/0000000100000000/000000010000000000000001', 'query_string': {'uploadId': 'OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ'}, 'method': 'POST', 'headers': {'User-Agent': 'Boto3/1.28.24 md/Botocore#1.31.24 ua/2.0 os/linux#6.1.46-0-lts md/arch#x86_64 lang/python#3.9.2 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.24 Resource'}, 'body': b'<CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Part><ETag>"a95439e4cb7db6ee1d31dcf7089309c4"</ETag><PartNumber>1</PartNumber></Part><Part><ETag>"43eb0258527983a28e7d7c0bae5e6807"</ETag><PartNumber>2</PartNumber></Part></CompleteMultipartUpload>', 'auth_path': '/test/postgres-db/wals/0000000100000000/000000010000000000000001', 'url': 'http://minio.minio.svc.cluster.local/test/postgres-db/wals/0000000100000000/000000010000000000000001?uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fcd22220160>, 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'test', 'params': {'Bucket': 'test', 'Key': 'postgres-db/wals/0000000100000000/000000010000000000000001', 'UploadId': 'OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ', 'MultipartUpload': {'Parts': [{'ETag': '"a95439e4cb7db6ee1d31dcf7089309c4"', 'PartNumber': 1}, {'ETag': '"43eb0258527983a28e7d7c0bae5e6807"', 'PartNumber': 2}]}}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} 2023-11-12 11:18:19,611 [27954] DEBUG: Event request-created.s3.CompleteMultipartUpload: calling handler <function signal_not_transferring at 0x7fcd2209c0d0> 2023-11-12 11:18:19,611 [27954] DEBUG: Event request-created.s3.CompleteMultipartUpload: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fcd22219d30>> 2023-11-12 11:18:19,611 [27954] DEBUG: Event choose-signer.s3.CompleteMultipartUpload: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fcd22a1cdc0>> 2023-11-12 11:18:19,611 [27954] DEBUG: Event choose-signer.s3.CompleteMultipartUpload: calling handler <function set_operation_specific_signer at 0x7fcd22b16b80> 2023-11-12 11:18:19,612 [27954] DEBUG: Event before-sign.s3.CompleteMultipartUpload: calling handler <function remove_arn_from_signing_path at 0x7fcd22b1ddc0> 2023-11-12 11:18:19,612 [27954] DEBUG: Calculating signature using v4 auth. 2023-11-12 11:18:19,612 [27954] DEBUG: CanonicalRequest: POST /test/postgres-db/wals/0000000100000000/000000010000000000000001 uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ host:minio.minio.svc.cluster.local x-amz-content-sha256:0c029987c54e5acc1ca2bd9e7426645cdd5a8b00ab258f05fe7b4db454244045 x-amz-date:20231112T111819Z host;x-amz-content-sha256;x-amz-date 0c029987c54e5acc1ca2bd9e7426645cdd5a8b00ab258f05fe7b4db454244045 2023-11-12 11:18:19,612 [27954] DEBUG: StringToSign: AWS4-HMAC-SHA256 20231112T111819Z 20231112/us-east-1/s3/aws4_request 9b9b17bc4ac9bd8a47c41d7e417339250a6760f0983cdae9b01d1058776fcf28 2023-11-12 11:18:19,612 [27954] DEBUG: Signature: e1f91c1ab06bed13036585874e398e0a1bc7f68b7826573a79fa113a137beec8 2023-11-12 11:18:19,612 [27954] DEBUG: Event request-created.s3.CompleteMultipartUpload: calling handler <function signal_transferring at 0x7fcd2209c160> 2023-11-12 11:18:19,613 [27954] DEBUG: Event request-created.s3.CompleteMultipartUpload: calling handler <function add_retry_headers at 0x7fcd22b1dc10> 2023-11-12 11:18:19,613 [27954] DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=http://minio.minio.svc.cluster.local/test/postgres-db/wals/0000000100000000/000000010000000000000001?uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ, headers={'User-Agent': b'Boto3/1.28.24 md/Botocore#1.31.24 ua/2.0 os/linux#6.1.46-0-lts md/arch#x86_64 lang/python#3.9.2 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.24 Resource', 'X-Amz-Date': b'20231112T111819Z', 'X-Amz-Content-SHA256': b'0c029987c54e5acc1ca2bd9e7426645cdd5a8b00ab258f05fe7b4db454244045', 'Authorization': b'AWS4-HMAC-SHA256 Credential=minio/20231112/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e1f91c1ab06bed13036585874e398e0a1bc7f68b7826573a79fa113a137beec8', 'amz-sdk-invocation-id': b'e13c1eeb-8d9a-4f0f-8fb9-bfe31c457bbe', 'amz-sdk-request': b'attempt=1', 'Content-Length': '271'}> 2023-11-12 11:18:19,626 [27954] DEBUG: http://minio.minio.svc.cluster.local:80 "POST /test/postgres-db/wals/0000000100000000/000000010000000000000001?uploadId=OTcyNjBlNDYtN2Y4YS00NjcxLWFmMzktZjY3MjQ4MmFhMDc2LmFhODVjNTA2LWM2MWUtNDdjMC04MjhlLTkwNThiMDUzYjZiYQ HTTP/1.1" 200 418 2023-11-12 11:18:19,626 [27954] DEBUG: Response headers: {'Accept-Ranges': 'bytes', 'Cache-Control': 'no-cache', 'Content-Length': '418', 'Content-Type': 'application/xml', 'ETag': '"0e6e8bf28fef6826afcdba607ccc96e1-2"', 'Server': 'MinIO', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Vary': 'Origin, Accept-Encoding', 'X-Accel-Buffering': 'no', 'X-Amz-Id-2': 'ad095a5437a80fddc583950b138ef69caabc97ee27d91c44a2614884173128a9', 'X-Amz-Request-Id': '1796DC16BE030952', 'X-Content-Type-Options': 'nosniff', 'X-Xss-Protection': '1; mode=block', 'Date': 'Sun, 12 Nov 2023 11:18:19 GMT'} 2023-11-12 11:18:19,626 [27954] DEBUG: Response body: b'<?xml version="1.0" encoding="UTF-8"?>\n<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>http://minio.minio.svc.cluster.local/test/postgres-db/wals/0000000100000000/000000010000000000000001</Location><Bucket>test</Bucket><Key>postgres-db/wals/0000000100000000/000000010000000000000001</Key><ETag>"0e6e8bf28fef6826afcdba607ccc96e1-2"</ETag></CompleteMultipartUploadResult>' 2023-11-12 11:18:19,627 [27954] DEBUG: Event needs-retry.s3.CompleteMultipartUpload: calling handler <function check_for_200_error at 0x7fcd22b16a60> 2023-11-12 11:18:19,627 [27954] DEBUG: Event needs-retry.s3.CompleteMultipartUpload: calling handler <botocore.retryhandler.RetryHandler object at 0x7fcd23264b80> 2023-11-12 11:18:19,627 [27954] DEBUG: No retry needed. 2023-11-12 11:18:19,627 [27954] DEBUG: Event needs-retry.s3.CompleteMultipartUpload: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fcd23264c10>> 2023-11-12 11:18:19,627 [27954] DEBUG: Releasing acquire 0/NoneThis works. This is fine. I can see an upload in my bucket.
Now this is how I use the controller manager from cli:
/controller/manager wal-archive --log-destination walarchive.log pg_wal/000000010000000000000001 --zap-log-level debugThis is the part of my cluster-config.yaml
apiVersion: v1 kind: Secret metadata: name: aws-creds namespace: postgres-db labels: cnpg.io/reload: "true" type: Opaque stringData: AWS_ACCESS_KEY_ID: bWluaW8K AWS_SECRET_ACCESS_KEY: bWluaW8xMjMKapiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: namespace: postgres-db name: postgres-db spec: backup: barmanObjectStore: destinationPath: s3://test endpointURL: http://minio.minio.svc.cluster.local s3Credentials: accessKeyId: name: aws-creds key: AWS_ACCESS_KEY_ID secretAccessKey: name: aws-creds key: AWS_SECRET_ACCESS_KEY wal: compression: gzip (...)And this is the response:
{"level":"info","ts":"2023-11-12T11:34:03Z","logger":"wal-archive","msg":"barman-cloud-check-wal-archive checking the first wal","logging_pod":"postgres-db-1"} {"level":"info","ts":"2023-11-12T11:34:04Z","logger":"barman-cloud-check-wal-archive","msg":"2023-11-12 11:34:04,318 [694] ERROR: Barman cloud WAL archive check exception: An error occurred (403) when calling the HeadBucket operation: Forbidden","pipe":"stderr","logging_pod":"postgres-db-1"} {"level":"error","ts":"2023-11-12T11:34:04Z","logger":"wal-archive","msg":"Error invoking barman-cloud-check-wal-archive","logging_pod":"postgres-db-1","currentPrimary":"postgres-db-1","targetPrimary":"postgres-db-1","options":["--endpoint-url","http://minio.minio.svc.cluster.local","--cloud-provider","aws-s3","s3://test","postgres-db"],"exitCode":-1,"error":"exit status 4","stacktrace":"github.com/cloudnative-pg/cloudnative-pg/pkg/management/log.(*logger).Error\n\tpkg/management/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/barman/archiver.(*WALArchiver).CheckWalArchiveDestination\n\tpkg/management/barman/archiver/archiver.go:257\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.checkWalArchive\n\tinternal/cmd/manager/walarchive/cmd.go:383\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.run\n\tinternal/cmd/manager/walarchive/cmd.go:169\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:81\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nmain.main\n\tcmd/manager/main.go:64\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.6/x64/src/runtime/proc.go:250"} {"level":"error","ts":"2023-11-12T11:34:04Z","msg":"while barman-cloud-check-wal-archive","logging_pod":"postgres-db-1","error":"unexpected failure invoking barman-cloud-wal-archive: exit status 4","stacktrace":"github.com/cloudnative-pg/cloudnative-pg/pkg/management/log.(*logger).Error\n\tpkg/management/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/log.Error\n\tpkg/management/log/log.go:166\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.checkWalArchive\n\tinternal/cmd/manager/walarchive/cmd.go:384\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.run\n\tinternal/cmd/manager/walarchive/cmd.go:169\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:81\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nmain.main\n\tcmd/manager/main.go:64\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.6/x64/src/runtime/proc.go:250"} {"level":"error","ts":"2023-11-12T11:34:04Z","logger":"wal-archive","msg":"failed to run wal-archive command","logging_pod":"postgres-db-1","error":"unexpected failure invoking barman-cloud-wal-archive: exit status 4","stacktrace":"github.com/cloudnative-pg/cloudnative-pg/pkg/management/log.(*logger).Error\n\tpkg/management/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:83\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nmain.main\n\tcmd/manager/main.go:64\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.6/x64/src/runtime/proc.go:250"}just to be sure in terms of credentials: `echo bWluaW8K | base64 -d minio
echo bWluaW8xMjMK | base64 -d minio123`
I would appreciate any help from your side
Open cluster’s primary container’s shell and: export AWS_SECRET_ACCESS_KEY= export AWS_ACCESS_KEY_ID= export AWS_CA_BUNDLE=/controller/certificates/backup-barman-ca.crt of course input Your S3 credentials in two first variables above. Than run: barman-cloud-wal-archive -vv --cloud-provider aws-s3 --endpoint-url https://my.minio.url s3://backup postgres-db /var/lib/postgresql/data/pgdata/pg_wal/000000010000000000000001 That should give You an insight to what is actually wrong.