docker-pgautoupgrade: initdb: error "/var/lib/postgresql/data" exists but is not empty

okay, now realizing that isn’t actually a standalone reproduction, requires the volume having something in it first. standby…

okay, in additinon to the above facepalm, i also realized i wasn’t deploying my patched version of your code but was building main.

now i’m back on track possibly getting better info. i got a build with all this glorious log output. then i put i more logging and for some reason the next run didn’t have any output. still trying to figure out if my fault or something else, but want to safe this here for posterity. the interesting things are

• in the middle mv: can't rename '/var/lib/postgresql/data/pgdata/old': Invalid argument - i put in some logging here to try to figure out what it’s trying to move, but that didn’t work… yet
• at the end

  2024-03-16 02:20:41.927 UTC [30] FATAL:  lock file "postmaster.pid" already exists
  2024-03-16 02:20:41.927 UTC [30] HINT:  Is another postmaster (PID 1) running in data directory "/var/lib/postgresql/data/pgdata/old"?
  

PostgreSQL Database directory appears to contain a database; Skipping initialization

************************************
PostgreSQL data directory: /var/lib/postgresql/data/pgdata
************************************
*******************************************************************************************
Performing PG upgrade on version 12 database files.  Upgrading to version 16
*******************************************************************************************
----------------------------------------------------------------------
Checking for left over artifacts from a failed previous autoupgrade...
----------------------------------------------------------------------
-------------------------------------------------------------------------------
No artifacts found from a failed previous autoupgrade.  Continuing the process.
-------------------------------------------------------------------------------
---------------------------------------
Creating OLD temporary directory /var/lib/postgresql/data/pgdata/old
---------------------------------------
--------------------------------------------
Creating OLD temporary directory is complete
--------------------------------------------
-------------------------------------------------------
Moving existing data files into OLD temporary directory
-------------------------------------------------------
'/var/lib/postgresql/data/pgdata/PG_VERSION' -> '/var/lib/postgresql/data/pgdata/old/PG_VERSION'
'/var/lib/postgresql/data/pgdata/base' -> '/var/lib/postgresql/data/pgdata/old/base'
mv: can't rename '/var/lib/postgresql/data/pgdata/old': Invalid argument
'/var/lib/postgresql/data/pgdata/global' -> '/var/lib/postgresql/data/pgdata/old/global'
'/var/lib/postgresql/data/pgdata/old' -> '/var/lib/postgresql/data/pgdata/old/old'
'/var/lib/postgresql/data/pgdata/pg_commit_ts' -> '/var/lib/postgresql/data/pgdata/old/pg_commit_ts'
'/var/lib/postgresql/data/pgdata/pg_dynshmem' -> '/var/lib/postgresql/data/pgdata/old/pg_dynshmem'
'/var/lib/postgresql/data/pgdata/pg_hba.conf' -> '/var/lib/postgresql/data/pgdata/old/pg_hba.conf'
'/var/lib/postgresql/data/pgdata/pg_ident.conf' -> '/var/lib/postgresql/data/pgdata/old/pg_ident.conf'
'/var/lib/postgresql/data/pgdata/pg_logical' -> '/var/lib/postgresql/data/pgdata/old/pg_logical'
'/var/lib/postgresql/data/pgdata/pg_multixact' -> '/var/lib/postgresql/data/pgdata/old/pg_multixact'
'/var/lib/postgresql/data/pgdata/pg_notify' -> '/var/lib/postgresql/data/pgdata/old/pg_notify'
'/var/lib/postgresql/data/pgdata/pg_replslot' -> '/var/lib/postgresql/data/pgdata/old/pg_replslot'
'/var/lib/postgresql/data/pgdata/pg_serial' -> '/var/lib/postgresql/data/pgdata/old/pg_serial'
'/var/lib/postgresql/data/pgdata/pg_snapshots' -> '/var/lib/postgresql/data/pgdata/old/pg_snapshots'
'/var/lib/postgresql/data/pgdata/pg_stat' -> '/var/lib/postgresql/data/pgdata/old/pg_stat'
'/var/lib/postgresql/data/pgdata/pg_stat_tmp' -> '/var/lib/postgresql/data/pgdata/old/pg_stat_tmp'
'/var/lib/postgresql/data/pgdata/pg_subtrans' -> '/var/lib/postgresql/data/pgdata/old/pg_subtrans'
'/var/lib/postgresql/data/pgdata/pg_tblspc' -> '/var/lib/postgresql/data/pgdata/old/pg_tblspc'
'/var/lib/postgresql/data/pgdata/pg_twophase' -> '/var/lib/postgresql/data/pgdata/old/pg_twophase'
'/var/lib/postgresql/data/pgdata/pg_wal' -> '/var/lib/postgresql/data/pgdata/old/pg_wal'
'/var/lib/postgresql/data/pgdata/pg_xact' -> '/var/lib/postgresql/data/pgdata/old/pg_xact'
'/var/lib/postgresql/data/pgdata/postgresql.auto.conf' -> '/var/lib/postgresql/data/pgdata/old/postgresql.auto.conf'
'/var/lib/postgresql/data/pgdata/postgresql.conf' -> '/var/lib/postgresql/data/pgdata/old/postgresql.conf'
'/var/lib/postgresql/data/pgdata/postmaster.opts' -> '/var/lib/postgresql/data/pgdata/old/postmaster.opts'
'/var/lib/postgresql/data/pgdata/postmaster.pid' -> '/var/lib/postgresql/data/pgdata/old/postmaster.pid'
-------------------------------------------------------------------
Moving existing data files into OLD temporary directory is complete
-------------------------------------------------------------------
---------------------------------------
Creating NEW temporary directory /var/lib/postgresql/data/pgdata/new
---------------------------------------
--------------------------------------------
Creating NEW temporary directory is complete
--------------------------------------------
-----------------------------------------------------
Changing permissions of temporary directories to 0700
-----------------------------------------------------
---------------------------------------------------------
Changing permissions of temporary directories is complete
---------------------------------------------------------
------------------------------------
Determining our own initdb arguments
------------------------------------
2024-03-16 02:20:41.927 UTC [30] FATAL:  lock file "postmaster.pid" already exists
2024-03-16 02:20:41.927 UTC [30] HINT:  Is another postmaster (PID 1) running in data directory "/var/lib/postgresql/data/pgdata/old"?

i knew i was flying too close to the sun thinking i could use false in shell scripting. my text editor highlights it a different color! why!

postgres-6d5984cd68-7fgng:/# if [ "one" == "two" ]; then echo yes; fi
postgres-6d5984cd68-7fgng:/# if [ false ]; then echo yes; fi
yes

i think it made it past this

initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.

and then it died after this

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /tmp/my-empty-dir ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

    pg_ctl -D /tmp/my-empty-dir -l logfile start

initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2024-03-16 00:13:11.133 UTC [39] LOG:  starting PostgreSQL 16.2 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit
2024-03-16 00:13:11.136 UTC [39] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-03-16 00:13:11.149 UTC [42] LOG:  database system was shut down at 2024-03-16 00:13:07 UTC
2024-03-16 00:13:11.156 UTC [39] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down....2024-03-16 00:13:11.283 UTC [39] LOG:  received fast shutdown request
2024-03-16 00:13:11.288 UTC [39] LOG:  aborting any active transactions
2024-03-16 00:13:11.289 UTC [39] LOG:  background worker "logical replication launcher" (PID 45) exited with exit code 1
2024-03-16 00:13:11.291 UTC [40] LOG:  shutting down
2024-03-16 00:13:11.294 UTC [40] LOG:  checkpoint starting: shutdown immediate
.2024-03-16 00:13:12.346 UTC [40] LOG:  checkpoint complete: wrote 935 buffers (5.7%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.016 s, sync=1.016 s, total=1.056 s; sync files=301, longest=0.012 s, average=0.004 s; distance=4327 kB, estimate=4327 kB; lsn=0/19495A8, redo lsn=0/19495A8
2024-03-16 00:13:12.352 UTC [39] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

************************************
PostgreSQL data directory: /tmp/my-empty-dir
************************************
****************************************************************************
Unrecognised version of PostgreSQL database files found, aborting completely
****************************************************************************

The pgautoupgrade container should definitely support upgrading from PG 12.

right but the makefile doesn’t support building PGTARGET=12 (upgrading to 12)

but changing up the invocation was trivial, so i think we are off to the races

seems to be building just fine (will be half an hour because building AMD on my ARM mac)

my existing version: 12 🫠

wow, thanks for the offer - although i fear the turnaround time on each experiment will be too long, so maybe we should get me a working shell first. you can find my email off of my website linked here https://github.com/jjb

Yep, for testing purposes, just to see if the script runs ok with it gone. 😄

okay, just nuke the whole volumes section?

sounds good, thanks! but i think maybe you missed the error message i got when i tried my hack - see the text under the image here https://github.com/pgautoupgrade/docker-pgautoupgrade/issues/7#issuecomment-2000638399

in my environment, i need a passing pg healthcheck before i can SSH in

Ahhhh. Oh, now that all makes sense.

Ahhh, I have an idea. Gimme a minute to write it… 😄

initdb: error: directory “/var/lib/postgresql/data/pgdata” exists but is not empty initdb: hint: If you want to create a new database system, either remove or empty the directory “/var/lib/postgresql/data/pgdata” or run initdb with an argument other than “/var/lib/postgresql/data/pgdata”.

never mind, i see the entrypoint needs to run… will figure out a better way to NOOP it

will work on that

I tested with dev tag, it works now!

No worries at all. Thanks for trying it out, and providing feedback, which is how projects improve over time. 😄