supabase: Fresh self-hosted Docker instance fails due to database users' passwords not being set

Bug report

  • I confirm this is a bug with Supabase, not with my own application.
  • I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

When running a new self-hosted instance of Supabase with docker compose, the supabase-analytics container fails to start. Examining the docker compose logs for supabase-analytics reveals:

Kernel pid terminated (application_controller) ({application_start_failure,logflare,{{shutdown,{failed_to_start_child,'Elixir.Cainophile.Adapters.Postgres',invalid_password}},{'Elixir.Logflare.Application',start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...done
sh: 19:50:15.648: not found
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 10973ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

  1. Ensuring your database is available and that you can connect to it
  2. Tracking down slow queries and making sure they are running fast enough
  3. Increasing the pool_size (although this increases resource consumption)
  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval

See DBConnection.start_link/2 for more information

Checking the supabase-db container logs reveals:

172.18.0.3 2023-11-08 19:53:42.971 UTC [2578] supabase_admin@postgres FATAL:  password authentication failed for user "supabase_admin"
172.18.0.3 2023-11-08 19:53:42.971 UTC [2578] supabase_admin@postgres DETAIL:  User "supabase_admin" has no password assigned.
        Connection matched pg_hba.conf line 89: "host  all  all  172.16.0.0/12  scram-sha-256"

If I exec into supabase-db and run echo $POSTGRES_PASSWORD, the correct password is printed. This error occurs regardless of whether the default values in .env have been changed.

Logging into the postgres shell as user ‘postgres’ and running \du shows the following users configured on the database:

 List of roles
         Role name          |                         Attributes                         |                    Member of
----------------------------+------------------------------------------------------------+--------------------------------------------------
 anon                       | No inheritance, Cannot login                               | {}
 authenticated              | No inheritance, Cannot login                               | {}
 authenticator              | No inheritance                                             | {supabase_admin,anon,authenticated,service_role}
 dashboard_user             | Create role, Create DB, Cannot login, Replication          | {}
 pgbouncer                  |                                                            | {}
 postgres                   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 service_role               | No inheritance, Cannot login, Bypass RLS                   | {}
 supabase_admin             | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 supabase_auth_admin        | No inheritance, Create role                                | {}
 supabase_read_only_user    | Bypass RLS                                                 | {pg_read_all_data}
 supabase_replication_admin | Replication                                                | {}
 supabase_storage_admin     | No inheritance, Create role                                | {}

Note that supabase_functions_admin is not listed here while it is present in roles.sql. If I manually set the passwords for each user listed in roles.sql and then run docker compose restart, the db and analytics containers build correctly but the auth container fails. Logs indicate an error while running the 00_init_auth_schema.up.sql: ERROR: must be owner of function uid

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

  1. Clone the supabase repository with git clone --depth 1 https://github.com/supabase/supabase
  2. cd into supabase/docker and copy environment vars with cp .env.example .env
  3. Pull the latest images with docker compose pull
  4. Start the services with docker compose up -d

Expected behavior

supabase-database container should be configured correctly with the accounts listed in roles.sql and password from .env file.

System information

  • OS: Red Hat 8

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 20 (4 by maintainers)

Most upvoted comments

Hi all, i want to test supabase on my own ubuntu VM. i just follow the instruction here https://supabase.com/docs/guides/self-hosting/docker. Without changing anything, i follow these steps. image

These are the container status: image

This is the supabase-analytics logs : image

I suspect it crashed in supabase-analytics which contribute to the storage-api, realtime-dev.supabase-realtime, supabase-rest, supabase-kong keep restarting.

May i know what should i do to solve this issue?

I had try to resolve by: docker system prune -a rm -rf docker/volumes/db/data docker compose up

I also try to exec it into supabase-db: ALTER USER supabase_admin WITH PASSWORD ‘your-super-secret-and-long-postgres-password’;

But the problem stil presists. My situation is difference from the other post is that i did not change any credential, i followed the official guide to fresh install.

I will close this one as it is resolved with better self-hosting instructions for analytics

Please comment if you are still experiencing issues or want to see more improvements

@joewin319 Same issue here, and I did not find a proper solution.

Sometimes it just worked to restart docker with sudo docker compose restart

But the safer approach is to reset the whole database:

  1. Local machine Dump data from the db like this:
    npx supabase db dump -s public -f supabase/seed.sql --db-url "postgres://postgres:PASSWORD@IP:5432/postgres"
  2. Ubuntu VM Stop all containers like this: sudo docker compose -f docker-compose.yml -f docker-compose.s3.yml down
  3. Ubuntu VM Remove all the db volumes like this: cd volumes/db then sudo rm -rf /data and go back cd ../..
  4. Ubuntu VM Start all containers again like this: sudo docker compose -f docker-compose.yml -f docker-compose.s3.yml up -d
  5. Local machine Push the dumped data again: npx supabase db push --include-seed --db-url postgres://postgres:PASSWORD@IP:5432/postgres

Take care especially with the last step to run it on a local db first.

To enable the analytics container you need to enable it in config.toml

[analytics]
enabled = true

Doc Link - https://supabase.com/docs/guides/cli/config#analytics-config

Look at @Jaxius6’s prcoess below. Multiple issues to solve, from proper authentication to proper LF encoding (not CRLF).

Yep, see the closing comment. If you have changed the password, ensure there are no special chars (or it is encoded properly) and that each container is picking up the change

I’m running through the Self-Hosting w/Docker instructions and hitting the same issue. After changing the the default POSTGRES_PASSWORD and attempting the solutions suggested in password authentication failed for user “supabase_auth_admin” - Docker Sefl Hosting #11957, the analytics and db containers fail to start with the following errors respectively:

17:10:03.190 [error] Postgrex.Protocol (#PID<0.4729.0>) failed to connect: ** (Postgrex.Error) FATAL 28P01 (invalid_password) password authentication failed for user "supabase_admin"
192.168.48.5 2024-02-08 17:31:32.953 UTC [118] supabase_admin@postgres FATAL:  password authentication failed for user "supabase_admin"

I can then run docker compose restart to get the some of the containers to start and the Supabase GUI to load in my browser, however several of the containers are in an unhealthy state and the logs show database connections errors.

CONTAINER ID   IMAGE                              COMMAND                   CREATED          STATUS                             PORTS                                                                                                      NAMES
d7680ea347e4   supabase/storage-api:v0.46.4       "docker-entrypoint.s…"    7 minutes ago    Restarting (1) 27 seconds ago                                                                                                                 supabase-storage
940ee1055630   supabase/edge-runtime:v1.33.5      "edge-runtime start …"    7 minutes ago    Up 5 minutes                                                                                                                                  supabase-edge-functions
146f42378a80   supabase/gotrue:v2.132.3           "auth"                    7 minutes ago    Restarting (1) 6 seconds ago                                                                                                                  supabase-auth
b0b5cd543ca7   supabase/postgres-meta:v0.75.0     "docker-entrypoint.s…"    7 minutes ago    Up 5 minutes (healthy)             8080/tcp                                                                                                   supabase-meta
568155ff7fe9   kong:2.8.1                         "bash -c 'eval \"echo…"   7 minutes ago    Up 5 minutes (healthy)             0.0.0.0:8000->8000/tcp, :::8000->8000/tcp, 8001/tcp, 0.0.0.0:8443->8443/tcp, :::8443->8443/tcp, 8444/tcp   supabase-kong
a11c661fd9a9   supabase/realtime:v2.25.50         "/usr/bin/tini -s -g…"    7 minutes ago    Up 10 seconds (health: starting)                                                                                                              realtime-dev.supabase-realtime
3391cbfbcb53   supabase/studio:20240101-8e4a094   "docker-entrypoint.s…"    7 minutes ago    Up 5 minutes (unhealthy)           3000/tcp                                                                                                   supabase-studio
a240a469f4d4   postgrest/postgrest:v12.0.1        "postgrest"               7 minutes ago    Restarting (1) 5 seconds ago                                                                                                                  supabase-rest
47d642cdf029   supabase/logflare:1.4.0            "sh run.sh"               7 minutes ago    Up 7 seconds (health: starting)    0.0.0.0:4000->4000/tcp, :::4000->4000/tcp                                                                  supabase-analytics
4c9e82a3c0f0   supabase/postgres:15.1.0.147       "docker-entrypoint.s…"    7 minutes ago    Up 5 minutes (healthy)             0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                                                                  supabase-db
3489ef40782c   darthsim/imgproxy:v3.8.0           "imgproxy"                7 minutes ago    Up 5 minutes (healthy)             8080/tcp                                                                                                   supabase-imgproxy
36c351ef306c   timberio/vector:0.28.1-alpine      "/usr/local/bin/vect…"    30 minutes ago   Up 4 minutes (healthy)                                                                                                                        supabase-vector
$ docker logs 146f42378a80
[...]
{"level":"fatal","msg":"running db migrations: Migrator: problem creating schema migrations: couldn't start a new transaction: could not create new transaction: failed to connect to `host=db user=supabase_auth_admin database=postgres`: failed SASL auth (FATAL: password authentication failed for user \"supabase_auth_admin\" (SQLSTATE 28P01))","time":"2024-02-08T17:34:50Z"}
{"level":"info","msg":"Go runtime metrics collection started","time":"2024-02-08T17:35:50Z"}
{"args":[0.064195059],"component":"pop","level":"info","msg":"%.4f seconds","time":"2024-02-08T17:35:50Z"}
{"level":"fatal","msg":"running db migrations: Migrator: problem creating schema migrations: couldn't start a new transaction: could not create new transaction: failed to connect to `host=db user=supabase_auth_admin database=postgres`: failed SASL auth (FATAL: password authentication failed for user \"supabase_auth_admin\" (SQLSTATE 28P01))","time":"2024-02-08T17:35:50Z"}

Docker version 25.0.2, build 29cf629 Debian GNU/Linux 12

To enable the analytics container you need to enable it in config.toml

[analytics]
enabled = true

Doc Link - supabase.com/docs/guides/cli/config#analytics-config

If that’s the case, shouldn’t that be noted in the Docker guide?