podman: Can't connect to exposed port (database)

/kind bug

Description

Can’t connect to exposed port (database)

Steps to reproduce the issue:

sudo podman run --name some-mysql -e MYSQL_ROOT_PASSWORD=root -d -p 3309:3306 docker://mysql:8
sudo podman ps -a -> Port is exposed
Connect to the database: <host_ip>:3309, user:root, pass:root

Describe the results you received: java.net.ConnectException: Connection refused: connect (I use IntelliJ to connect to the db)

Describe the results you expected: Successfull connection

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.4
  podman version: 1.6.2
host:
  BuildahVersion: 1.11.3
  CgroupVersion: v1
  Conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.2, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "18.04"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 777121792
  MemTotal: 4133961728
  OCIRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 554102784
  SwapTotal: 595521536
  arch: amd64
  cpus: 2
  eventlogger: journald
  hostname: vm1-VirtualBox
  kernel: 5.0.0-32-generic
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: 'slirp4netns: /usr/bin/slirp4netns'
    Version: |-
      slirp4netns version 0.4.2
      commit: unknown
  uptime: 9h 4m 38.69s (Approximately 0.38 days)
registries:
  blocked: null
  insecure: null
  search: null
store:
  ConfigFile: /home/vm1/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: {}
  GraphRoot: /home/vm1/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 2
  RunRoot: /run/user/1000
  VolumePath: /home/vm1/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/bionic,now 1.6.2-1~ubuntu18.04~ppa1 amd64 [installed]

Additional environment details (AWS, VirtualBox, physical, etc.): VirtualBox

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 17 (10 by maintainers)

Most upvoted comments

Ok, I have this one nailed down. It is mariadb specific and not the fault of podman or cni. In one terminal, run top. In another, run the command as provided … and in another terminal, run a podman container log -f. What you will observe is this:

In the container, by way of top, the mysqld binary is chewing up CPU presumably to get setup. The maria db logs are as follows:

2020-01-27 21:52:54+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.4.11+maria~bionic started.
2020-01-27 21:52:55+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-01-27 21:52:55+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.4.11+maria~bionic started.
2020-01-27 21:52:55+00:00 [Note] [Entrypoint]: Initializing database files


PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
To do so, start the server, then issue the following commands:

'/usr/bin/mysqladmin' -u root password 'new-password'
'/usr/bin/mysqladmin' -u root -h  password 'new-password'

Alternatively you can run:
'/usr/bin/mysql_secure_installation'

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the MariaDB Knowledgebase at http://mariadb.com/kb or the
MySQL manual for more instructions.

Please report any problems at http://mariadb.org/jira

The latest information about MariaDB is available at http://mariadb.org/.
You can find additional information about the MySQL part at:
http://dev.mysql.com
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/

2020-01-27 21:52:58+00:00 [Note] [Entrypoint]: Database files initialized
2020-01-27 21:52:58+00:00 [Note] [Entrypoint]: Starting temporary server
2020-01-27 21:52:58+00:00 [Note] [Entrypoint]: Waiting for server startup
2020-01-27 21:52:58 0 [Note] mysqld (mysqld 10.4.11-MariaDB-1:10.4.11+maria~bionic) starting as process 117 ...
2020-01-27 21:52:58 0 [Note] InnoDB: Using Linux native AIO
2020-01-27 21:52:58 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-01-27 21:52:58 0 [Note] InnoDB: Uses event mutexes
2020-01-27 21:52:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-01-27 21:52:58 0 [Note] InnoDB: Number of pools: 1
2020-01-27 21:52:58 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-01-27 21:52:58 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2020-01-27 21:52:58 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2020-01-27 21:52:58 0 [Note] InnoDB: Completed initialization of buffer pool
2020-01-27 21:52:58 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-01-27 21:52:58 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-01-27 21:52:58 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-01-27 21:52:58 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-01-27 21:52:58 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-01-27 21:52:58 0 [Note] InnoDB: 10.4.11 started; log sequence number 60972; transaction id 21
2020-01-27 21:52:58 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-01-27 21:52:58 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-01-27 21:52:58 0 [Warning] 'user' entry 'root@4729f11f0714' ignored in --skip-name-resolve mode.
2020-01-27 21:52:58 0 [Warning] 'user' entry '@4729f11f0714' ignored in --skip-name-resolve mode.
2020-01-27 21:52:58 0 [Warning] 'proxies_priv' entry '@% root@4729f11f0714' ignored in --skip-name-resolve mode.
2020-01-27 21:52:58 0 [Note] InnoDB: Buffer pool(s) load completed at 200127 21:52:58
2020-01-27 21:52:58 0 [Note] Reading of all Master_info entries succeeded
2020-01-27 21:52:58 0 [Note] Added new Master_info '' to hash table
2020-01-27 21:52:58 0 [Note] mysqld: ready for connections.
Version: '10.4.11-MariaDB-1:10.4.11+maria~bionic'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
2020-01-27 21:52:59+00:00 [Note] [Entrypoint]: Temporary server started.

Then, as observed, after a short time, the mysqld drops in cpu, and the log furthers:


2020-01-27 21:53:38 10 [Warning] 'proxies_priv' entry '@% root@4729f11f0714' ignored in --skip-name-resolve mode.

2020-01-27 21:53:38+00:00 [Note] [Entrypoint]: Stopping temporary server
2020-01-27 21:53:38 0 [Note] mysqld (initiated by: root[root] @ localhost []): Normal shutdown
2020-01-27 21:53:38 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-01-27 21:53:38 0 [Note] InnoDB: FTS optimize thread exiting.
2020-01-27 21:53:38 0 [Note] InnoDB: Starting shutdown...
2020-01-27 21:53:38 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2020-01-27 21:53:38 0 [Note] InnoDB: Buffer pool(s) dump completed at 200127 21:53:38
2020-01-27 21:53:40 0 [Note] InnoDB: Shutdown completed; log sequence number 19887082; transaction id 14632
2020-01-27 21:53:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-01-27 21:53:40 0 [Note] mysqld: Shutdown complete

2020-01-27 21:53:40+00:00 [Note] [Entrypoint]: Temporary server stopped

2020-01-27 21:53:40+00:00 [Note] [Entrypoint]: MySQL init process done. Ready for start up.

2020-01-27 21:53:40 0 [Note] mysqld (mysqld 10.4.11-MariaDB-1:10.4.11+maria~bionic) starting as process 1 ...
2020-01-27 21:53:40 0 [Note] InnoDB: Using Linux native AIO
2020-01-27 21:53:40 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-01-27 21:53:40 0 [Note] InnoDB: Uses event mutexes
2020-01-27 21:53:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-01-27 21:53:40 0 [Note] InnoDB: Number of pools: 1
2020-01-27 21:53:40 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-01-27 21:53:40 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2020-01-27 21:53:40 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2020-01-27 21:53:40 0 [Note] InnoDB: Completed initialization of buffer pool
2020-01-27 21:53:40 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-01-27 21:53:40 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-01-27 21:53:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-01-27 21:53:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-01-27 21:53:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-01-27 21:53:40 0 [Note] InnoDB: Waiting for purge to start
2020-01-27 21:53:40 0 [Note] InnoDB: 10.4.11 started; log sequence number 19887082; transaction id 14632
2020-01-27 21:53:40 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-01-27 21:53:40 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-01-27 21:53:40 0 [Note] Server socket created on IP: '::'.
2020-01-27 21:53:40 0 [Warning] 'proxies_priv' entry '@% root@4729f11f0714' ignored in --skip-name-resolve mode.
2020-01-27 21:53:40 0 [Note] InnoDB: Buffer pool(s) load completed at 200127 21:53:40
2020-01-27 21:53:40 0 [Note] Reading of all Master_info entries succeeded
2020-01-27 21:53:40 0 [Note] Added new Master_info '' to hash table
2020-01-27 21:53:40 0 [Note] mysqld: ready for connections.
Version: '10.4.11-MariaDB-1:10.4.11+maria~bionic'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

Once the “ready” message appears, the port works. So to put this to bed, all of the container and network plumbing are in fact working. We are simply waiting for maria db to complete its initial setup and bind to the port.

baude on Jan 27, 2020