compose: New TUI spams lines when pulling multiple images causing you to lose your terminal history

I don’t think it just build, although multiple builds will definitely be adding to the output. The example video shows that pulling (if the compose-file has many images) can also be problematic if there’s more pulls than fit on the screen.

Doing a quick test with a stack that uses a postgres image (which has various layers);

services:
    foo:
        image: postgres

Produces this output (ignore the error at the end, it’s just for illustration):

docker compose up
[+] Running 14/14
 ⠿ foo Pulled                                          14.6s
   ⠿ e5ae68f74026 Pull complete                         5.6s
   ⠿ 7b8fcc7e1ad0 Pull complete                         6.0s
   ⠿ 7527d03e2f77 Pull complete                         6.1s
   ⠿ 80e55689f4d0 Pull complete                         6.2s
   ⠿ 8a79eb6d69c9 Pull complete                         6.7s
   ⠿ 397705f2d093 Pull complete                         6.8s
   ⠿ de36ec4eb0a5 Pull complete                         6.9s
   ⠿ 08d878a022c1 Pull complete                         7.0s
   ⠿ 7677029670ff Pull complete                        11.4s
   ⠿ 1d24b3d9557e Pull complete                        11.5s
   ⠿ e085b018338c Pull complete                        11.5s
   ⠿ 063b09ff12e9 Pull complete                        11.6s
   ⠿ a39fee215a44 Pull complete                        11.7s
[+] Running 2/2
 ⠿ Network compose-progress_default  Created            0.1s
 ⠿ Container compose-progress-foo-1  Created            1.4s
Attaching to compose-progress-foo-1
compose-progress-foo-1  | Error: Database is uninitialized and superuser password is not specified.
compose-progress-foo-1  |        You must specify POSTGRES_PASSWORD to a non-empty value for the
compose-progress-foo-1  |        superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
compose-progress-foo-1  |
compose-progress-foo-1  |        You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
compose-progress-foo-1  |        connections without a password. This is *not* recommended.
compose-progress-foo-1  |
compose-progress-foo-1  |        See PostgreSQL documentation about "trust":
compose-progress-foo-1  |        https://www.postgresql.org/docs/current/auth-trust.html
compose-progress-foo-1 exited with code 1

While the pull is running, Downloading and Extracting progress bars are shown for each layer:

   ⠼ e5ae68f74026 Extracting      [==================================================>]  31.37MB/31.37MB    4.4s
   ⠼ 7b8fcc7e1ad0 Download complete                                                                         4.4s
   ⠼ 7527d03e2f77 Download complete                                                                         4.4s
   ⠼ 80e55689f4d0 Download complete                                                                         4.4s
   ⠼ 8a79eb6d69c9 Download complete                                                                         4.4s
   ⠼ 397705f2d093 Download complete                                                                         4.4s
   ⠼ de36ec4eb0a5 Download complete                                                                         4.4s
   ⠼ 08d878a022c1 Download complete                                                                         4.4s
   ⠼ 7677029670ff Downloading     [=============================>                     ]  53.55MB/91.23MB    4.4s
   ⠼ 1d24b3d9557e Download complete                                                                         4.4s

Based on that output, some ideas:

Remove the pull complete lines after they have completed / after the image has fully downloaded

Collapse the section, and replace with a summary.

⠿ foo Pulled                                          14.6s

Perhaps the summary should have more details than just “pulled”; users may be interested to know what image (and tag, digest?) was pulled, and how large it was in total (?), as well as the architecture that was pulled (?) haven’t given that a lot of thinking yet;

⠿ foo Pulled postgres:latest (digest sha256:xxx) linux/amd64, 12345MB                  14.6s

Use a single progress (or counter) per image during pull

Perhaps depending on the amount of available space or the number of services / pulls, perhaps it’s possible to group the progress bars. The size would show the total size / sum of all layers (I’m not sure if this works if layers are already downloaded before though);

postgres:latest  Downloading  3/13   [=============================>                     ]  53.55MB/91.23MB    4.4s
postgres:latest  Extracting   2/13   [==================================================>]  31.37MB/31.37MB    4.4s

I’m not sure line-height is the ultimate culprit here. I’m seeing hectic-as-heck output even with single service build in a full height terminal window on a laptop with a built-in 1600x1200 display with scaling disabled. It started happening with the release of v2.17. With v2.16, output with the same services was as-expected.

I’m at least still seeing this on Docker Compose version v2.17.3 with docker version 23.0.5

@williamdes this is Docker Desktop version https://github.com/docker/compose/issues/10256 would indeed be a better place as an open issue

Please open a fresh new issue. The main question to address is “how to display status for N services when N > terminal height?”. Any suggestion is welcome

Here’s my use case to reproduce the problem.

• Docker 20.10.17
• Docker Compose v2.6.0
• This compose.yaml:

x-service: &default-service
  image: nginx:latest
  command: >
    bash -c "sleep 10 && nginx -g 'daemon off;'"
  healthcheck:
    test: service nginx status || exit 1
    interval: 2s
    timeout: 1s
    retries: 200
    start_period: 600s

services:
  svc_a:
    <<: *default-service
  svc_b:
    <<: *default-service
    depends_on:
      svc_a:
        condition: service_healthy
  svc_c:
    <<: *default-service
    depends_on:
      svc_b:
        condition: service_healthy
  svc_d:
    <<: *default-service
    depends_on:
      svc_c:
        condition: service_healthy
  svc_e:
    <<: *default-service
    depends_on:
      svc_d:
        condition: service_healthy
  svc_f:
    <<: *default-service
    depends_on:
      svc_e:
        condition: service_healthy
  svc_g:
    <<: *default-service
    depends_on:
      svc_f:
        condition: service_healthy
  svc_h:
    <<: *default-service
    depends_on:
      svc_g:
        condition: service_healthy
  svc_i:
    <<: *default-service
    depends_on:
      svc_h:
        condition: service_healthy
  svc_j:
    <<: *default-service
    depends_on:
      svc_i:
        condition: service_healthy
  svc_k:
    <<: *default-service
    depends_on:
      svc_j:
        condition: service_healthy
  svc_l:
    <<: *default-service
    depends_on:
      svc_k:
        condition: service_healthy

• Make terminal 11 lines high
• docker compose up -d
• Notice terminal scroll

Have a large number of containers within a docker-compose file. Same problem:

1. It spams the output buffer by jumping around like crazy
2. it flashes as fireworks like crazy while it’s happening
3. The ultimate output simply says “[container] Pulled … X.Xs” but provides no indications of which were up to date in the first place and which were actively pulled
4. The ultimately output is a mix of 'containername Pulled", “XXXX already exists” and “xxxx pull complete” with some items showing the breakout of parts and others not.

As pointed out, it’s very easy to exceed a typical terminal height in this case.

Possible ideas:

1. Do the docker compose pull sequentially or in batches, which reduces the number of pulls at any given time and thus concludes some items before the next few are started
2. Spare the output of any that are “up to date” to the end. A quick test of whether the image is up to date, and if it is, output nothing until after all are updated and then output the “XXXX is up to date”
3. A better terminal implementation

Boy is this a mess on a large compose pull. How about an option to just suppress individual layer progress? It doesn’t take much to overshoot the line height of most terminals with a reasonable compose file. Otherwise you have to either watch the fireworks, or resort to something like:

docker compose ps --services |xargs -n 1 docker compose pull

@ericslandry I can reproduce using your config.

I think the reason that v2.6.0 fixed it for me was because it made it so only one service would show the pull progress at the same time. After that change, my terminal was never small enough to not be able to fit it all on one screen without scrolling.

Your config works fine for me if my terminal is tall enough to fit it all (about 23 lines), but if I do 11 lines like you said, I see the issue happen again.

@thaJeztah buildkit indeed consider the build requests we schedule in parallel for services to be fully independent, and as such won’t do any magic on concurrent pulls for same layers.

@ericslandry for your information, this output is fully managed by buildx.