skaffold: Recently started getting "Too many open files..." spinning up cluster on Apple M1

Expected behavior

That my project would be deployed and spin-up without any issues.

Actual behavior

It is safe to say, I think, that this is isolated to Apple Silicon.

I get all of these “Too many open files…” warnings, the deployment slows down to snails crawl, other messages related to “failed to port forward <service> is taken, retrying…” and then eventually it acts like it starts trying to rebuild the images again. Basically it is a mess…

Interestingly you can access the services, but none of the communication between them is working.

Without -vdebug:

https://user-images.githubusercontent.com/59094233/123156973-d2762e80-d41e-11eb-8e55-f91cba88a532.mov

With -vdebug:

https://user-images.githubusercontent.com/59094233/123155811-68a95500-d41d-11eb-8b5f-b680ba2d7f76.mov

I’ve only tested it is still working on AMD and Intel Linux and Windows (WSL2) machines. Works perfectly fine, the application spins up quickly and without issues.

I don’t have an Intel Mac around me at this point in time to test it, but I’d imagine it works fine there as well. At least that is what I was working on for two years up until about a month ago.

Pretty sure this was working perfectly about a month ago on an M1 Mac that has since been reformatted… but I could be mistaken. I reinstalled everything following the exact same steps, however.

Information

• Skaffold version: 1.26.1
• Operating system: macOS 11.4
• Installed via: skaffold.dev, but did also try brew with the same results… using the darwin-arm64 version
• Contents of skaffold.yaml:

You can comment out the individual artifacts, or the manifests, to just deploy one service at a time. The results are the same… a ton of errors about “Too many open files…” and eventually it becomes unresponsive.

apiVersion: skaffold/v2beta12
kind: Config
build:
  artifacts:
  - image: admin
    context: admin
    sync:
      manual:
      - src: "src/**/*.php"
        dest: .
      - src: "conf/**/*.conf"
        dest: .
      - src: "src/Templates/**/*.tbs"
        dest: .
      - src: "src/css/**/*.css"
        dest: .
      - src: "src/js/**/*.js"
        dest: .
    docker:
      dockerfile: Dockerfile
  - image: admin-v2
    context: admin-v2
    sync:
      manual:
      - src: 'src/**/*.ts'
        dest: .
      - src: 'src/**/*.tsx'
        dest: .
      - src: '**/*.json'
        dest: .
      - src: 'public/**/*.html'
        dest: .
      - src: 'src/assets/sass/**/*.scss'
        dest: .
      - src: 'src/build/**/*.js'
        dest: .
    docker:
      dockerfile: Dockerfile.dev
  - image: api
    context: api
    sync:
      manual:
      - src: "**/*.py"
        dest: .
    docker:
      dockerfile: Dockerfile.dev
  - image: api-v2
    context: api-v2
    sync:
      manual:
      - src: "**/*.py"
        dest: .
    docker:
      dockerfile: Dockerfile.dev
  - image: client
    context: client
    sync:
      manual:
      - src: 'src/**/*.js'
        dest: .
      - src: 'src/**/*.jsx'
        dest: .
      - src: '**/*.json'
        dest: .
      - src: 'public/**/*.html'
        dest: .
      - src: 'src/assets/sass/**/*.scss'
        dest: .
      - src: 'src/build/**/*.js'
        dest: .
    docker:
      dockerfile: Dockerfile.dev
  - image: postgres
    context: postgres
    sync:
      manual:
      - src: "**/*.sql"
        dest: .
    docker:
      dockerfile: Dockerfile.dev
  local:
    push: false
deploy:
  kubectl:
    manifests:
      - k8s/dev/ingress.yaml
      - k8s/dev/postgres.yaml
      - k8s/dev/client.yaml
      - k8s/dev/admin.yaml
      - k8s/dev/admin-v2.yaml
      - k8s/dev/api.yaml
      - k8s/dev/api-v2.yaml
    defaultNamespace: dev

Steps to reproduce the behavior

1. Unfortunately I can’t post the project due to it being proprietary. I’d suspect anyone with not even a moderate size or complex project will also run into the issue.
2. skaffold dev --port-forward -n dev

These are other dependencies:

• Docker Desktop for Apple Silicon 3.4.0 (65384)
• Kubernetes v1.21.1 (enabled via Docker Desktop)
• Minikube v1.21.0 (arm64)
• Skaffold v1.26.1 (arm64)
• Homebrew 3.2.0

Pretty sure can’t install anything but arm64 for these and haven’t tried the non-arm versions.

This is what I’ve tried:

• Several reformats of the system.
• Using virtualization.framework and hypervisor.framework.
• Installing all in terminal under as “Universal” or “Apple Silicon”.
• Running with VS Code as “Universal” or “Apple Silicon”.
• Running with VS Code as “Intel”.
• Installing all in VS Code as “Intel” and running in VS Code as “Intel”.
• 8GB and 16GB M1 models.
• 4 CPU cores and 6GB RAM in Docker Desktop.
• Minikube has 2 CPU cores and 4GB RAM.
• Using Docker driver in Minikube.