skaffold: High CPU usage on Mac OS X

Actual behavior

skaffold process generates about 100% sustained cpu usage when it is started with “skaffold dev” command. The project itself is quite small (4 deployments of third-party services and 1 locally built docker image), and the high cpu usage is present even without any local file changes.

VBoxHeadless process also has a sustained 150-200% CPU usage in Activity Monitor.

Information

Skaffold version: v0.5.0
Operating system: Mac OS X 10.12.6
Content of skaffold.yaml:

apiVersion: skaffold/v1alpha2
kind: Config
build:
  artifacts:
  - imageName: my-tools-image
    workspace: ./tools
    docker: {}
  local: {}
deploy:
  kubectl:
    manifests:
    - ./kubernetes/*

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 3
Comments: 28 (21 by maintainers)

Commits related to this issue

Fix issue with 100% CPU usage in logs.go. Resolves: https://github.com/GoogleContainerTools/skaffold/issues/531 See also: https://github.com/kubernetes/client-go/issues/12 There is an issue in which... — committed to d11wtq/skaffold by d11wtq 6 years ago
Fix issue with 100% CPU usage in logs.go. Resolves: https://github.com/GoogleContainerTools/skaffold/issues/531 See also: https://github.com/kubernetes/client-go/issues/12 There is an issue in which... — committed to d11wtq/skaffold by d11wtq 6 years ago

Most upvoted comments

I’ve instrumented my local build of skaffold and found that this line:

https://github.com/GoogleContainerTools/skaffold/blob/762c3ada56d758913e15dfaadfa73b0e68328e8b/pkg/skaffold/kubernetes/log.go#L74

Seems to suddenly return a nil/empty Event repeatedly. I have been able to trigger it by switching from one internet connection to another. I think the watch channel is closed and we’re stuck in an infinite loop because of it.

I’m not certain this is the source of the problem, but I’m experimenting locally with:

	case evt, ok := <-watcher.ResultChan():
		if !ok {
			// Try reopening
			logrus.Infof("Log aggregator watch has stopped, respawning it.")
			a.Start(ctx, client)
			break
		}

ajbouh on May 16, 2018

@davidfernandezm glad to hear that fixed your issue! we have some brief documentation on the trigger types at https://skaffold.dev/docs/workflows/dev/ under “File Watcher and Watch Modes”, but this could definitely be improved. I don’t have time to do this right now but if you’d like to become a contributor feel free and throw some improvements together 😃

FYI it’s not manual that goes crazy, it’s fsnotify. especially on OS X, when a project has lots and lots of files, the CPU usage skyrockets. the polling trigger simply computes changes in your workspace on a time interval, so it doesn’t actually have to “monitor” anything. not as quick, but much less overhead.

nkubala on Jun 24, 2020