skaffold: skaffold dev does not show live tail logs after running for a while

Expected behavior

running skaffold dev should live tail logs after running for a while

Actual behavior

After multiple image rebuilds and changes to source code, I don’t get live logs when the server receives HTTP requests.

Information

  • Skaffold version: 0b8bd59
  • Operating system: macOS Catalina - version 10.15.5
  • Contents of skaffold.yaml:
apiVersion: skaffold/v2alpha1
kind: Config
build:
  local:
    push: false
  artifacts:
    - image: jon/kludge-client
      context: client
      sync:
        manual:
          - src: 'src/**/*.tsx'
            dest: '/usr/src/app'
          - src: 'src/**/**/*.tsx'
            dest: '/usr/src/app'
          - src: 'src/**/**/*.ts'
            dest: '/usr/src/app'
          - src: 'src/**/**/*.js'
            dest: '/usr/src/app'
          - src: 'src/**/**/*.scss'
            dest: '/usr/src/app'
          - src: '*.js'
            dest: '/usr/src/app'
      docker:
        dockerfile: Dockerfile
        buildArgs: 
          network: host
    - image: jon/kludge-api
      context: api
      sync:
        manual:
          - src: 'resources/views/*.php'
            dest: '/var/www/api'
          - src: 'app/Models/*.php'
            dest: '/var/www/api'
          - src: 'app/Http/Controllers/*.php'
            dest: '/var/www/api'
          - src: 'app/Http/Controllers/**/*.php'
            dest: '/var/www/api'
          - src: 'app/Abstracts/Repository/*.php'
            dest: '/var/www/api'
          - src: 'app/Repository/**/*.php'
            dest: '/var/www/api'
          - src: 'app/Services/*.php'
            dest: '/var/www/api'
          - src: 'app/Rules/*.php'
            dest: '/var/www/api'
          - src: 'app/Http/Middleware/*.php'
            dest: '/var/www/api'
          - src: 'app/Http/Requests/**/*.php'
            dest: '/var/www/api'
          - src: 'routes/*.php'
            dest: '/var/www/api'
          - src: 'database/**/*.php'
            dest: '/var/www/api'
          - src: 'config/*.php'
            dest: '/var/www/api'
          - src: 'app/Compiler/Languages/**/*'
            dest: '/var/www/api'
          - src: 'app/Examples/*.php'
            dest: '/var/www/api'
          - src: 'app/Examples/**/**/*'
            dest: '/var/www/api'
          - src: 'app/Providers/*.php'
            dest: '/var/www/api'
      docker:
        dockerfile: Dockerfile
deploy:
  kubectl:
    manifests:
      - k8-dev/client/deployment-manifest.yaml
      - k8-dev/api/deployment-manifest.yaml
      - k8-dev/client/service-manifest.yaml
      - k8-dev/api/service-manifest.yaml

Here’s the k8 deployment for the pod that does not live tail

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: kludge
spec:
  replicas: 1
  selector:
    matchLabels:
      name: api
  template:
    metadata:
      labels:
        name: api
      namespace: kludge
    spec:
      containers:
        - image: jon/kludge-api
          name: api
          ports:
            - containerPort: 80
          imagePullPolicy: Never
          env:
            - name: APP_NAME
              value: "api"
            - name: APP_ENV
              value: "local"
            - name: APP_DEBUG
              value: "true"
            - name: APP_LOG_LEVEL
              value: "debug"
            - name: APP_URL
              value: "http://kludge.info"
            - name: LOG_CHANNEL
              value: "stack"
            - name: DB_CONNECTION
              value: "mysql"
            - name: DB_HOST
              value: "mysql"
            - name: DB_DATABASE
              value: "kludgedb"
            - name: DB_USERNAME
              value: "kludgedbuser"
            - name: DB_PASSWORD
              value: "secret"

Steps to reproduce the behavior

Running skaffold dev but usually after 2 days or so live tail logs does not work, but the cluster is still running and everything else works.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 7
  • Comments: 18 (4 by maintainers)

Most upvoted comments

Same as @tonsV2 – Experiencing issue in my application – logs correctly stream after the first invocation of skaffold dev but logs do not stream after I make a change and Skaffold rebuilds and redeploys.

Skaffold 1.35.1

I am seeing log tailing fail sometimes because it looks like skaffold is not streaming log from the correct pod. I used the suggestion above to start skaffold with verbose logging (i.e. skaffold dev -v DEBUG --tail)

When I modify a source file, skaffold rebuilds the image, kills the old pod and starts a new one. I the detailed logs I see lines like this: INFO[0120] Streaming logs from pod: app-backend-5b44dd6db6-rl4tb container: app-backend subtask=-1 task=DevLoop

The issue is that sometimes this does not match the pod that is actually running.

$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
app-backend-86c77cfcd8-lsrxh   1/1     Running   0          9m2s

When the pod names match, tailing works. When they don’t match it doesn’t. Seems to be a race condition in starting the pod and skaffold setting up the log streaming. I’m not sure if this is the same issue identified by the OP or not.

My config: skaffold v1.31.0 minikube v1.23.1 hyperkit v0.20210107-9-gacbc2d kubectl v1.22.1 MacOS BigSur (x86)