skaffold: skaffold dev does not show live tail logs after running for a while
Expected behavior
running skaffold dev should live tail logs after running for a while
Actual behavior
After multiple image rebuilds and changes to source code, I don’t get live logs when the server receives HTTP requests.
Information
- Skaffold version: 0b8bd59
- Operating system: macOS Catalina - version 10.15.5
- Contents of skaffold.yaml:
apiVersion: skaffold/v2alpha1
kind: Config
build:
local:
push: false
artifacts:
- image: jon/kludge-client
context: client
sync:
manual:
- src: 'src/**/*.tsx'
dest: '/usr/src/app'
- src: 'src/**/**/*.tsx'
dest: '/usr/src/app'
- src: 'src/**/**/*.ts'
dest: '/usr/src/app'
- src: 'src/**/**/*.js'
dest: '/usr/src/app'
- src: 'src/**/**/*.scss'
dest: '/usr/src/app'
- src: '*.js'
dest: '/usr/src/app'
docker:
dockerfile: Dockerfile
buildArgs:
network: host
- image: jon/kludge-api
context: api
sync:
manual:
- src: 'resources/views/*.php'
dest: '/var/www/api'
- src: 'app/Models/*.php'
dest: '/var/www/api'
- src: 'app/Http/Controllers/*.php'
dest: '/var/www/api'
- src: 'app/Http/Controllers/**/*.php'
dest: '/var/www/api'
- src: 'app/Abstracts/Repository/*.php'
dest: '/var/www/api'
- src: 'app/Repository/**/*.php'
dest: '/var/www/api'
- src: 'app/Services/*.php'
dest: '/var/www/api'
- src: 'app/Rules/*.php'
dest: '/var/www/api'
- src: 'app/Http/Middleware/*.php'
dest: '/var/www/api'
- src: 'app/Http/Requests/**/*.php'
dest: '/var/www/api'
- src: 'routes/*.php'
dest: '/var/www/api'
- src: 'database/**/*.php'
dest: '/var/www/api'
- src: 'config/*.php'
dest: '/var/www/api'
- src: 'app/Compiler/Languages/**/*'
dest: '/var/www/api'
- src: 'app/Examples/*.php'
dest: '/var/www/api'
- src: 'app/Examples/**/**/*'
dest: '/var/www/api'
- src: 'app/Providers/*.php'
dest: '/var/www/api'
docker:
dockerfile: Dockerfile
deploy:
kubectl:
manifests:
- k8-dev/client/deployment-manifest.yaml
- k8-dev/api/deployment-manifest.yaml
- k8-dev/client/service-manifest.yaml
- k8-dev/api/service-manifest.yaml
Here’s the k8 deployment for the pod that does not live tail
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
namespace: kludge
spec:
replicas: 1
selector:
matchLabels:
name: api
template:
metadata:
labels:
name: api
namespace: kludge
spec:
containers:
- image: jon/kludge-api
name: api
ports:
- containerPort: 80
imagePullPolicy: Never
env:
- name: APP_NAME
value: "api"
- name: APP_ENV
value: "local"
- name: APP_DEBUG
value: "true"
- name: APP_LOG_LEVEL
value: "debug"
- name: APP_URL
value: "http://kludge.info"
- name: LOG_CHANNEL
value: "stack"
- name: DB_CONNECTION
value: "mysql"
- name: DB_HOST
value: "mysql"
- name: DB_DATABASE
value: "kludgedb"
- name: DB_USERNAME
value: "kludgedbuser"
- name: DB_PASSWORD
value: "secret"
Steps to reproduce the behavior
Running skaffold dev but usually after 2 days or so live tail logs does not work, but the cluster is still running and everything else works.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 7
- Comments: 18 (4 by maintainers)
Same as @tonsV2 – Experiencing issue in my application – logs correctly stream after the first invocation of skaffold dev but logs do not stream after I make a change and Skaffold rebuilds and redeploys.
Skaffold 1.35.1
I am seeing log tailing fail sometimes because it looks like skaffold is not streaming log from the correct pod. I used the suggestion above to start skaffold with verbose logging (i.e.
skaffold dev -v DEBUG --tail)When I modify a source file, skaffold rebuilds the image, kills the old pod and starts a new one. I the detailed logs I see lines like this:
INFO[0120] Streaming logs from pod: app-backend-5b44dd6db6-rl4tb container: app-backend subtask=-1 task=DevLoopThe issue is that sometimes this does not match the pod that is actually running.
When the pod names match, tailing works. When they don’t match it doesn’t. Seems to be a race condition in starting the pod and skaffold setting up the log streaming. I’m not sure if this is the same issue identified by the OP or not.
My config: skaffold v1.31.0 minikube v1.23.1 hyperkit v0.20210107-9-gacbc2d kubectl v1.22.1 MacOS BigSur (x86)