woodpecker: k8s agent fails to tail step logs
Component
agent
Describe the bug
For some of our repos, trivy scan step, triggers the agent to log this error. I noticed this not just with trivy but also with megalinter.
{ "level": "error", "error": "rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8", "time": "2023-08-20T07:18:46Z", "message": "grpc error: log(): code: Internal: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8" }
Trivy command executed as part of the step.
trivy fs \ --exit-code 1 \ --skip-dirs vendor,node_modules \ --report summary \ --severity CRITICAL \ --scanners vuln \ --ignore-unfixed \ --quiet \ .
System Info
{"source":"https://github.com/woodpecker-ci/woodpecker","version":"next-0c282e86e8"}
Additional context
No response
Validations
- Read the Contributing Guidelines.
- Read the docs.
- Check that there isn’t already an issue that reports the same bug to avoid creating a duplicate.
- Checked that the bug isn’t fixed in the
nextversion already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use] - Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 3
- Comments: 16 (12 by maintainers)
Thanks for the help.
Just had the feeling as we are getting the logs base64 encoded from the api 🤔
https://github.com/woodpecker-ci/woodpecker/blob/d0057736f164a8a12e671ff3c39ac2c33b80a7f6/web/src/components/repo/pipeline/PipelineLog.vue#L298
https://github.com/woodpecker-ci/woodpecker/blob/d0057736f164a8a12e671ff3c39ac2c33b80a7f6/web/src/components/repo/pipeline/PipelineLog.vue#L303
Anyways I guess the solution should be quite simple by base64 encoding in the agent code directly
https://github.com/woodpecker-ci/woodpecker/pull/3601#issuecomment-2056975576
Indeed, WP doesn’t print them.
I’ve reproduced this with pipeline:
WP output:
wp-01hwa0ya0qqn1exr68j5ftj5tq.log
Pod was in
succeededstate before pipeline cancellation.But I didn’t notice high CPU usage (at least after I cancelled pipeline):
Tested on Woodpecker
2.3.0in Kubernetes. woodpecker-server.log woodpecker-agent.logAs you can see, Agent logs error even after pipeline was cancelled.
Now my pipeline is in status
killed, but the step stays in therunning.With WOODPECKER_DEBUG_PRETTY=true and WOODPECKER_LOG_LEVEL=trace i got: