woodpecker: k8s agent fails to tail step logs

Component

agent

Describe the bug

For some of our repos, trivy scan step, triggers the agent to log this error. I noticed this not just with trivy but also with megalinter.

{ "level": "error", "error": "rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8", "time": "2023-08-20T07:18:46Z", "message": "grpc error: log(): code: Internal: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8" }

Trivy command executed as part of the step.

trivy fs \ --exit-code 1 \ --skip-dirs vendor,node_modules \ --report summary \ --severity CRITICAL \ --scanners vuln \ --ignore-unfixed \ --quiet \ .

System Info

{"source":"https://github.com/woodpecker-ci/woodpecker","version":"next-0c282e86e8"}

Additional context

No response

Validations

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 3
  • Comments: 16 (12 by maintainers)

Commits related to this issue

Most upvoted comments

Thanks for the help.

Just had the feeling as we are getting the logs base64 encoded from the api 🤔

https://github.com/woodpecker-ci/woodpecker/blob/d0057736f164a8a12e671ff3c39ac2c33b80a7f6/web/src/components/repo/pipeline/PipelineLog.vue#L298

https://github.com/woodpecker-ci/woodpecker/blob/d0057736f164a8a12e671ff3c39ac2c33b80a7f6/web/src/components/repo/pipeline/PipelineLog.vue#L303

Anyways I guess the solution should be quite simple by base64 encoding in the agent code directly

https://github.com/woodpecker-ci/woodpecker/pull/3601#issuecomment-2056975576

I haven’t noticed any missing logs on our builds. However I think the true test would be to run a build that outputs non UTF-8 characters and see how Woodpecker behaves. Given the issue, I’d expect Woodpecker to not print them

Indeed, WP doesn’t print them.

I’ve reproduced this with pipeline:

skip_clone: true
steps:
  logs:
    image: alpine
    commands:
      - apk add openssl
      - openssl rand 1000
      - echo Done

WP output:

+ apk add openssl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/3) Upgrading libcrypto3 (3.1.4-r5 -> 3.1.4-r6)
(2/3) Upgrading libssl3 (3.1.4-r5 -> 3.1.4-r6)
(3/3) Installing openssl (3.1.4-r6)
Executing busybox-1.36.1-r15.trigger
OK: 8 MiB in 16 packages
+ openssl rand 1000

wp-01hwa0ya0qqn1exr68j5ftj5tq.log

Pod was in succeeded state before pipeline cancellation.

But I didn’t notice high CPU usage (at least after I cancelled pipeline):

> kubectl top pods -n test-scm-cicd
NAME                                 CPU(cores)   MEMORY(bytes)
gitea-0                              3m           125Mi
woodpecker-agent-d9b864c76-lqdgj     1m           10Mi
woodpecker-server-75c9794774-nglv7   3m           29Mi

Tested on Woodpecker 2.3.0 in Kubernetes. woodpecker-server.log woodpecker-agent.log

As you can see, Agent logs error even after pipeline was cancelled.


Now my pipeline is in status killed, but the step stays in the running.

With WOODPECKER_DEBUG_PRETTY=true and WOODPECKER_LOG_LEVEL=trace i got:

1:07PM TRC src/pipeline/rpc/log_entry.go:82 > grpc write line: gpg: key 4F9BDB78DB596386: public key "Vincent Siveton <vsiveton@apache.org>" imported
gpg: key B95BBD3FA43C4492: 22 signatures not checked due to missing keys
gpg: key B95BBD3FA43C4492: public key "Carlos Sanchez <carlos.sanchez@mergere.com>" imported
gpg: key E181870DC625BAFB: public key "Maria Odea Ching <oching@apache.org>" imported
gpg: key D2A765937BA507E8: public key "Mauro Talevi <mauro@apache.org>" imported
gpg: key C92C5FEC70161C62: public key "Herve Boutemy <hboutemy@apache.org>" imported
gpg: key 873A8E86B4372146: public key "Olivier Lamy <olamy@apache.org>" imported
gpg: key 8F0FABD30F353251: public key "Dan Fabulich <dfabulich@apache.org>" imported
gpg: key 0B7B8BED064C851C: public key "Rapha�l Pi�roni <rafale@apache.org>" imported
gpg: key C2D879C8B0874707: public key "Milos Kleint <mkleint@apache.org>" imported
gpg: key 8FE7CB51365A46EF: public key "Mark Hobson <markh@apache.org>" imported
gpg: key A3F9CCC081C4177D: public key "Oleg Gusakov <ogusakov@apache.org>" imported
gpg: key 30E6F80434A72A7F: public key "John Dennis Casey <jdcasey@commonjava.org>" imported
gpg: key E8579C183571506F: public key "Rapha�l Pi�roni <rafale@apache.org>" imported
gpg: key A8AD2A7E9A25CE21: public key "Barrie Treloar <baerrach@apache.org>" imported
gpg: key 3B58205B9D7013A9: public key "Dan T. Tran <dantran@apache.org>" imported
 step-uuid=01HQ5WD8SBBCTMPE81QTQJS272
1:07PM ERR src/agent/rpc/client_grpc.go:335 > grpc error: log(): code: Internal error="rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8"