starboard: Support private container registries for Pods which do not specifying image pull secrets

What steps did you take and what happened:

Followed install instructions. Tried to run scan using below on command.

$ kubectl starboard  find vulnerabilities deployment/XXX-qa-web -n XXXX-qa -v 3
I0814 12:52:38.530373   28112 scanner.go:56] Getting Pod template for workload: {Deployment XXXX-qa-web XXXX-qa}
I0814 12:52:40.842053   28112 scanner.go:71] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:true}
I0814 12:52:41.183767   28112 runner.go:79] Running task and waiting forever
I0814 12:52:41.183840   28112 runnable_job.go:47] Creating runnable job: starboard/b75ba5e8-82c9-4915-ad35-4b35c37987ab
I0814 12:52:41.535929   28112 reflector.go:207] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/client-go@v0.19.0-alpha.3/tools/cache/reflector.go:156
I0814 12:52:41.535978   28112 reflector.go:243] Listing and watching *v1.Job from pkg/mod/k8s.io/client-go@v0.19.0-alpha.3/tools/cache/reflector.go:156
I0814 12:52:50.479003   28112 runnable_job.go:73] Stopping runnable job on task failure with status: Failed
I0814 12:52:50.479115   28112 runner.go:83] Stopping runner on task completion with error: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit
E0814 12:52:52.784556   28112 manager.go:177] Container default terminated with Error: 2020-08-14T07:22:49.629Z FATAL   unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:
        * unable to inspect the image (us.gcr.io/XXXX-1/XXXX:116579-23d73da-release-2019-10): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
        * GET https://us.gcr.io/v2/token?scope=repository%3xxxl-1%2FXXXX%3Apull&service=us.gcr.io: UNKNOWN: Unable to parse json key.


error: running scan job: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit

What did you expect to happen: Scan should completed without error

Anything else you would like to add: trivy support gcr. But I am not to able find way to pass custom ENV to trivy using starboard.

Environment:

Starboard version (use starboard version): Starboard Version: {Version:0.2.6 Commit:d43faefc56021ae55d4574054ce7de13175ca206 Date:2020-07-09T20:30:45Z}
Kubernetes version (use kubectl version): client:v1.17.10, server: v1.17.2
OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Ubuntu 18.04

About this issue

Original URL
State: open
Created 4 years ago
Reactions: 4
Comments: 20 (11 by maintainers)

Most upvoted comments

I found what was causing this problem. Essentially you can create image pull secret which is actually concatenation of registry server (rg.pl-waw.scw.cloud) and repository name (private). If that was the case Starboard could not parse registry server properly and match it with registry server taken from image reference. BTW, docker login strips down the repository part when it updates ~/.docker/config.json file

This was causing problem in v0.9.0

$ kubectl create secret docker-registry regcred \
  --docker-server=rg.pl-waw.scw.cloud/private \
  --docker-username=nologin \
  --docker-password=$SCW_SECRET_TOKEN --dry-run -o yaml | kubectl apply -f -

This would work in v0.9.0

$ kubectl create secret docker-registry regcred \
  --docker-server=rg.pl-waw.scw.cloud \
  --docker-username=nologin \
  --docker-password=$SCW_SECRET_TOKEN --dry-run -o yaml | kubectl apply -f -

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      imagePullSecrets:
      - name: regcred
      containers:
      - image: rg.pl-waw.scw.cloud/private/nginx:1.16
        imagePullPolicy: Always
        name: nginx
EOF

This particular problem is fixed in #379. However the original issue was created to support private registries without specifying image pull secrets on a pod. For example, by configuring nodes to authenticate with private registry.

danielpacak on Feb 10, 2021

Hey @mathieudebrito I can reproduce this error so it seems to be regression or a corner case that’s not handled properly by Starboard (both operator and CLI as they share this functions). I’ll track it down and try to fix it in the upcoming bug fix release v0.9.1.

Note: For some reason we cannot map registry server to container that refers to private image around these lines https://github.com/aquasecurity/starboard/blob/main/pkg/kube/secrets.go#L49

danielpacak on Feb 9, 2021

@danielpacak awesome thank you ! Indeed, I understand the issue is not the same, I should have created a new one, sorry for the inconvenience 😦

Edit : Just tested the 0.9.1, it fixes well the issue I was having. I can now see the vulnerability reports, thank you so much for you so quick fix !

mathieudebrito on Feb 10, 2021