cloud-foundation-toolkit: Scorecard: New error generating violations

CFT Scorecard is generating a new error on the daily scheduled scan we run. There are no relevant changes to our infrastructure or usage of the CLI tool. Have repeated the pipeline and tweaked a little bit still get this result.

Could a change in GCP’s CAI output format be responsible?

Resource names are redacted.

cft scorecard --workers 2 --policy-path policy-library/ --dir-path output/cai-dump/ --output-path output/cft-report/
Generating CFT scorecard
Error: Fetching inventory from local directory: converting asset {"name":"//pubsub.googleapis.com/projects/{{PROJECT}}/topics/{{PUBSUB_TOPCI}}","asset_type":"pubsub.googleapis.com/Topic","iam_policy":{"etag":"{{ETAG}}","bindings":[{"role":"roles/pubsub.publisher","members":["serviceAccount:{{SA_NAME}@{{PROJECT}}.iam.gserviceaccount.com"]}]},"ancestors":["projects/{{PROJECT_NUM}}","folders/{{FOLDER_ID}}","folders/{{FOLDER_ID}}","folders/{{FOLDER_ID}}","organizations/{{ORD_ID}}"],"update_time":"2021-09-09T04:04:09.586830Z"} to proto: unmarshaling to proto: illegal base64 data at input byte 7

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 26 (21 by maintainers)

Most upvoted comments

The link depends on your platform:

# OS X
curl -o cft https://storage.googleapis.com/cft-cli/latest/cft-darwin-amd64
# Linux
curl -o cft https://storage.googleapis.com/cft-cli/latest/cft-linux-amd64
# executable
chmod +x cft

# Windows
curl -o cft.exe https://storage.googleapis.com/cft-cli/latest/cft-windows-amd64

We do hope to get the release pipeline working again soon. /cc @bharathkkb

morgante on Nov 11, 2021

Thanks @morgante , we point out Dockerfile to directly download the binary from the releases page… Could you please elaborate if the fork will have the binary to be pulled from ?

I suggest building a binary in GCS and pointing your Dockerfile there temporarily. Or if you open a PR with the change I can consider merging as a workaround.

https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/pull/1013

hussainak on Oct 27, 2021

A possible solution can be to remove the etag key altogether from the map. I have tested the following code and looks like it works until a better solution is found.

// protoViaJSON uses JSON as an intermediary serialization to convert a value into
// a protobuf message.
func protoViaJSON(from interface{}, to proto.Message) error {
	if m, ok := from.(map[string]interface{}); ok {
		if val, ok := m["iam_policy"]; ok {
			if m, ok := val.(map[string]interface{}); ok {
				delete(m, "etag")
			}
		}
	}
	jsn, err := json.Marshal(from)
	if err != nil {
		return errors.Wrap(err, "marshaling to json")
	}
	umar := &jsonpb.Unmarshaler{AllowUnknownFields: true}
	if err := umar.Unmarshal(strings.NewReader(string(jsn)), to); err != nil {
		return errors.Wrap(err, "unmarshaling to proto")
	}

	return nil
}

hussainak on Oct 25, 2021

@morgante We have seen this happening in our scheduled run as well. A possible solution could be https://stackoverflow.com/questions/40407819/golang-illegal-base64-data-at-input-byte-0

I took just a casual browse through the code and it looks like it is not doing the un-quote business.

hussainak on Oct 21, 2021