kubernetes: "invalid" base64 value results in weird oom_score_adj error

On GKE, running version 1.3.4:

We had an AWS secret that we pushed in a secret, but forgot to base64 encode. By chance, the secret was valid base64, and was decoded on the server to some binary garbage.

The pod wouldn’t start, returning the error:

  22s    22s    1    {kubelet gke-xxx-d8d4ff70-91s0}        Warning    FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "secor" with RunContainerError: "runContainer: Error response from daemon: rpc error: code = 2 desc = \"oci runtime error: write /proc/12499/oom_score_adj: invalid argument\""```

Although this was human error, the returned event/error isn’t very helpful to debug the issue.