kube2iam: First run during deployment produce 404 error

we wrote a shell script that runs as part of our run command:

attempt=0
exitCode=0
response=0
max_attempts=${MAX_ATTEMPTS:-30}
jitter=${JITTER:-5}

while [[ $attempt -lt $max_attempts ]] ; do
  # Jitter requests to break resource contention
  timeout=$((1 + RANDOM % jitter))

  # 169.254.169.254/latest/meta-data is reserved by AWS to access the Metadata API to retrieve IAM credentials.
  #   https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
  # This curl command is hitting the Metadata API which routes through Kube2iam via IP Tables
  #   https://github.com/jtblin/kube2iam#iptables
  # If the requests return a 200OK, then it's implied that we can communicate with Kube2iam
  response=$(curl --write-out %{http_code} --silent --output /dev/null --connect-timeout 2 'http://169.254.169.254/latest/meta-data/iam/security-credentials/')

  if [[ $response == 200 ]] ; then
    exitCode=0
    break
  fi

  echo "Attempt #${attempt}. Failure connecting to Kube2iam! Retrying in ${timeout}s." 1>&2

  sleep $timeout
  attempt=$(( attempt + 1 ))
done

once kube2iam returns 200 OK, the rest of the run command proceeds. Note that this has implications for local development, since presumably you won’t have kube2iam running on developers’ laptops when they run these containers.

but the real solution is to switch to kiam which does not have this kube2iam race condition

With kiam you have to run some dedicated machines (> 1 in order to avoid a single point of failure) just to run the kiam server component, so it’s not a drop-in replacement for kube2iam (since kiam requires more infrastructure than kube2iam).

This does seem to be the same type of timing issue that we have seen before. You probably pieced this together, but in your 2nd example, the Kube2IAM watch has not received notification that the Pod has become running (which is when we get the IP and cache the IP -> Role) until after you have made two failing calls.

There are a few events that Kube2IAM is registering in your example, but if you notice the pod.status.ip is nil until the OnUpdate event @ time="2018-05-31T15:08:59Z". Your calls to Kube2IAM will not return anything until then.

I don’t know how, in it’s current form, we could give people both things they need:

• A very fast response as some SDKs have extremely aggressive timeouts for Metadata calls
• Blocking semantics until the pod updates.

A much bigger solution that has floated around would be to augment the “Pod based” IAM approach and augment it with a “Service Account” based IAM approach. This would solve part of this problem potentially, because we can see the Pod’s service account before it ever gets an IP so we could start the STS handshake and caching of credentials even if we hadn’t gotten the OnUpdate event with the Phase == Running.

I don’t want to “Fork” the repo because I feel like @jtblin deserves to get credit for the good work he’s put into this and the very crafty idea, but if there isn’t any updates on the #132 I will probably just start working and publishing something myself