aws-iam-authenticator: heptio-authenticator occassionally returns Unauthorized when used in a kubernetes go client

Issue:

I have a sample HTTP server that listens on /. When / is called, the handler simply authenticates with eks cluster using the Heptio-authenticator-module and returns to the user the number of pods running in that cluster.For authentication I use the kubernetes go client. Also note that for every new request on / I create a new kuberentes config so there is no config reuse between requests.

I observed that after ~15 min the eks authentication fails with a Unauthorized error which is unexpected as I am creating generating a new kuberenets config object for every request which should result in a token generated for every request. Also the immediate request to / (after unauthorized request) seems to be successful again which is more confusing.

Attached is a snippet of my code and logs

1. First request: success curl -v -s localhost:9000/ log output:

There are 2 pods in the cluster

1. Second request: success curl -v -s localhost:9000/ log output:

There are 2 pods in the cluster

1. Third request after ~15 min: failed curl -v -s localhost:9000/ log output:

2018/08/10 14:10:27 http: panic serving [::1]:63678: Unauthorized
goroutine 11 [running]:

1. Fourth request: success curl -v -s localhost:9000/ log output:

There are 2 pods in the cluster

package main

import (
  //"flag"
  "fmt"
  //"path/filepath"
  //"time"
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

  //"k8s.io/apimachinery/pkg/api/errors"
  "k8s.io/client-go/kubernetes"
  //_ "k8s.io/client-go/plugin/pkg/client/auth/oidc"
  restclient "k8s.io/client-go/rest"
  capi "k8s.io/client-go/tools/clientcmd/api"
  "net/http"
)

var (
  endpoint string = "REDACTED"
)

func getClientSet() *kubernetes.Clientset {
  awsenv := capi.ExecEnvVar{
    Name:  "AWS_PROFILE",
    Value: "dv",
  }
  config := &restclient.Config{
    Host: endpoint,
    TLSClientConfig: restclient.TLSClientConfig{
      //CAData: clusterCert,
      CAFile: "REDACTED",
    },
    ExecProvider: &capi.ExecConfig{
      Command:    "heptio-authenticator-aws",
      Args:       []string{"token", "-i", "dev-v1"},
      APIVersion: "client.authentication.k8s.io/v1alpha1",
      Env:        []capi.ExecEnvVar{awsenv},
    },
  }
  clientset, err := kubernetes.NewForConfig(config)
  if err != nil {
    panic(err.Error())
  }
  return clientset
}

func getClusterCredentials(w http.ResponseWriter, r *http.Request) {
  clientset := getClientSet()
  fmt.Println(clientset)
  pods, err := clientset.CoreV1().Pods("local").List(metav1.ListOptions{})
  if err != nil {
    panic(err.Error())
  }
  fmt.Printf("There are %d pods in the cluster\n", len(pods.Items))
  fmt.Fprintf(w, "There are %d pods in the cluster\n", len(pods.Items))
}

func main() {
  http.HandleFunc("/", getClusterCredentials)
  http.ListenAndServe(":9000", nil)

  
}