minikube: Template parsing error when getting docker network

Template parsing error when getting docker network

We’re running some minikube deployments on CI and recently (around the 21st) we started seeing failures with the following error message when starting minikube. This is currently happening only on CI, so I don’t have full detailed logs, but I’m working on reproducing it locally so I can get more.

This is from Minikube latest builds, the last published version continues to work with no issues.

Based on the timing and error message I suspect it may be related to this PR but I’m not sure!

Steps to reproduce the issue:

  1. Run minikube start with minikube from master

Full output of minikube start command used, if not already included:

minikube v1.15.1 on Ubuntu 16.04
      - MINIKUBE_WANTUPDATENOTIFICATION=false
    * Using the docker driver based on user configuration
    * Starting control plane node minikube in cluster minikube
    * Pulling base image ...
    * Downloading Kubernetes v1.19.4 preload ...
        > preloaded-images-k8s-v6-v1....: 42.10 MiB / 486.35 MiB [>_] 8.66% ? p/s ?    > preloaded-images-k8s-v6-v1....: 90.44 MiB / 486.35 MiB [>] 18.60% ? p/s ?    > preloaded-images-k8s-v6-v1....: 167.30 MiB / 486.35 MiB [] 34.40% ? p/s ?    > preloaded-images-k8s-v6-v1....: 230.84 MiB / 486.35 MiB  47.46% 314.62 Mi    > preloaded-images-k8s-v6-v1....: 304.00 MiB / 486.35 MiB  62.51% 314.62 Mi    > preloaded-images-k8s-v6-v1....: 382.67 MiB / 486.35 MiB  78.68% 314.62 Mi    > preloaded-images-k8s-v6-v1....: 448.80 MiB / 486.35 MiB  92.28% 317.76 Mi    > preloaded-images-k8s-v6-v1....: 486.35 MiB / 486.35 MiB  100.00% 373.80 M* Creating docker container (CPUs=2, Memory=3700MB) ...
    * Preparing Kubernetes v1.19.4 on Docker 19.03.13 ...
    E1130 08:37:33.452875     388 start.go:99] Unable to get host IP: network inspect: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{(index .Options "com.docker.network.driver.mtu")}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}": exit status 64
    stdout:

    stderr:
    Template parsing error: template: :1: unexpected "=" in operand

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 4
  • Comments: 20 (3 by maintainers)

Most upvoted comments

@medyagh Here’s my operation:

cshi@~$ docker network rm minikube
minikube
cshi@~$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
600eed1e8f3d        bridge              bridge              local
a09d489d2abd        host                host                local
ef2dbca65b2a        none                null                local
cshi@~$ minikube start
😄  minikube v1.16.0 on Centos 7.2
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🏃  Updating the running docker "minikube" container ...
🐳  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
cshi@~$ docker network inspect minikube
[]
Error: No such network: minikube
cshi@~$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
600eed1e8f3d        bridge              bridge              local
a09d489d2abd        host                host                local
ef2dbca65b2a        none                null                local
cshi@~$ dk ps
CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS              PORTS                                                                                                      NAMES
08c082778cc1        gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4   "/usr/local/bin/entr…"   2 minutes ago       Up 2 minutes        127.0.0.1:32779->22/tcp, 127.0.0.1:32778->2376/tcp, 127.0.0.1:32777->5000/tcp, 127.0.0.1:32776->8443/tcp   minikube

Thanksfully after I deleted the minikube network, it seemed to be OK ? But why was minikube network not created ?

I did the above on another host that had installed docker 19.03. Here’s output of minikube start:

minikube start
😄  minikube v1.16.0 on Centos 7.2
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🤷  docker "minikube" container is missing, will recreate.
🔥  Creating docker container (CPUs=2, Memory=3600MB) ...
🌐  Found network options:
    ▪ http_proxy=http://devnet-proxy.oa.com:8080
❗  You appear to be using a proxy, but your NO_PROXY environment does not include the minikube IP (192.168.49.2).
📘  Please see https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/ for more details
    ▪ https_proxy=http://devnet-proxy.oa.com:8080
    ▪ no_proxy=10.10.10.0,,localhost,.oa.com,.local,10.240.99.238,10.12.216.232,10.168.134.78,172.17.0.3,192.168.99.0/24,192.168.39.0/24,192.168.49.0/24,10.96.0.0/12
❗  This container is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.20.0 on Docker 19.03.2 ...
    ▪ env HTTP_PROXY=http://proxy.oa.com:8080
    ▪ env HTTPS_PROXY=http://proxy.oa.com:8080
    ▪ env NO_PROXY=10.10.10.0,localhost,.oa.com,.local,10.240.99.238,10.12.216.232,10.168.134.78,172.17.0.3,192.168.99.0/24,192.168.39.0/24,192.168.49.0/24,10.96.0.0/12
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass

❗  /data/cshi/bin/kubectl is version 1.18.3, which may have incompatibilites with Kubernetes 1.20.0.
    ▪ Want kubectl v1.20.0? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "" namespace by default

And this time minikube network was created. Anyway deleting the previous network minikube created by older version is correct @medyagh

@baiwfg2 @fsladkey I think you are on to something that we missed.

we used not set the MTU for minikube but later versions we started setting the MTU for the custom networks.

is it possible that your minikube docker network was created by an older minikube ? and now the newer minikube is not liking that it can not get the MTU ?

does deleting minikube and re-creating it with latest version fix this problem?

regardless if that fixes the problem I consider this a bug, we should not fail on older minikube created networks.

(we do have an integration test for upgrading from an older minikube verison to newest version, but I belive that test doesnt cover the version that we added mtu to the “docker network”

Seems like that there was a bug in that particular version (18.09.7), but not in 18.09.9

$ ./docker/docker version --format '{{ .Client.Version }}'
18.09.7
$ ./docker/docker network inspect minikube --format '{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{(index .Options "com.docker.network.driver.mtu")}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}'
Template parsing error: template: :1: unexpected "=" in operand
$ ./docker/docker version --format '{{ .Client.Version }}'
18.09.9
$  ./docker/docker network inspect minikube --format '{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{(index .Options "com.docker.network.driver.mtu")}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}'
{"Name": "minikube","Driver": "bridge","Subnet": "192.168.49.0/24","Gateway": "192.168.49.1","MTU": 1500, "ContainerIPs": ["192.168.49.2/24"]}

Please upgrade.

Sorry for that. docker network inspect minikube

docker network inspect minikube
[
    {
        "Name": "minikube",
        "Id": "24785864bc6cae7d94ffd8af64c966369522f079ecf927c7f9c3b8953489c188",
        "Created": "2020-12-19T16:47:17.65911718+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.49.0/24",
                    "Gateway": "192.168.49.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "478abaf880dc66a5a829d1d2ab9cae3ada975d73c3dfa44fc3daf25fcb501f96": {
                "Name": "minikube",
                "EndpointID": "70ea099edcb2dfe18dec7428487596961df4e075e8053e082dde1ff7bc1574ee",
                "MacAddress": "02:42:c0:a8:31:02",
                "IPv4Address": "192.168.49.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "--icc": "",
            "--ip-masq": ""
        },
        "Labels": {
            "created_by.minikube.sigs.k8s.io": "true"
        }
    }
]

docker version :

docker version
Client:
 Version:           18.09.7
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        2d0083d
 Built:             Thu Jun 27 17:56:06 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:19:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683