moby: Networking broken on Windows Server 2019 with transparent network and gMSAs

When creating a container with a gMSA on a transparent network on Windows Server 2019, the name of the gMSA is used as hostname instead of the hostname of the container

Steps to reproduce the issue:

  1. Create a gMSA and a transparent network on Windows Server 2019
  2. Run a container using the gMSA and transparent network docker run -ti --security-opt "credentialspec=file://cont4.json" --name testgmsa1 --hostname testgmsa1 --network transpNet mcr.microsoft.com/windows/servercore:ltsc2019
  3. Run a second container using the same gMSA and transparent network docker run -ti --security-opt "credentialspec=file://cont4.json" --name testgmsa2 --hostname testgmsa2 --network transpNet mcr.microsoft.com/windows/servercore:ltsc2019

Describe the results you received: Inside of the container, the hostname is correct and it also gets a valid IP address.

First container:

Microsoft Windows [Version 10.0.17763.194]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\>hostname
testgmsa1

C:\>ipconfig
Windows IP Configuration

Ethernet adapter vEthernet (Ethernet):

   Connection-specific DNS Suffix  . : global.fum
   Link-local IPv6 Address . . . . . : fe80::8896:4ece:4065:2c63%22
   IPv4 Address. . . . . . . . . . . : 10.111.90.79
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . : 10.111.0.253

Second container:

Microsoft Windows [Version 10.0.17763.194]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\>hostname
testgmsa2

C:\>ipconfig
Windows IP Configuration

Ethernet adapter vEthernet (Ethernet) 2:

   Connection-specific DNS Suffix  . : global.fum
   Link-local IPv6 Address . . . . . : fe80::5dec:9175:4b44:8a8c%27
   IPv4 Address. . . . . . . . . . . : 10.111.95.39
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . : 10.111.0.253

However outside of the container (on the Docker host or on another machine), the container is not reachable by name and nslookup seems to think it is called like the gMSA:

PS H:\> ping testgmsa1.global.fum
Ping request could not find host testgmsa1.global.fum. Please check the name and try again.
PS H:\> ping testgmsa2.global.fum
Ping request could not find host testgmsa2.global.fum. Please check the name and try again.
PS H:\> nslookup 10.111.90.79
Server:  s00-dc2.global.fum
Address:  10.111.4.102

Name:    cont4.global.fum
Address:  10.111.90.79

PS H:\> nslookup 10.111.95.39
Server:  s00-dc2.global.fum
Address:  10.111.4.102

Name:    cont4.global.fum
Address:  10.111.95.39

Pinging the gMSA name seems to return the IP address of the container created last:

PS H:\> ping cont4.global.fum

Pinging cont4.global.fum [10.111.95.39] with 32 bytes of data:
Reply from 10.111.95.39: bytes=32 time<1ms TTL=128
Reply from 10.111.95.39: bytes=32 time<1ms TTL=128
Reply from 10.111.95.39: bytes=32 time<1ms TTL=128
Reply from 10.111.95.39: bytes=32 time<1ms TTL=128

Ping statistics for 10.111.95.39:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

Describe the results you expected: Even when using a gMSA the container should use its hostname, not the gMSA. This actually works fine when not using a gMSA. E.g. if I create the container like this (same as before, just without the gMSA):

docker run -ti --name testgmsa2 --hostname testgmsa2 --network transpNet mcr.microsoft.com/windows/servercore:ltsc2019

Then I can ping it with the expected name and nslookup also works fine:

PS H:\> ping testgmsa2.global.fum

Pinging testgmsa2.global.fum [10.111.90.87] with 32 bytes of data:
Reply from 10.111.90.87: bytes=32 time<1ms TTL=128
Reply from 10.111.90.87: bytes=32 time<1ms TTL=128
Reply from 10.111.90.87: bytes=32 time<1ms TTL=128
Reply from 10.111.90.87: bytes=32 time<1ms TTL=128

Ping statistics for 10.111.90.87:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
PS H:\> nslookup 10.111.90.87
Server:  s00-dc2.global.fum
Address:  10.111.4.102

Name:    testgmsa2.global.fum
Address:  10.111.90.87

Additional information you deem important (e.g. issue happens only occasionally): I have talked to some other people trying to move their container hosts to Windows Server 2019 and they have the same symptoms, so I think it is not something specific to my environment

Output of docker version:

Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        1ac774dfdd
 Built:             unknown-buildtime
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.24)
  Go version:       go1.10.6
  Git commit:       1ac774dfdd
  Built:            02/10/2019 04:13:25
  OS/Arch:          windows/amd64
  Experimental:     false

Output of docker info:

Containers: 10
 Running: 1
 Paused: 0
 Stopped: 9
Images: 19
Server Version: 18.09.2
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Datacenter Version 1809 (OS Build 17763.107)
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 23.99GiB
Name: s00-nsys-cont4
ID: VVLV:JWHI:MUVA:TVGP:FMNS:W662:FFOS:AJUJ:HZHY:W2BA:YSTS:OSBG
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface

Additional environment details (AWS, VirtualBox, physical, etc.): Virtual machine, not sure about the hypervisor

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 16 (2 by maintainers)

Most upvoted comments

Well, that forces clustering mechanisms on everyone, even those who don’t want to use a cluster. And I still don’t see why the gMSA needs to always define the hostname. Wouldn’t it make more sense to respect the configuration if a user explicitly defines the hostname?

And it breaks scenarios where you want to easily dynamically create new containers for new workloads as you always need to create the gMSA, the credspec file and store it on the host. If those are just single-container workloads, it complicates the setup a lot - unnecessarily, from my point of view

I can see why the design makes sense for more complicated setups but I really don’t like how it breaks the simple ones. Just my two cents…

Sorry for the confusion here – I’m still working on the new doc updates to cover the changes to gMSA support in WS2019. The behavior you’re seeing is expected. We fixed gMSA support in WS2019 to no longer require the hostname to match the gMSA name, but as a result it means the container generally ignores its hostname in favor of the gMSA name. This is by design since you would (should) not address a single container when using gMSA authentication – instead, you’re addressing a multi-instance service and getting a Kerberos ticket that’s issued to the gMSA name and not the individual container.

In short: while the behavior worked in WS2016, it was actually not intentional and the WS2019 behavior is correct. It’s on my plate to document this better.