moby: Incorrect time in containers
Description
I experience incorrect time in containers based on various images. Put shortly, it seems that for many images, the container time always start at the same time regardless of host time.
Here are CMD commands/output that demonstrate the problem and can be used for reproduction:
C:\Users\Christer>time
The current time is: 9.32.03,72
Enter the new time: <Ctrl-C>
C:\Users\Christer>docker run -it microsoft/nanoserver:1803 cmd
Microsoft Windows [Version 10.0.17134.48]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\>time
The current time is: 18:23:46.36
Enter the new time: <Ctrl-C>
C:\>exit
C:\Users\Christer>time
The current time is: 9.32.30,37
Enter the new time: <Ctrl-C>
C:\Users\Christer>docker run -it microsoft/nanoserver:1803 cmd
Microsoft Windows [Version 10.0.17134.48]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\>time
The current time is: 18:23:46.12
Enter the new time:
As you can see, the container’s time always seem to start at the same time regardless of host time. This means it does not not seem related to timezones.
I experience this consistently e.g. with microsoft/nanoserver:1803 as well as microsoft/nanoserver:1709, but microsoft/nanoserver:sac2016 works fine.
However, when I posted the steps in this MSDN thread, another poster was unable to reproduce it. This makes me think it’s not related to the images themselves. And since the closing comment of the seemingly similar issue https://github.com/docker/for-win/issues/1288 made it clear that moby was the right place for such a bug report, here I am, quite lost and unable to figure out how to proceed.
As mentioned in the MSDN thread I linked to, I tried running the images with process isolation, but it seems that’s not possible:
C:\Users\Christer>docker run --isolation process -it microsoft/nanoserver:1803 cmd
docker: Error response from daemon: Windows client operating systems only support
Hyper-V containers.
See 'docker run --help'.
I’ll happily provide more info and do more testing, just tell me what’s necessary.
Output of docker version:
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:12:48 2018
OS/Arch: windows/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.24)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:21:42 2018
OS/Arch: windows/amd64
Experimental: false
Output of docker info:
Containers: 54
Running: 0
Paused: 0
Stopped: 54
Images: 28
Server Version: 18.03.1-ce
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd gelf json-file logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 17134 (17134.1.amd64fre.rs4_release.180410-1804)
Operating System: Windows 10 Pro
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 15.89GiB
Name: Navi
ID: VUXH:PMCX:6USB:3B7W:QTEP:WQT3:BV42:6VJN:XOYG:RQRO:QW4K:SZER
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: -1
Goroutines: 23
System Time: 2018-06-14T15:14:57.4023897+02:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 12
- Comments: 60 (17 by maintainers)
I can confirm somewhat similar behavior. 2 images based on windowsservercore 1803, both run about 7 hours ahead of the host. The host has never been suspended, only running or completely shut down (but shut down only for maybe 1 hour in total). Removing and recreating the containers doesn’t help. The containers are used as guild agents and they timestamp the builds, so this is very inconvenient.
Especially because there is currently also no way to get w32time working so there is no way to correct the time manually either.
The current ETA for updated RS3/RS4 base-image-backports is 2018/8B which means container images release on patch-Tuesday of August 2018. There’s no guarantee though - it may slip to Sept 2018 updates. (I have no control over backports, just relaying on information from internal sources).
MS internal bug number is 16936366. Fix still in the works for a future push to
microsoft/windowsservercore, nanoserver and child containersIsolation mode can be very useful when:
It looks like it’s a combination of a race condition, and if your host is set to a TZ east of PST. An RS4 backport is in the works. I’m chasing the RS3 backport (RS1 is not affected) but cannot promise any ETA yet. Note the fix is to the utility VM image, not the host, so will require images built from updated base images.
Just to add to this. I’m also experiencing this issue and it’s completely stopping Application Insights from working.
Adding this here even though it’s closed because it’s the most exhaustive thread i’ve found about this issue. @olljanat or @lowenna can you offer any insight?
Similar to what @hongaar reports above I still see this on some w10 1903 docker hosts’ containers (some deltas, see below). The time is always about four hours ahead.
The host is able to sync to time.windows.com without issue and does show the current time. The host is in EST timezone, but changing the timezone does not sync the time. Have tried to follow this thread, do not see a conclusive answer, maybe I missed something
Source image: I’m using mcr.microsoft.com/dotnet/framework/aspnet:4.7.2 as my source image, so, servercore
This may be relevant: the hosts that exhibit this symptom are all azure-hosted virtual machines that are not domain-joined. Conversely, I do not see this symptom on corporate network win10 domain-joined 1903 hosted containers using the same image, these appear to work properly and sync time to the host.
Windows 10 1903 18362.387 Host: Wednesday, October 2, 2019 11:47:54 AM Container: Wednesday, October 2, 2019 2:19:08 PM
Client: Docker Engine - Community Version: 19.03.2 API version: 1.40 Go version: go1.12.8 Git commit: 6a30dfc Built: Thu Aug 29 05:26:49 2019 OS/Arch: windows/amd64 Experimental: false
Server: Docker Engine - Community Engine: Version: 19.03.2 API version: 1.40 (minimum version 1.24) Go version: go1.12.8 Git commit: 6a30dfc Built: Thu Aug 29 05:39:49 2019 OS/Arch: windows/amd64 Experimental: false
As far as I can tell from internal info, it went out in the patch-Tuesday container images updates in August. You need to use images built on those base images.
@rogersillito I was meaning that original message on this issue which links to MSDN thread but looks that they actually just suggested to try process isolation mode.
Anyway, I can tell that we have multiple Win 2016 and 1803 hosts and containers running on production using process isolation mode and there is not any issues with incorrect time (we even OAuth with apps so time much be correct to make authentication working).
For me the Hyper-V containers are running 6h58m36s ahead of local time, for @olljanat this is 6h58m21s. Some difference in seconds due to when we run the command probably. Much more important is that this is almost exactly 7 hours. And Redmond’s local time is currently… yep, 7 hours behind UTC.
From how on are only speculations but: I don’t think this is really a bug in the images, but rather something wrong with the build process. The build machine might have it’s time set equal to UTC but it’s time zone still set to Redmond time or visa versa, and then somehow this causes a constant offset in the base images. If the build machine does never sync time with the internet this won’t get automatically corrected. This would also explain why the offset is 1m24s different from the 7 hours: due to the lack of time syncing the time of the build machine drifted over time. If this scenario is true then this is quite worrying though: the time in containers can then only be as correct as the time on the build machine. IMO the containers should sync with the host they are running on without any offset caused by the build.
I will try the suggestion of @jhowardmsft later tonight. BTW maybe the reason you cannot repro is because you are actually in Redmond.
@Stannieman Yea. That is most probably reason why they are developing these things.
I just belong to that rare group of people who are crazy enough to run Windows containers on production and that I have pent huge amount for time to figure out what is actually production ready. Hyper-V isolation mode is it not.
I can confirm information what is said on MSDN thread. This issue only happens if you are using Hyper-V isolation mode. And based on my experience I do not recommend that for anyone.
To be able to use process isolation mode you need use same OS version on host and containers so:
If you are new with Windows containers I highly recommend to check my git repo where I have been keep track what works and what not https://github.com/olljanat/docker-issue-tracking
Can confirm this behaviour. Highly frustrating 😕