podman: OS X, podman machine time stops sometime
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description Date output is wrong for both containers and podman machine itself on OS X.
Steps to reproduce the issue: I’m not sure 100% in my reproduction guide
- Install podman on OSX
- Create podman machine
- Wait couple of days
- Get date from podman machine
Describe the results you received: OS X date:
$ date
Sun Sep 12 15:55:30 MSK 202
Podman container and podman machine ssh date:
$ date
Fri Sep 10 19:47:08 UTC 2021
I have ran same command again and time was completely the same. After stoping and starting machine again time started to go.
Describe the results you expected: Date inside podman machine should be right
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md) Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
Client:
Version: 3.3.1
API Version: 3.3.1
Go Version: go1.17
Built: Mon Aug 30 22:15:26 2021
OS/Arch: darwin/amd64
Server:
Version: 3.3.1
API Version: 3.3.1
Go Version: go1.16.6
Built: Mon Aug 30 23:46:36 2021
OS/Arch: linux/amd64
OS X info:
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 33
- Comments: 125 (52 by maintainers)
Commits related to this issue
- podman machine: Adjust Chrony makestep config This allows Chrony to update the system time when it has drifted far from NTP time. By default Chrony only makes slight adjustments, but in the case wher... — committed to xordspar0/podman by xordspar0 a year ago
- podman machine: Adjust Chrony makestep config This allows Chrony to update the system time when it has drifted far from NTP time. By default Chrony only makes slight adjustments, but in the case wher... — committed to xordspar0/podman by xordspar0 a year ago
- podman machine: Adjust Chrony makestep config This allows Chrony to update the system time when it has drifted far from NTP time. By default Chrony only makes slight adjustments, but in the case wher... — committed to xordspar0/podman by xordspar0 a year ago
- podman machine: Adjust Chrony makestep config This allows Chrony to update the system time when it has drifted far from NTP time. By default Chrony only makes slight adjustments, but in the case wher... — committed to xordspar0/podman by xordspar0 a year ago
- podman machine: Adjust Chrony makestep config This allows Chrony to update the system time when it has drifted far from NTP time. By default Chrony only makes slight adjustments, but in the case wher... — committed to xordspar0/podman by xordspar0 a year ago
- podman machine: Adjust Chrony makestep config This allows Chrony to update the system time when it has drifted far from NTP time. By default Chrony only makes slight adjustments, but in the case wher... — committed to openshift-cherrypick-robot/podman by xordspar0 a year ago
Alright, I’m starting to actively work on this, thanks for holding tight y’all 😃
Might be a little ham-fisted, but whenever I notice the machine’s date is out of sync I just set it to the same as the host with
podman machine ssh date --set $(date -Iseconds)makestepseems to fit this use-case exactly:To update the default Podman-Machine:
I ran into this and used this as a workaround:
On macOS (Catalina) I had to modify the command a bit:
In MacOS I do this for rootless containers
And then in the machine shell
May be this could become an optional auto apply script in the desktop app companion for MacOS. Executed, when OS resumes from sleep/suspend.
You could also run as root directly without sudo. Monterey & podman 4.1.1
Devs are using Podman in our organisation. We have all MacBook Pros and we all have frequent time drifts on all recent versions (in fact all versions we tried), including latest 4.1, causing us frequent random issues. It would be good if podman could have some built in solution to the problem. It wouldn’t be anything unusual since there are other OSX specific fixes available already.
Is there anyway to easily configure the VM? No real need to install additional services as one possible issue is to add NTP servers to the systemd-timesyncd service. Will report back after the weekend if this hasn’t worked.
This of course won’t help if there’s firewalls blocking stuff, as in https://github.com/containers/podman/issues/11541#issuecomment-984179128
There is a long blog post about this on the Docker blog. https://www.docker.com/blog/addressing-time-drift-in-docker-desktop-for-mac/
Docker Desktop runs an embedded NTP server on the host. It takes his source on the system time. A NTP client in the VM keeps it in sync.
hwclockwas not sufficient for me on my M1 MBP: I would still get some skew (not as long as the machine had been off but enough to cause problems with e.g. certificate and signature validation). Instead. I tell chrony to do its thing:podman machine ssh "sudo chronyc -m 'burst 4/4' makestep; date -u"Note that this will step the clock instantly instead of slewing. Hence it is appropriate for running immediately on resume but it may impact existing containers negatively. See chrony’s
makestepand related documentation for details.Same issue here, noticed today after my mac hibernated over the weekend. Nothing helped except restarting the machine.
@ashley-cui / @rhatdan Sadly this is still an issue after over a year.
While this gets fixed, should we mark
podmanas beta for macOS users?Took me on an unbelievable ride on three instances to triage what’s wrong with signature based auth and log event timing.
Will try several options posted here and see if anything helps with internal QEMU VM. Do share any workaround if you have any!
It’s happening for me especially after resuming work after weekends. (The symptom I always see is that package repositories are “not valid yet”, e.g.,
E: Release file for http://ports.ubuntu.com/ubuntu-ports/dists/focal-updates/InRelease is not valid yet (invalid for another 3d 6h 54min 17s). Updates for this repository will not be applied..sudo systemctl restart systemd-timesyncd.service(mentioned in https://github.com/containers/podman/issues/11541#issuecomment-990998454) worked to fix it for me, @zeekoe, maybe you can try it:Ok, I wondered if Podman might already had its own Ignition configuration. I found it and did just that in #17661.
I edited
/etc/chrony.confand addedmaxchange 100000 1 2which lets chrony fix the time when it is very off. (Disclaimer: Not properly researched. Just seems to work for now.)I installed sleepwatcher via
brewand then added the following to my~/.wakeupscript to update the date and time whenever my Mac wakes up:So far, so good (until a long term fix becomes available).
This is still an issue with podman 3.4.4 on macOs.
When doing
podman buildof a Dockerfile withRUN apt-get update -qit crashes with exit status 100.E: Release file for http://security.debian.org/debian-security/dists/buster/updates/InRelease is not valid yet (invalid for another 9h 11min 44s). Updates for this repository will not be applied.I suspect this impacts many seeing this is a very common use case, so I added the error message here for discoverability.
Workaround:
The command
hwclock --hctosyswas still off by 20 minutes somehow, butsudo rebootfixed the clock. It now outputs correcttimedatectland that also fixed the above build error.I think so
@dm3ch @baude @ashley-cui Is this still a bug?
This situation can also happen after suspend/resume of the system. See a solution here: https://github.com/linuxkit/linuxkit/tree/master/pkg/host-timesync-daemon
hmm… NTP should be utilized on Fedora CoreOS by default
What’s the output of the following commands run on the Fedora CoreOS host?
timedatectlsystemctl status chronydsudo journalctl -b 0 -u chronydI asked Fedora CoreOS if it would make sense to use
makestep 1 -1as the default, but it doesn’t appear to be a good default for all qemu VMs. In that case I propose that we add this configuration to the default Podman machine image.Makestep does indeed seem like the right solution. It looks like
makestep 1 -1was discussed in a Fedora CoreOS bug and the conclusion was that it should be the default.I’m not sure why that doesn’t show up in our images.Oh, I see. The change spawned by that discussion is only for cloud VM images. Either we should make Podman do the same for its image or we should bring this up in the Fedora CoreOS project.@ashley-cui Thanks. Ping me for testing, macos support is important for me and I would be more than happy to spend some time and help making podman on macos easier to use.
The host side code for time sync for vfkit/crc is https://github.com/cfergeau/vfkit/blob/timesync/cmd/vfkit/timesync.go This should be fairly easy to adapt for use with qemu (replace the call to
vf.ConnectVsockSyncwith something connecting overvirtio-serial)@gbraad pointed me at https://github.com/prashantgupta24/mac-sleep-notifier which should be much better than
sleepwatcheras it’s native go/cgo code so it can directly be reused in crc/podman machine/…The issue is that you have to apply band-aid fix each time macOS M1 wakes up from sleep! If underlaying qemu VM had auto-NTP (which isn’t hard to setup inside VM), it would probably solve the issue.
For a perm fix, would you be able to take a PR if raised and review it? or is it that someone is actually working on the fix?
I believe the fix would help a lot of users who doesn’t even know that this is a big issue for applications that relies on time!
Alternatives such as Docker Desktop has become expensive possession to have for infrequent container building efforts.
Please guide!
had the same problem. with the latest podman update 4.2.0, i realised there is a message regarding firewall access. haven’t seen that before and was now able to allow access. it seems to be working even after sleep now. at least that’s what timedatectl indicates…
EDIT: nope, still stopping randomly or after longer sleep cycle…
I’m also reproducing it with current podman/podman machine on macOS 12.5 and I’m connected to internet
machine is
RootfulDate on my computer:
Wed Aug 10 12:06:43 CEST 2022in the VM:Tue 2022-08-09 23:50:35 CESTtimedatectl output
systemctl status chronyd
sudo journalctl -b 0 -u chronyd
Pinging from the podman machine
Just spent 2 hours trying to work out why my app using the AWS API was failing running in podman, but worked fine running locally. The culprit was the date shift/freeze - AWS includes the date in it’s API auth/signing scheme (presumably to inhibit some kinds of replay attacks) and the AWS APIs were going “that
X-Amz-Date:data isn’t in the valid window”So just a “me too” on the issue is still happening.
podman machine stop; podman machine startfixed it, albeit only temporarily based on the rest of this thread.Can’t reproduce, @ssbarnea. No errors in 50 tries. Two keys in ssh-agent.
Sure, you can do that with hammerspoon, obb, talon… It’s not particularly easy to do without 3rd party software (and requires IOKit to access the events; unfortunately launchd doesn’t expose them directly). You could do something like:
Via Talon (thanks to lunixbochs):
@flouthoc @baude @ashley-cui Should we do this in the ignition?
I think podman machine should be self-contained and not require an Internet access. What if you code in the plane? The time should not drift.