hyperkit: Memory Leaks in Docker for Mac stemming from Hyperkit

@rn I’ll try to answer your points and see if we can get this moving. A lot of people are having serious disruptions because of this issue and the Docker team and the hyperkit team are blaming each other, which is taking us nowhere…

1. We did, and we’ve been ignored for almost 3 months (https://github.com/docker/for-mac/issues/3232) as the Docker team probably think it’s Hyperkit causing the issue. We can’t say that for sure, of course but since the process having this issue is com.docker.hyperkit we also came here asking for help.
2. This connects to what @Vanuan also said, but you guys are ignoring 2 important points: a) the issue only started for most of us after the update to Mojave (we didn’t change the settings or the way we use Docker/Hyperkit, it just suddenly started using more RAM); b) Even though we set a RAM limit, this seems like completely ignored by the com.docker.hyperkit process. I’ve attached my screenshot below to show you the config and the memory usage, but if you check the docker thread you’ll see people having more than 10GB used. I don’t know why “Memory” differs so much from the combination of “Real Memory” and “Compressed Memory”, but all I know is that my Mac starts swapping like crazy and becomes unusable shortly after I start Docker, so I guess the process is not just taking the “real” memory but it’s actually holding the whole “Memory” (3.47GB in my case) from the OS. Happy to get some clarification from you if that’s not the case.
3. I agree we’ve not been very consistent with our screenshot, if you think this might help you, please provide a macOS command you’d deem specific enough us to execute and we’ll gladly report the result.
4. It might be a Memory leak, it might simply be that the Memory limit setting is being ignored. People labelled it memory leak because the memory keeps growing over time. Again, we’re happy to provide additional information if you tell us exactly which command you want us to run.

We don’t like to spend time complaining on random repositories, we’re here because our work depends on this software and we’re currently unable to work efficiently. We’re happy to collaborate as much as possible to get this solved.

Using ~4GB memory with no containers running. OS X Mojave, Docker Version 2.0.0.0-mac81 (29211)

Same issue here on OS X Mojave. About 3-4gb, without ever having run a container after its start.

hyperkit has to allocate all the memory for the VM when creating it and then pass it to the hypervisor framework in the macOS kernel. The hypervisor framework then uses this memory to populate the pagetable for the VM. Since the macOS kernel does not know if the pages are used by the VM or not it can’t just remove them

Same issue with Mojave. Using 4GB of RAM after restart with no containers running

No software solution to this. Only switching to Linux. This ticket is just being ignored. Docker for Mac says “ask Hyperkit”, Hyperkit says “ask Docker for Mac”. To what @rn said when closing: guest OS uses some memory to operate and even if you don’t run any payload, there are some OS tasks. And just by running those small OS tasks memory is consumed and not released back. It’s not a memory leak in the traditional sense, it’s just inefficiency.

Apart from consuming 2g of RSS(!) in an idle state it also consumes 5% CPU, which is eating the battery. Why doesn’t anybody talk about it?

@rn I’ll keep that command running and update here when I see something strange, and I’d invite everyone else doing the same.

However, there’s still something I’d like to clear… You say that vsz actually doesn’t matter (which is probably the huge value we see in Activity Monitor, will confirm over the next days that we know which command to use), but according to Google:

RSS is how much memory this process currently has in main memory (RAM). VSZ is how much virtual memory the process has in total. This includes all types of memory, both in RAM and swapped out.

So from an OS point of view, isn’t that the actual memory the process is using? It’s up to the OS then to optimise that (swapping, compression, etc…) but nevertheless, that’s still the whole amount of memory the process is taking. I would expect that value to be limited by my setting, not the amount that actually sits in RAM only.

Am I missing something?

Same here… iMac, Mojave Docker Version 18.06.1-ce-mac73 (26764)

com.docker.hyperkit is over 5.5GB of RAM with no container running com.docker.hyperkit is over 9.5GB of RAM with elk stack and nginx + php and node containers

Diagnostic: DC2D5189-36AD-43D6-89CB-63042103D808/20181106125353

Having the same issue: A24EF539-54C9-4F2D-A326-BD3F808506DD/20181025141141

Here is a screenshot showing my Docker for Mac settings plus activity monitor showing memory usage for hyperkit and the suggested command output showing vsz and rss.

This is all with no containers running (if that makes any difference) after starting Docker for Mac from fresh and waiting for the memory usage to stabilise.

Hopefully this helps.

I’m running docker for mac on Mojave/10.14.2

The command to check the memory used by hyperkit would be to find out the process id and then something like:

ps -o vsz,rss -p <hyperkit pid>

vsz is the virtual size (doesn’t matter) and rss is the resident set size, this is the memory actually consumed. I have no idea what the Activity Monitor reports as memory and the documentation is very scant. There are additional columns availabe in Activity Monitor and one is called Real Memory, which looks like it most closely resembles rss…

I’m running this with watch (brew install watch) in one window to see changes over time. Or if you want to see other Docker for Mac processes as well, use something like pstree (brew install pstree) to find the child processes of Docker for Mac (which includes things likes VPNKit and osxfs) and run watch ps -o vsz,rss,pid,comm -p <pid 1>,<pid2>,<pid3>...

I have my docker for mac set to have 2GB of memory. I’ve started kubernetes (just to run a few container) and run various other containers. I’ve also run several parallel:

for i in $(seq 1000); do docker run --rm alpine wget http://www.google.com; done

and something egnerate a bit more traffic, like a speed test:

for i in $(seq 100); do docker run --rm alpine sh -c 'apk add curl python; curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -'; done

Initially the rss increases, as I expected, but then maxes out at something like 1972180. This is in KB so the rss is 1.8GB.

It stays stable there.

@Vanuan Based on your screenshots, I think the issue might be the confusion between “Real Mem” and “Memory”. Suffice to say I doubt I’m alone in not knowing about “Real Mem” until reading up on it just now.

Having shown that column on both of my Macs, Real Mem does indeed show slightly less RAM usage than I have limited Docker to.

My expectations would be that the VM would use a lower amount of RAM on first start, and increase progressively up to the limit as Docker is utilised. Real Mem usage should not be higher than the limit set.

@Vanuan u make a fair point in that from docker you can set ‘memory’. I think the expectation is that memory is allocated as needed up to the limit, rather than its all ‘taken’ at once when its not in use.

Please help me understand your logic and expectations. I think everybody ITT understands how Docker on Mac works: it is not native containers, it’s container inside VM running on Mac.

If you have any experience with VM (I suppose everybody has it here), you know that VM eats memory and there’s a setting of how much memory it eats. In Docker for Mac this setting resides in Advanced tab of settings:

What I would expect that it eats exactly how much is set here. Realistically, I understand there’s overhead and it actually eats more.

My expectations aren’t met. I’m surprised it eats LESS memory than I expect it to.

And I’m surprised you expect it to consume no memory at all. How is it so? Is there something I don’t understand?

Same issue in Mojave

@McGitty see https://stackoverflow.com/questions/36571917/memory-leaks-in-golang

I disabled experimental features and debug.

{
  "debug" : false,
  "experimental" : false
}

Without any running container com.docker.hyperkit uses about 2.7Gb Running a percona:latest container uses 3.4Gb

I’m using the default settings: 4CPUs, 2.0 GiB, Swap 1.0 GiB

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     false

MacOS: Mojave 10.14.1 (18B75) MacBook Pro (15-inch, 2016) 2.7 GHz Intel Core i7 16 GB 2133 MHz LPDDR3

Diagnostic Id: 161D79AD-D3ED-4BB9-8223-AA725E6B0EA1/20181108023806

Also seeing this. Memory leaks seem to have been a pervasive issue with hyperkit for a few years now, and the leaks keep coming back when upgrades occur. Seems that something must be done differently.

Here is a diagnostic ID which should(?) tell you everything you need to know. Please ask if any more info would be helpful.

3CBF0FFC-A3ED-4F24-944A-650842923449/20181010135026

I am running Kubernetes with a few containers: tomcat, ldap, mysql, and a few Spring Boot microservices.