android-emulator-runner: Why running the action on Linux VMs is slow

@JavierSegoviaCordoba I can do 😄, to be a bit more centralised for everyone watching I wrote up where we are at under https://github.com/actions/runner-images/issues/183

If y’all do any testing I would ❤️ feedback on the perf bits, I saw better perf on the 4-core than on the Macs which netted out pretty well 😃

Hey all, sorry I haven’t followed up. We are actively working on migration plans still for the existing 2-core machines to enable this. I am sorry it has taken longer than I expected 😞 and please keep an eye here for me to follow up.

Just did some benchmarking of the 4, 8, 16, 32 core linux agent with KVM enabled vs macOS

The project I used was a non-compose project with about 10 UI tests. This was a fully fledged and shipped android application and not a sample app.

The 4 core is super cheap and stable, and still significantly faster than macOS. (7min46sec vs 14min22sec total build + test time

The 8 core seemed to be the most cost-efficient and was about 33% faster than the 4 core.

The 16 core was a slight build improvement above the 8 core but almost the same testing time.

The 32 core had negligible increases from the 16 core and the testing time sometimes still took longer.

Seems like the 8 core will be what I’ll be trying to use, and is still only .032 cents per minute compared to the .08 cents per minute of the macOS agent.

4 Core .016 cents per minute Build 3m59s Test 3m47s Total 7m46s

8 Core .032 cents per minute Build 2m41s Test 2m47s Total 5m28s

16 Core .064 cents per minute Build 2m13s Test 2m50s Total 5m3s

32 Core: .128 cents per minute Build 2m4s Test 2m59s Total 5m3s

MacOS .08 cents per minute Build 5m15s Test 9m7s Total 14m22s

I actually ran this test with GMD and not ReactiveCircus but will do more benchmarking with avd caching + reactive circus later.

        managedDevices {
            devices {
                pixel2api30 (ManagedVirtualDevice) {
                    device = "Pixel 2"
                    apiLevel = 30
                    systemImageSource = "google"
                }
            }
        }

Linux is certainly not slow anymore! 😍

@nebuk89 - Alternatively, it would be awesome if the 4-core linux runner was free for public repos. In my case, when we switched from macOS to Linux for the KVM support, we drastically improved the stability of our test runs, but we also went from completely free to having a monthly cost associated with this. Our repo is public because its for an open source library, not an application.

FWIW, we started using a 4-core larger runner for our Android tests and it’s been working great so far.

See https://github.com/getsentry/sentry-dotnet/blob/main/.github/workflows/device-tests-android.yml#L46

The results are interesting. It’s not any slower than macOS, but it’s not dramatically faster either. However - it does seem to be more stable. When we were running on macOS, we frequently got failures booting the emulator before our tests could run at all. That doesn’t seem to be happening after the switch.

The downside is that the macOS runner was free (for public repos) and the 4-core Linux runner costs $0.016/min. We matrix for multiple versions of android, and run often, so it adds up.

I ran a test on the Ubuntu 20.04 64 core “large runner” and from what I could tell there was no KVM support.

See the run here: https://github.com/Expensify/App/actions/runs/3276676725/jobs/5393009096

Relevant Logs:

ProbeKVM: This user doesn't have permissions to use KVM (/dev/kvm).
  The KVM line in /etc/group is: [kvm:x:108:]
  
  If the current user has KVM permissions,
  the KVM line in /etc/group should end with ":" followed by your username.
  
  If we see LINE_NOT_FOUND, the kvm group may need to be created along with permissions:
      sudo groupadd -r kvm
      # Then ensure /lib/udev/rules.d/50-udev-default.rules contains something like:
      # KERNEL=="kvm", GROUP="kvm", MODE="0660"
      # and then run:
      sudo gpasswd -a $USER kvm
  
  If we see kvm:... but no username at the end, running the following command may allow KVM access:
      sudo gpasswd -a $USER kvm
  
  You may need to log out and back in for changes to take effect.
  
  WARNING | x86 emulation may not work without hardware acceleration!

I’m happy to run any further tests if I’ve misconfigured anything in the test, just let me know. cc @hannojg

I am glad this has helped folks out! Always feel free to reach out in the future ❤️ (I am working on 2-core support for KVM support currently so watch this space as well 😃 )

One gotcha I should mention. When first switching over, nothing worked. It crashed immediately when trying to start the emulator. The culprit turned out to be caching. We were using an AVD image built on macOS on the Linux runner. Easy fix - make sure that runner.os is part of the cache key.

@ychescale9 It seems like the new emulator only has that default Hypervisor Framework when called from the new tooling location, e.g. $ANDROID_HOME/emulator/emulator -accel-check and not from $ANDROID_HOME/tools/emulator – So, this means that VM Acceleration is always enabled by default when using the new emulator tooling from that $ANDROID_HOME/emulator/emulator location with the macOS agents on Github.

This is good information, and I will put in a PR to update the README. But, we still know how flakey things are without Graphics Acceleration as well.

So, it seems we don’t need HAXM – because that is only for fallback. But this is still less ideal than having both GPU + VM Acceleration.

This means that on Github Actions, MacOS Agents will have an advantage over Linux Agents when CPU/GPU stats are equal. (The tradeoff is that macOS agents are 10x more expensive than base ubuntu agents, 0.08 cents per minute vs 0.008 cents per minute)

I am trying to find time to benchmark a medium size UI Test repo on a larger ubuntu agent (beta - 16 core for example) vs the macOS agent.

PR here: https://github.com/ReactiveCircus/android-emulator-runner/pull/292

There is no better option at the moment @sisoje. Neither Linux nor Windows instances provided by Github action support hardware-accelerated VMs.

i just saw that windows machine has HAXM component installed https://github.com/actions/virtual-environments/blob/2378e1c967bc72fdf555f9b33e8c99446c06b4c4/images/win/Windows2016-Readme.md maybe thats the way to go (windows is still way cheaper than macOS)

This issue is no longer relevant as running hardware accelerated emulators with the upgraded free Linux runner is much faster than than the macOS runners:

https://github.blog/2024-01-17-github-hosted-runners-double-the-power-for-open-source/ https://github.blog/changelog/2023-02-23-hardware-accelerated-android-virtualization-on-actions-windows-and-linux-larger-hosted-runners/

On the KVM setup, we use the bits mentioned here: https://github.blog/changelog/2023-02-23-hardware-accelerated-android-virtualization-on-actions-windows-and-linux-larger-hosted-runners

- name: Enable KVM group perms
  run: |
      echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
      sudo udevadm control --reload-rules
      sudo udevadm trigger --name-match=kvm

@nebuk89 can’t you share anything here?

I came to a similar conclusion as @alexlapa. HAXM doesn’t do anything on macOS now. See https://github.com/ReactiveCircus/android-emulator-runner/discussions/286

Looks like some people are running their tests using ubuntu and are happy with https://github.com/ably/ably-flutter/issues/232#issuecomment-1054020409, even if it is slower. Me I would be interested too, because we have a very short tests suite and because the cost. I found their config here https://github.com/ably/ably-flutter/blob/2481f2ba55caf64c9f3df3e34afea1e5b0db966c/.github/workflows/flutter_integration.yaml