android-test: Instrumented test execution with ANDROIDX_TEST_ORCHESTRATOR does not work with Android 11

Description

Similar to #355, but the execution hangs

Steps to Reproduce

Create new project using AS wizard
Enable Android orchestrator following the https://developer.android.com/training/testing/junit-runner
Run tests on Android Emulator with API 30

Expected Results

The test passes

Actual Results

The test hangs
logcat shows:

2020-09-15 21:28:39.486 12791-12791/com.example.myapplication E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.example.myapplication, PID: 12791
    java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo{com.example.myapplication.test/androidx.test.runner.AndroidJUnitRunner}: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6709)
        at android.app.ActivityThread.access$1300(ActivityThread.java:237)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loop(Looper.java:223)
        at android.app.ActivityThread.main(ActivityThread.java:7656)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
     Caused by: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService
        at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.connect(OrchestratedInstrumentationListener.java:93)
        at androidx.test.runner.AndroidJUnitRunner.onCreate(AndroidJUnitRunner.java:309)
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6704)
        at android.app.ActivityThread.access$1300(ActivityThread.java:237) 
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913) 
        at android.os.Handler.dispatchMessage(Handler.java:106) 
        at android.os.Looper.loop(Looper.java:223) 
        at android.app.ActivityThread.main(ActivityThread.java:7656) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) 
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947) 
2020-09-15 21:28:39.486 12791-12791/com.example.myapplication E/AndroidJUnitRunner: An unhandled exception was thrown by the app.
2020-09-15 21:28:39.493 12791-12791/com.example.myapplication E/InstrumentationResultPrinter: Failed to mark test No Tests as finished after process crash

(might be unrelated, but doesn’ happen on api 29) my private app shows in addition: mkdir: ‘/storage/emulated/0/Android/data/my.app.package’: Permission denied which points at DeviceProviderInstrumentTestTask and getAdditionalTestOutputEnabled which tries to save some log file. disabling the android.enableAdditionalTestOutput property removes the mkdir error but the test still hangs

AndroidX Test and Android OS Versions

    androidTestImplementation 'androidx.test.ext:junit:1.1.2'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
    androidTestImplementation 'androidx.test:runner:1.3.0'
    androidTestUtil 'androidx.test:orchestrator:1.3.0'

Android OS Versions: the issue exists only on API 30. The test passes on lower apis.

Link to a public git repo demonstrating the problem:

https://github.com/mateuszkwiecinski/orchestrator_doesnt_work task: ./gradlew connectedDebugAndroidTest hangs and never ends

Github action run confirming the issue: https://github.com/mateuszkwiecinski/orchestrator_doesnt_work/pull/1/checks?check_run_id=1119649523

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 18
Comments: 70 (2 by maintainers)

Commits related to this issue

Repro for https://github.com/android/android-test/issues/743 Enable clearPackageData — committed to android/testing-samples by brettchabot 4 years ago
Add forceQueryable=true to androidx.test.runner. This fixes using 'clearPackageData' on API 30 by making test apps (that build androidx.test.runner) callable from androidx.test.orchestrator. Fixes #... — committed to android/android-test by brettchabot 4 years ago
Add forceQueryable=true to androidx.test.runner. This fixes using 'clearPackageData' on API 30 by making test apps (that build androidx.test.runner) callable from androidx.test.orchestrator. Fixes #... — committed to android/android-test by brettchabot 4 years ago
Add forceQueryable=true to androidx.test.runner. This fixes using 'clearPackageData' on API 30 by making test apps (that build androidx.test.runner) callable from androidx.test.orchestrator. Fixes #... — committed to android/android-test by brettchabot 4 years ago
Solucionar provisionalmente un bug en Orchestrator Bajar targetSdkVersion y compileSdkVersion a 29. Con 30, los test quedan retenidos indefinidamente mientras se instancian. https://github.com/andro... — committed to carlos-mg89/flickrgallery by ImanolOlveira 3 years ago

Most upvoted comments

Hey all, sorry for delayed response. We are looking into this, but the fix is proving more difficult to get released than anticipated. We hope to have something out soon to address this.

brettchabot on Oct 5, 2020

Can we get an update from the Android test team when this will be addressed? This is a really bad bug. We rely on the orchestrator for our tests to pass and right now we just can’t run our tests on Android 11 test devices.

prfarlow1 on Sep 22, 2020

Thanks again for reporting this.

We were able to reproduce on a Samsung and Pixel 4 device. We’re still confused why devices have different behavior than emulators, and are following up with the Android platform team.

In the meantime, there is a workaround.

Download the orchestrator and service apks from https://maven.google.com/web/index.html#androidx.test:orchestrator:1.4.0-alpha04 and https://maven.google.com/web/index.html#androidx.test.services:test-services:1.4.0-alpha04 respectively
Install with 'adb install -g -r --force-queryable ’

Then run tests.

We’re also working with the android gradle team to put this solution into AGP.

brettchabot on Feb 18, 2021

@alex-tyro thanks for the report. I was able to repro on FTL and I filed bug against them. Unfortunately due to the android11 package visibility changes there is nothing we can do on the androidx.test library to address this. FTL will need the same fix as Studio: adding a ‘–force-queryable’ flag when installing the service apks.

brettchabot on May 3, 2021

Same issue is occurring when compiling and targeting SDK 30 and running against an emulator with 30.

2020-09-21 14:26:44.942 31419-31419/com.scompany.consumer E/AndroidRuntime: FATAL EXCEPTION: main Process: com.scompany.consumer, PID: 31419 java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6709) at android.app.ActivityThread.access$1300(ActivityThread.java:237) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913) at android.os.Handler.dispatchMessage(Handler.java:106) at android.os.Looper.loop(Looper.java:223) at android.app.ActivityThread.main(ActivityThread.java:7656) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947) Caused by: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService

no issue when compiling and targeting 30 and running on an emulator with 29.
no issue when compiling and targeting 29 and running on an emulator with 30.

LmKupke on Sep 21, 2020

I’m seeing the same result as @krisjwolff : Using clearPackageData 'true; with com.example.android.testing.androidtestorchestratorsample indeed fails on android 11 Pixel 4), with the --force-queryable workaround.

We’re investigating

brettchabot on Mar 2, 2021

I am also using testInstrumentationRunnerArguments clearPackageData: 'true' Just update core and runner versions to make it working on android 11:

androidTestImplementation 'androidx.test:core-ktx:1.4.0-alpha04'
androidTestImplementation 'androidx.test:runner:1.4.0-alpha04'
androidTestUtil 'androidx.test:orchestrator::1.4.0-alpha04'

Treat it as a reference.

cmcheungapp on Feb 28, 2021

reopening so its on our radar, will try to repro

brettchabot on Feb 16, 2021

No sorry I’m not aware of a workaround when using the com.android.support.test packages.

brettchabot on Dec 16, 2020

Thank you @brettchabot 👍 I can confirm it works in my sample project 🚀

mateuszkwiecinski on Oct 20, 2020

Please give 1.3.1-alpha02 a try

brettchabot on Oct 20, 2020

Same here. Switched to Android 10 and it’s working fine for me.

tillchen on Oct 15, 2020

@QAutomatron Here you go. If you have any questions, please ask there instead. I don’t want to continue pinging others on this thread.

larryng on May 6, 2021

Please give Studio Beta 6 and androidx.test orchestrator 1.4.0-alpha05 a try. This should fix the issues running tests on android 11+ with clearPackageData true, without requiring any workarounds.

brettchabot on Mar 15, 2021

Check out the docs on the new package visibility restrictions on API 30 here https://developer.android.com/training/basics/intents/package-visibility

IIUC using forceQueryable basically lets other apps (like the orchestrator app) communicate with the test package.

brettchabot on Oct 19, 2020

A workaround for the clearPackageData issue is to declare a

application android:forceQueryable=“true”

in the androidTest manifest

What does android:forceQueryable=“true” actually do? I can’t find any documentation for it

prfarlow1 on Oct 19, 2020

Same here. Everything was working fine with emulator api 29 and Orchestrator until we migrated to api 30.

nimeacuerdo on Sep 28, 2020