appium: Appium (and Inspector) cannot render elements on hybrid iOS Native/Web mobile app page
Do I have the most recent component updates?
- I use the most recent available driver/plugin and server versions
Is the component officially supported by the Appium team?
- I have verified the component repository is present under the Appium organization in GitHub
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
The application under test is the Walt Disney World mobile application on iOS. The issue is currently blocking over 1000 test cases. The issue involves the “Dine Reservation” flow. This is when a guest at Walt Disney World uses the app to query for available dining reservations amongst all of Disney World’s restaurants on a given date for a party size under 10 people. The query returns a JavaScript Collection object with a JSON payload of about 145KB. When we run our automation, no elements that we need to search for can be seen by Appium, so all such tests fail. When the same flow is performed manually with Desktop Inspector, we can see the enclosing DOM markup for the container page, but the contents of the JSON file are not displayed. This list would include images of each restaurant returned by the server which have openings for a party size under 10 people on the given date, along with the restaurant name, their location, what their cuisine is like, what their ratings are, along with quite a lot of other relevant information. This information all displays fine in the app itself–both real and simulated devices, but Appium displays nothing related to the restaurant data returned by the server.
The issue happens regardless of whether we are in Native or Web View modes.
Now, if we go through the same query again but this time select a party size of 10 or more people, the object that comes back from the server is much smaller and Appium can find all required elements. Appium Inspector is also able to display all elements for the page. The issue may be related to this bug but I am unsure: https://github.com/appium/appium/issues/18085
Expected Behavior
The contents of the page returned by the server should be readable by Appium. Likewise, during manual testing with the Appium Inspector, all page elements returned by the server should be clickable and identifiable by Appium Inspector.
Minimal Reproducible Example
Here is a cleaned up JSON blob (I’m sure I’ve screwed up syntax while cleansing data) for a SINGLE restaurant that’s included in the response by the server. But the full query will include, in this instance, about 71 restaurants! So as you will be able to see after looking at this cleaned up JSON blob (381 lines of data) for a SINGLE restaurant, that this amount of info has to be implemented for each of the 70+ restaurants (27,000+ lines) returned by the query: https://gist.github.com/lvconley/72217b0c801922a348689e9d6f3bdca1
Environment
- Operating system: iOS 16.2
- Appium server version: 2.0.1
- Appium driver(s) and their version(s): xcuitest@4.33.2
- Appium plugin(s) and their version(s): NA
- Node.js version (output of
node --version): v16.19.0 npmversion (output ofnpm --version): 9.8.1- Last component(s) version which did not exhibit the problem: NA dine flow is new to current release effort
- Platform and version under test: Walt Disney World mobile application for iOS 7.29
- Real device or emulator/simulator: Both real and simulator – iPhone 14
Link to Appium Logs
No response
Further Information
No response
About this issue
- Original URL
- State: closed
- Created 10 months ago
- Comments: 28 (12 by maintainers)
Ah ok, makes sense! Yes universal scroll will likely be much more performant for the user as well.
And for what it’s worth, we don’t really recommend the inspector for web inspection. It’s typically a better experience to use Safari for that anyway.
My apologies to you both for the delayed response here. My update above was factually incorrect. I had not switched to web view when I made that update. I was still in native mode view. When I do switch to web view, the elements can be seen, but Appium Inspector essentially becomes non functional. First, when in web view mode with Appium Inspector, the whole page takes about 4 minutes to load when we query for available restaurants where a party of two can make a reservation. The blob that comes back from the server varies in size depending on the list of restaurants that are returned. The properties sheet for every restaurant returned is 381 lines. On average, for a party of two on any given day at Walt Disney World, a list of 91 restaurants will be returned. That’s 34,671 lines in the json that’s returned. When this page is done loading in Appium Inspector, navigating the DOM tree by opening an element takes about 6 seconds per element click. And if the element you are trying to find is hundreds of elements deep, forget about finding it manually. So, we have an initiative in place to do a universal scroll approach, where the server sends back an initial list of 14 restaurants–which always display in Inspector, and can always be seen at automation runtime. And then as the user swipes down through the list of 14 restaurants, another fetch is made dynamically as they continue to scroll until the whole list is finally navigated by the user. So, we’re pretty sure this issue will go away as soon as we’ve implemented this approach. Thanks to you and your team for all the great feedback and advice you’ve given us! It’s been very useful!
It seems like there is still some confusion here about this whole JSON object thing. At the end of the day, Appium webview inspection can only see web elements. Whatever is in the HTML of the page. Just like a user. Users don’t see JSON objects, although maybe the app parses that JSON object and turns it into web elements. So any discussion of JSON here is by definition irrelevant, however true it may be that the app receives a JSON blob in order to populate its data.
But if you are attaching the Inspector to the same webview as the one you are attaching Safari to, and not seeing the same data, then that is something we can investigate. But as Kazu said, to do that we’d need a full set of Appium logs.
We cannot say anything since no appium server log, but it depends on the meaning. If you meant these elements were available in WEBVIEW context in Appium (not inspector), but cannot see anything in the Appium Inspector, then probably appium inspector can do something. So, could you clarify:
Full Appium server log would help to take a look at possible issues
I note that this issue has been closed, but do have a significant update to add. So, our developers enabled inspectability on the problematic webview, and we can indeed see all elements with the Safari dev console on this webview, but Appium Inspector CANNOT see any elements in the returned list of Walt Disney World restaurants (it can see elements outside of the element displaying the list, which is a massive JSON object), and behaves as initially reported above. Does this mean that this is in fact an Appium issue and not an Apple issue?
My guess is still the same in that case:
Your best bet for a next step is to ask the dev team to make the test version of the hybrid app with debuggable/inspectable webviews so that you can work in the web context.
Well, it just means Appium can’t detect any webviews. This could be for a number of reasons. Make sure have read the webview setup guide for the XCUITest driver.
The way to prove you have things configured correctly is to connect to the webview from Desktop Safari’s developer tools. If you can’t do that, then it’s either not configured correctly or is not a webview.
Thanks for the updated instructions @eglitise, guess I had an older version on my system!
@lvconley since Appium Inspector 2023.8.1, the dropdown is shown directly in the top application header (see example video in appium-inspector/pull/1006). In your screenshots you have enabled hybrid mode but you have the orange warning icon, which means that no webview contexts were found. I would suggest trying to connect to your app with Desktop Safari developer tools, like @jlipps said.
if nothing shows up there, then you will want to see if you can connect to the webview with Desktop Safari. if you can’t do that, then it’s likely that the webview itself is not debuggable and you’re out of luck. (Note that webviews in newer versions of iOS are not debuggable by default and require setting an inspectable flag on them, which must be done in the app source code).
The part about the app querying for JSON blobs is, I think, irrelevant to this point. Fundamentally what the app has in its native or web content is all that matters, not how it gets there. My guess is that what you’re running into is the fact that you have a hybrid app, and iOS is promoting some of these elements to native elements for accessibility convenience. But when the number of elements is too large, it stops doing that. The solution would be to switch into the webview context.
The way to do this in the inspector is to first enable webview detection, which it looks like you have done. But there is also a web context dropdown within which you must actually choose/active the webview context. It’s a 2-step process in other words (since there might be multiple webviews, so Appium cannot know which one you might want to switch into). Let me know if you have trouble finding the webview context switcher dropdown after looking for it, and I can hunt for a screenshot.
I don’t observe any screenshots made from the webview context above. Being in web context the page source cannot contain native elements whose class names start with XCUI
XCUITest relies on Apple’s XCTest framework, which queries the accessibility layer to gain the page tree. I would try to check the app with the native Xcode’s Accessibility Inspector as described in https://medium.com/yay-its-erica/ios-accessibility-the-accessibility-inspector-voiceover-swift-3-5f8e2bc50b20
If the inspector is unable to recognize elements then there is no way they could be recognized by XCUITest. In case the app under test itself properly exposes components to the accessibility then only Apple could fix such issues.
Some known issues that could be possible duplicates of this one: