selenium: [🐛 Bug]: Selenium randomly drops the body portion of an HTML response completely

What happened?

Several times a day while working on tests run with Selenium the page source for a test which has been passing all day long (and for which neither the test code nor the server code has changed) suddenly has the complete <head> block inside the <html> element, but no <body> element at all, not even an empty <body> element. The closing </html> tag is present, so it doesn’t seem as if Selenium is still waiting for the rest of the web server’s response. Nor are we dealing with missing content expected to be filled in later by client-side scripting. I realize that it’s theoretically possible for the server-side script to leave out the entire <body> element and let the client-side scripting created it, but I own that code, and I can assure you that that’s not what’s going on here.

How can we reproduce the issue?

I cannot reproduce this behavior on demand, so I am unable to provide the normal repro case which I would usually create for a bug report. So far, the only pattern I have been able to identify is that when the problem happens, Selenium is consistently omitting the entire <body/> element, rather than randomly dropping arbitrary sections of the response document. So at this stage, I don’t really expect you to be able to reproduce the problem. What I’m hoping for is that (a) someone else will have experienced this behavior (though I haven’t found anything similar in the open issues) and will have gotten further than I have so far in coming up with clues/possible explanations/techniques for making the problem happen more frequently; and/or (b) someone will have some useful suggestions on how to make progress along those lines on my own.

Relevant log output

Because the problem (at this point) cannot be reproduced on demand, I don’t have Selenium debug logging enabled. If I get some clues that will help make the problem happen more reliably I’ll take that step.

Operating System

MacOS M1 Sonoma 14.1.2

Selenium version

4.16 on Python 3.11.4

What are the browser(s) and version(s) where you see this issue?

Google Chrome for Testing 120.0.6099.71

What are the browser driver(s) and version(s) where you see this issue?

Google Chrome for Testing 120.0.6099.71

Are you using Selenium Grid?

About this issue

Original URL
State: closed
Created 7 months ago
Comments: 17 (8 by maintainers)

Most upvoted comments

That Stack Overflow answer is well out of date. In the current w3c spec you can set the page load strategy for navigations (and incidentally for clicks in chrome, but that’s technically a bug they haven’t fixed): https://www.selenium.dev/documentation/webdriver/drivers/options/#pageloadstrategy Realize that a lot of the page is loaded after document ready state is “complete”

Implicit wait has no effect on page loading. If you want to get deep in the details of what has loaded, you’ll need to capture network traffic with BiDirectional interface: https://www.selenium.dev/documentation/webdriver/bidirectional/chrome_devtools/bidi_api/#network-interception

titusfortner on Dec 13, 2023

@bkline your page is not loaded yet, and you try to interact with it.

This is about usual waiting strategies in selenium: https://www.selenium.dev/documentation/webdriver/waits/

And this one might help in your case: https://stackoverflow.com/questions/15122864/selenium-wait-until-document-is-ready relying on document.readyState.

nvborisenko on Dec 13, 2023