AutoGPT: browse_website fails with non-headless browser

@aeiberra comments worked along with :

sudo apt-get update
sudo apt-get install wget gnupg
wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
sudo apt-get update
sudo apt-get install google-chrome-stable

The error you are encountering is related to Chrome not starting correctly when using Selenium WebDriver. You can try adding a few more arguments to the options object to resolve this issue.

Update the scrape_text_with_selenium(url) function in your script as follows:

File: autogpt/web.py:

def scrape_text_with_selenium(url):
    logging.getLogger("selenium").setLevel(logging.CRITICAL)

    options = Options()
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
    )
    
    # Add the following lines:
    options.add_argument("--no-sandbox")
    options.add_argument("--headless")
    options.add_argument("--disable-dev-shm-usage")
    
    driver = webdriver.Chrome(
        executable_path=ChromeDriverManager().install(), options=options
    )

Make sure chrome is actually installed in your container, the current Dockerfile doesn’t install it for example.

looks like web.py got removed from Stable in a recent merge. This broken again and the error persists even with the commands from @darrynv

SYSTEM: Command browse_website returned: Error: Message: unknown error: Chrome failed to start: crashed. (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x560011b04fe3 <unknown> #1 0x560011843d36 <unknown> #2 0x56001186cb20 <unknown> #3 0x560011868a9b <unknown> #4 0x5600118aaaf7 <unknown> #5 0x5600118aa11f <unknown> #6 0x5600118a1693 <unknown> #7 0x56001187403a <unknown> #8 0x56001187517e <unknown> #9 0x560011ac6dbd <unknown> #10 0x560011acac6c <unknown> #11 0x560011ad44b0 <unknown> #12 0x560011acbd63 <unknown> #13 0x560011a9ec35 <unknown> #14 0x560011aef138 <unknown> #15 0x560011aef2c7 <unknown> #16 0x560011afd093 <unknown> #17 0x7f4d24f1dea7 start_thread

I think this is a specific problem for M1 or other ARM processor users. I assume you also have a problem with macOS. Try setting up Docker to use Rosetta for x86/amd64 emulation on Apple Silicon

For me, the Chat-GPT fixed the problem in web_selenium.py file:

options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument("--headless")

I am using headless server for this.

https://github.com/Significant-Gravitas/Auto-GPT/issues/2600#issuecomment-1518220293 This solution worked for me

Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011:

def scrape_text_with_selenium(url):
    logging.getLogger("selenium").setLevel(logging.CRITICAL)

    options = Options()
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
    )
    
    # Add the following lines:
    options.add_argument("--no-sandbox")
    options.add_argument("--headless")
    options.add_argument("--disable-dev-shm-usage")
    
    driver = webdriver.Chrome(
        executable_path=ChromeDriverManager().install(), options=options
    )

Plus running the commands: sudo apt-get update sudo apt-get install wget gnupg wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add - sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' sudo apt-get update sudo apt-get install google-chrome-stable Posted by @darrynv

Solved for me, running on Kali Linux Subsystem for windows.

PS: last commands produced the following errors:

Failed to retrieve available kernel versions.

Failed to check for processor microcode upgrades.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

But browse_website COMMAND works now. Thanks for the continuous support!

Modifying web_selenium.py with the options per @darkcount2011 and installing chrome per @darrynv has got me past the errors. Though the options are for chrome, if anyone is getting these errors with Safari or Firefox the options would need to be modified somewhere else.

The error you are encountering is related to Chrome not starting correctly when using Selenium WebDriver. You can try adding a few more arguments to the options object to resolve this issue.

Update the scrape_text_with_selenium(url) function in your script as follows:

File: autogpt/web.py:

def scrape_text_with_selenium(url):
    logging.getLogger("selenium").setLevel(logging.CRITICAL)

    options = Options()
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
    )
    
    # Add the following lines:
    options.add_argument("--no-sandbox")
    options.add_argument("--headless")
    options.add_argument("--disable-dev-shm-usage")
    
    driver = webdriver.Chrome(
        executable_path=ChromeDriverManager().install(), options=options
    )

I added it, but it didn’t seem to fix the problem unfortunately. Maybe it has something to do with the pathing? I don’t know what the file system usually looks like in a docker container? ` options.binary_location = “/usr/local/bin/chromedriver” also probably wrong file name but it’s an example

I’m still having this problem on an M1 mac on master with docker-compose.

Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011:

Change this part of code:

 else:

        if platform == "linux" or platform == "linux2":

            options.add_argument("--disable-dev-shm-usage")

            options.add_argument("--remote-debugging-port=9222")

            options.add_argument("--no-sandbox") # indent this line

            options.add_argument("--headless") # Add this line

        driver = webdriver.Chrome(

            executable_path=ChromeDriverManager().install(), options=options

        )

    driver.get(url)

Work for me on docker container

I’ll try again and see if it works, thanks

Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011:

Change this part of code:

 else:
        if platform == "linux" or platform == "linux2":
            options.add_argument("--disable-dev-shm-usage")
            options.add_argument("--remote-debugging-port=9222")
            options.add_argument("--no-sandbox") # indent this line
            options.add_argument("--headless") # Add this line
        driver = webdriver.Chrome(
            executable_path=ChromeDriverManager().install(), options=options
        )
    driver.get(url)

Work for me on docker container