AutoGPT: browse_website is broken on Apple M1 & M1 Max due to chromedriver mismatch

Joining late to the party.

I had the same issue and this is what I did that solved the issue for me in my M1

• Dockerfile is as it comes from master
• I modified web_selenium.py and changed:

driver = webdriver.Chrome( executable_path="/usr/bin/chromedriver", options=options )

it seems that the original line that installs the chrome driver is installing a wrong version while the version installed i the docker file is the proper one because:

• Chromium installed is v122
• Chromedriver installed by Dockerfile is v122
• Chromedriver installed in the python code is the WRONG one

Hope this helps others.

I have been hitting a very similar issue, albeit a different error code and @Pwuts suggested I add my logs to this Issue thread:

I am running AutoGPT within a Docker container in continuous mode and everything works great until it tries to access a website and I receive the following error:

SPEAK:  I will use the 'browse_website' command to visit the third website on the list and take notes on important segments that will help me form my thesis on the optimal strategy.
Attempting to fix JSON by finding outermost brackets 
Apparently json was fixed. 
NEXT ACTION:  COMMAND = browse_website ARGUMENTS = {'url': 'https://www.investopedia.com/terms/q/quantitative-trading.asp', 'question': 'optimal strategies for quantitative trading using Pinescript'}
SYSTEM:  Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255

This results in a near-iterative loop as it works through the array of websites it initially finds in the Google API search. I am on a MacBook Pro with an M1 Max chip and I have configured my Docker container to use Pinecone, Google, Hugging Face, and my OpenAI GPT-4 API Key but the issue seems to be solely related to the browse_website function. I have tested on master, stable, and some other branches that claimed to solve similar issues to no avail.

It sounds like another user has ran into the same issue and we are pretty certain it is due to the Mac M1 architecture but we are unclear on how to work around. Any help would be much appreciated!

          executable_path="/usr/bin/chromedriver", options=options
   )

Editing web_selenium.py fixed my issue with the browse_website command without doing anything else! (Mac M1 Max; using Dockerfile to run in a container)

This is the complete function I’m using in case it’s easier to copy and paste:

def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
    """Scrape text from a website using selenium

    Args:
        url (str): The url of the website to scrape

    Returns:
        Tuple[WebDriver, str]: The webdriver and the text scraped from the website
    """
    logging.getLogger("selenium").setLevel(logging.CRITICAL)

    options_available = {
        "chrome": ChromeOptions,
        "safari": SafariOptions,
        "firefox": FirefoxOptions,
    }

    options = options_available[CFG.selenium_web_browser]()
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
    )

    if CFG.selenium_web_browser == "firefox":
        driver = webdriver.Firefox(
            executable_path=GeckoDriverManager().install(), options=options
        )
    elif CFG.selenium_web_browser == "safari":
        # Requires a bit more setup on the users end
        # See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
        driver = webdriver.Safari(options=options)
    else:
        if platform == "linux" or platform == "linux2":
            options.add_argument("--disable-dev-shm-usage")
            options.add_argument("--remote-debugging-port=9222")

        options.add_argument("--no-sandbox")
        if CFG.selenium_headless:
            options.add_argument("--headless")
            options.add_argument("--disable-gpu")

        driver = webdriver.Chrome(
            executable_path="/usr/bin/chromedriver", options=options
    )
    driver.get(url)

    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, "body"))
    )

    # Get the HTML content directly from the browser's DOM
    page_source = driver.execute_script("return document.body.outerHTML;")
    soup = BeautifulSoup(page_source, "html.parser")

    for script in soup(["script", "style"]):
        script.extract()

    text = soup.get_text()
    lines = (line.strip() for line in text.splitlines())
    chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
    text = "\n".join(chunk for chunk in chunks if chunk)
    return driver, text

Thank you so much for the fix!

@jcalderonzumba

Joining late to the party. I had the same issue and this is what I did that solved the issue for me in my M1

• Dockerfile is as it comes from master
• I modified web_selenium.py and changed:

` driver = webdriver.Chrome(

        executable_path="/usr/bin/chromedriver", options=options

    )`

it seems that the original line that installs the chrome driver is installing a wrong version while the version installed i the docker file is the proper one because:

• Chromium installed is v122
• Chromedriver installed by Dockerfile is v122
• Chromedriver installed in the python code is the WRONG one

Hope this helps others.

#2600 (comment) Did you not receive the same error I did that claims it cannot find the driver located at /usr/local/bin/chromedriver?

You are using /usr/local/bin/chromedriver, but you need to use /usr/bin/chromedriver because that is the location of the chromedriver installed in the Dockerfile definition.

Just tried it, and it works! Even when Rosetta emulation is disabled in Docker.

Have a stable branch, just changed one line in web_selenium.py So the problem is with webdriver-manager setup, link to repo.

This could be a solution, detect the platform, then use this setup (from README.md of webdriver_manager repo):

If the Opera browser is installed in a location other than C:/Program Files or C:/Program Files (x86) on windows and /usr/bin/opera for all unix variants and mac, then use the below code,

options = webdriver.ChromeOptions()
options.binary_location = "path/to/opera.exe"
driver = webdriver.Remote(webdriver_service.service_url, options=options)

The only thing I can do that makes it work is changing the Dockerfile

FROM --platform=linux/amd64 python:3.10-slim

the build time 5x’s though

This didn’t resolve the issue for me. It produces a new error instead. Is this the only change you made?

Even if you try all suggested solutions to get it working on the latest release v0.2.2 on Docker & Mac M1, you will still get:

Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255

There are two main issues:

1. you need to reference the chromedriver path in the docker container directly in the scrape_text_with_selenium function, i.e.

driver = webdriver.Chrome( executable_path="/usr/bin/chromedriver", options=options )

you can verify this is the docker container path for chromedriver by doing the following:

docker exec -it <container-id> /bin/bash 

and then once in the container

which chromedriver

1. this is where most people are going wrong, if you are on mac m1 you need to use:

FROM --platform=linux/arm64 python:3.10-slim

not linux/amd64 as others have suggested. mac m1 is arm not amd

Doing both of the changes above fixed it

@jcalderonzumba

Joining late to the party. I had the same issue and this is what I did that solved the issue for me in my M1

• Dockerfile is as it comes from master
• I modified web_selenium.py and changed:

` driver = webdriver.Chrome(

        executable_path="/usr/bin/chromedriver", options=options

    )`

it seems that the original line that installs the chrome driver is installing a wrong version while the version installed i the docker file is the proper one because:

• Chromium installed is v122
• Chromedriver installed by Dockerfile is v122
• Chromedriver installed in the python code is the WRONG one

Hope this helps others.

#2600 (comment)

Did you not receive the same error I did that claims it cannot find the driver located at /usr/local/bin/chromedriver?

You are using /usr/local/bin/chromedriver, but you need to use /usr/bin/chromedriver because that is the location of the chromedriver installed in the Dockerfile definition.

Just tried the latest stable, and here are results from my M1:

[WDM] - Downloading: 100%|███████████████████████████████████████████████████████████████████████| 6.75M/6.75M [00:00<00:00, 15.7MB/s]
SYSTEM:  Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255

Status code was: 255

when enabled Dockers Rosetta for x86/amd64 emulation on Apple Silicon:

[WDM] - Downloading: 100%|███████████████████████████████████████████████████████████████████████| 6.75M/6.75M [00:00<00:00, 12.8MB/s]
SYSTEM:  Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: -5

Status code was: -5

hey I’m a noob feeling concerned about that here on a M1 max chip I get

Command browse_website returned: Error: Message: unknown error: cannot find Chrome binary Stacktrace: 0 chromedriver 0x0000000105409670 chromedriver + 4298352 1 chromedriver 0x0000000105401bbc chromedriver + 4266940 2 chromedriver 0x0000000105034758 chromedriver + 280408 3 chromedriver 0x0000000105059e40 chromedriver + 433728 4 chromedriver 0x0000000105058308 chromedriver + 426760 5 chromedriver 0x0000000105098994 chromedriver + 690580 6 chromedriver 0x0000000105098114 chromedriver + 688404 7 chromedriver 0x00000001050622d0 chromedriver + 467664 8 chromedriver 0x0000000105063354 chromedriver + 471892 9 chromedriver 0x00000001053c96c4 chromedriver + 4036292 10 chromedriver 0x00000001053cdc64 chromedriver + 4054116 11 chromedriver 0x00000001053d42d8 chromedriver + 4080344 12 chromedriver 0x00000001053ce970 chromedriver + 4057456 13 chromedriver 0x00000001053a58dc chromedriver + 3889372 14 chromedriver 0x00000001053ed25c chromedriver + 4182620 15 chromedriver 0x00000001053ed3b4 chromedriver + 4182964 16 chromedriver 0x00000001053fc0f4 chromedriver + 4243700 17 libsystem_pthread.dylib 0x00000001a99de06c _pthread_start + 148 18 libsystem_pthread.dylib 0x00000001a99d8e2c thread_start + 8

So i’m subscribing to this issue

def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]: “”"Scrape text from a website using selenium

Args:
    url (str): The url of the website to scrape

Returns:
    Tuple[WebDriver, str]: The webdriver and the text scraped from the website
"""
logging.getLogger("selenium").setLevel(logging.CRITICAL)

options_available = {
    "chrome": ChromeOptions,
    "safari": SafariOptions,
    "firefox": FirefoxOptions,
}

options = options_available[CFG.selenium_web_browser]()
options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
)

if CFG.selenium_web_browser == "firefox":
    driver = webdriver.Firefox(
        executable_path=GeckoDriverManager().install(), options=options
    )
elif CFG.selenium_web_browser == "safari":
    # Requires a bit more setup on the users end
    # See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
    driver = webdriver.Safari(options=options)
else:
    if platform == "linux" or platform == "linux2":
        options.add_argument("--disable-dev-shm-usage")
        options.add_argument("--remote-debugging-port=9222")

    options.add_argument("--no-sandbox")
    if CFG.selenium_headless:
        options.add_argument("--headless")
        options.add_argument("--disable-gpu")

    driver = webdriver.Chrome(
        executable_path="/usr/bin/chromedriver", options=options
)
driver.get(url)

WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.TAG_NAME, "body"))
)

# Get the HTML content directly from the browser's DOM
page_source = driver.execute_script("return document.body.outerHTML;")
soup = BeautifulSoup(page_source, "html.parser")

for script in soup(["script", "style"]):
    script.extract()

text = soup.get_text()
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
text = "\n".join(chunk for chunk in chunks if chunk)
return driver, text

I tried this on my Macbook Pro M1 Pro but get this

Command browse_website returned: Error: Message: invalid argument (Session info: headless chrome=112.0.5615.138) Stacktrace: #0 0xaaaae639439c <unknown> #1 0xaaaae60fbcc4 <unknown> #2 0xaaaae60e9988 <unknown> #3 0xaaaae60e7f7c <unknown> #4 0xaaaae60e80dc <unknown> #5 0xaaaae60fe840 <unknown> #6 0xaaaae616dea8 <unknown> #7 0xaaaae616d920 <unknown> #8 0xaaaae6126adc <unknown> #9 0xaaaae6127d68 <unknown> #10 0xaaaae636200c <unknown> #11 0xaaaae6364878 <unknown> #12 0xaaaae636439c <unknown> #13 0xaaaae636af90 <unknown> #14 0xaaaae63651ec <unknown> #15 0xaaaae6343de8 <unknown> #16 0xaaaae637f458 <unknown> #17 0xaaaae637f620 <unknown> #18 0xaaaae638cd84 <unknown> #19 0xffffbc144648 start_thread #20 0xffffbb8ebc1c <unknown>

@Pwuts

Please try adding these packages to the end of line 16 in the Dockerfile: libgconf-2-4 libfontconfig1

https://github.com/Significant-Gravitas/Auto-GPT/blob/14d3ecaae75a80f03cd118b5767e1c6affb4e3cc/Dockerfile#L16

Then rebuild and try again. Does that help?

Tried within devcontainer but failed as follows:

NEXT ACTION:  COMMAND = browse_website ARGUMENTS = {'url': 'https://freebird.in', 'question': 'What services does Freebird offer?'}
[WDM] - Downloading: 100%|██████████████████████████████████| 6.75M/6.75M [00:01<00:00, 4.55MB/s]
SYSTEM:  Command browse_website returned: Error: Message: unknown error: Chrome failed to start: crashed. (chrome not reachable) (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x0040006e6fe3 <unknown> #1 0x004000425d36 <unknown> #2 0x00400044eb20 <unknown> #3 0x00400044aa9b <unknown> #4 0x00400048caf7 <unknown> #5 0x00400048c11f <unknown> #6 0x004000483693 <unknown> #7 0x00400045603a <unknown> #8 0x00400045717e <unknown> #9 0x0040006a8dbd <unknown> #10 0x0040006acc6c <unknown> #11 0x0040006b64b0 <unknown> #12 0x0040006add63 <unknown> #13 0x004000680c35 <unknown> #14 0x0040006d1138 <unknown> #15 0x0040006d12c7 <unknown> #16 0x0040006df093 <unknown> #17 0x00400258aea7 start_thread