crawl4ai: Error on 'charmap' codec can't encode character

Hi,

Can someone explain why I’m experiencing this. I tried different urls and same outcome :

🌟 Welcome to the Crawl4ai Quickstart Guide! Let's dive into some web crawling fun! 🌐
⛳️ First Step: Create an instance of WebCrawler and call the `warmup()` function.
If this is the first time you're running Crawl4ai, this might take a few seconds to load required model files.
[LOG] πŸš€ Initializing LocalSeleniumCrawlerStrategy

DevTools listening on ws://127.0.0.1:61566/devtools/browser/ff15fd50-5eab-447a-97bb-1ca9d57126bb
[LOG] 🌀️  Warming up the WebCrawler
Error retrieving cached URL: no such column: links
[LOG] 🌞 WebCrawler is ready to crawl
πŸ› οΈ  Basic Usage: Simply provide a URL and let Crawl4ai do the magic!
[LOG] πŸ•ΈοΈ Crawling https://www.morningstar.fr/fr using LocalSeleniumCrawlerStrategy...
Checking page load: 0
Checking page load: 1
Checking page load: 2
Checking page load: 3
Checking page load: 4
Checking page load: 5
[ERROR] 🚫 Failed to crawl https://www.morningstar.fr/fr, error: Failed to crawl https://www.morningstar.fr/fr: 'charmap' codec can't encode character '\uff1a' in position 237304: character maps to <undefined>
[LOG] πŸ“¦ Basic crawl result:
        Result:
        url: https://www.mornings...
        error_message: Failed to crawl http... 

Windows 11 Python 3.11

Any insight ?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

@unclecode thanks for your feedback. that worked for me with little update regarding subprocess call in setup.py. Actually you code :

if os.path.exists(f"{crawl4ai_folder}/cache"):
     subprocess.run(["rm", "-rf", f"{crawl4ai_folder}/cache"])

is referring to Unix-style commands where in Windows that should looks something like this :

cache_path = os.path.join(crawl4ai_folder, 'cache')
if os.path.exists(cache_path):
    shutil.rmtree(cache_path)

That’s being said. I would like to thank you for all the hard work you’re doing to make this a package a successful one.