ArchiveBox: Bug: `UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 110372: surrogates not allowed` when trying to render unprintable filesystem path in view

Describe the bug

Access impossible due to unicode issue

Steps to reproduce

I dont know what exactly changed/happened.

Screenshots or log output

Mar 06 19:18:07 archivebox archivebox[864]: "GET /public/ HTTP/1.1" 500 145
Mar 06 19:29:04 archivebox archivebox[864]: Internal Server Error: /public/
Mar 06 19:29:04 archivebox archivebox[864]: Traceback (most recent call last):
Mar 06 19:29:04 archivebox archivebox[864]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:29:04 archivebox archivebox[864]:     response = get_response(request)
Mar 06 19:29:04 archivebox archivebox[864]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:29:04 archivebox archivebox[864]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/base.py", line 204, in _get_resp>
Mar 06 19:29:04 archivebox archivebox[864]:     response = response.render()
Mar 06 19:29:04 archivebox archivebox[864]:                ^^^^^^^^^^^^^^^^^
Mar 06 19:29:04 archivebox archivebox[864]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/template/response.py", line 105, in render
Mar 06 19:29:04 archivebox archivebox[864]:     self.content = self.rendered_content
Mar 06 19:29:04 archivebox archivebox[864]:     ^^^^^^^^^^^^
Mar 06 19:29:04 archivebox archivebox[864]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/template/response.py", line 134, in content
Mar 06 19:29:04 archivebox archivebox[864]:     HttpResponse.content.fset(self, value)
Mar 06 19:29:04 archivebox archivebox[864]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:29:04 archivebox archivebox[864]:     content = self.make_bytes(value)
Mar 06 19:29:04 archivebox archivebox[864]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:29:04 archivebox archivebox[864]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:29:04 archivebox archivebox[864]:     return bytes(value.encode(self.charset))
Mar 06 19:29:04 archivebox archivebox[864]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:29:04 archivebox archivebox[864]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 110372: surrogates not allowed
Mar 06 19:29:04 archivebox archivebox[864]: "GET /public/ HTTP/1.1" 500 145

ArchiveBox version

0.7.2
ArchiveBox v0.7.2 BUILD_TIME=2024-03-06 19:08:32 1709752112
IN_DOCKER=False IN_QEMU=False ARCH=x86_64 OS=Linux PLATFORM=Linux-6.7.8-arch1-1-x86_64-with-glibc2.39 PYTHON=Cpython
FS_ATOMIC=True FS_REMOTE=True FS_USER=1000:1000 FS_PERMS=644
DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND=ripgrep LDAP=False

[i] Dependency versions:
 √  PYTHON_BINARY         v3.11.8         valid     /usr/bin/python3.11                                                         
 √  SQLITE_BINARY         v2.6.0          valid     /usr/lib/python3.11/sqlite3/dbapi2.py                                       
 √  DJANGO_BINARY         v3.1.14         valid     /home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/__init__.py
 √  ARCHIVEBOX_BINARY     v0.7.2          valid     /home/ol/.local/pipx/venvs/archivebox/bin/archivebox                        

 √  CURL_BINARY           v8.6.0          valid     /usr/bin/curl                                                               
 √  WGET_BINARY           v1.21.4         valid     /usr/bin/wget                                                               
 √  NODE_BINARY           v21.6.2         valid     /usr/bin/node                                                               
 √  SINGLEFILE_BINARY     v1.1.49         valid     ./node_modules/single-file-cli/single-file                                  
 √  READABILITY_BINARY    v0.0.11         valid     ./node_modules/readability-extractor/readability-extractor                  
 √  MERCURY_BINARY        v1.0.0          valid     ./node_modules/@postlight/parser/cli.js                                     
 √  GIT_BINARY            v2.44.0         valid     /usr/bin/git                                                                
 √  YOUTUBEDL_BINARY      v2023.12.30     valid     /usr/bin/yt-dlp                                                             
 √  CHROME_BINARY         v122.0.6261.111  valid     /usr/bin/chromium                                                           
 √  RIPGREP_BINARY        v14.1.0         valid     /usr/bin/rg                                                                 

[i] Source-code locations:
 √  PACKAGE_DIR           23 files        valid     /home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/archivebox
 √  TEMPLATES_DIR         3 files         valid     /home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/archivebox/templates
 -  CUSTOM_TEMPLATES_DIR  -               disabled  None                                                                        

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled  None                                                                        
 -  COOKIES_FILE          -               disabled  None                                                                        

[i] Data locations:
 √  OUTPUT_DIR            8 files @       valid     /home/ol/data                                                               
 √  SOURCES_DIR           48 files        valid     ./sources                                                                   
 √  LOGS_DIR              1 files         valid     ./logs                                                                      
 √  ARCHIVE_DIR           1683 files      valid     ./archive                                                                   
 √  CONFIG_FILE           149.0 Bytes     valid     ./ArchiveBox.conf                                                           
 √  SQL_INDEX             27.1 MB         valid     ./index.sqlite3    

logs with DEBUG=True

Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/base.py", line 204, in _get_response
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response.render()
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/template/response.py", line 105, in render
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = self.rendered_content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/template/response.py", line 134, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     HttpResponse.content.fset(self, value)
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 110372: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: During handling of the above exception, another exception occurred:
Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/archivebox/core/middleware.py", line 32, in middleware
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 49, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response_for_exception(request, exc)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     return debug.technical_500_response(request, *exc_info)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/views/debug.py", line 53, in technical_500_response
Mar 06 19:39:25 archivebox archivebox[2001]:     return HttpResponse(html, status=status_code, content_type='text/html')
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 298, in __init__
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 6662: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: During handling of the above exception, another exception occurred:
Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/utils/deprecation.py", line 114, in __call__
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response or self.get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 49, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response_for_exception(request, exc)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     return debug.technical_500_response(request, *exc_info)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/views/debug.py", line 53, in technical_500_response
Mar 06 19:39:25 archivebox archivebox[2001]:     return HttpResponse(html, status=status_code, content_type='text/html')
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 298, in __init__
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 6658: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: During handling of the above exception, another exception occurred:
Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/utils/deprecation.py", line 114, in __call__
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response or self.get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 49, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response_for_exception(request, exc)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     return debug.technical_500_response(request, *exc_info)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/views/debug.py", line 53, in technical_500_response
Mar 06 19:39:25 archivebox archivebox[2001]:     return HttpResponse(html, status=status_code, content_type='text/html')
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 298, in __init__
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 6658: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: During handling of the above exception, another exception occurred:
Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/utils/deprecation.py", line 114, in __call__
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response or self.get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 49, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response_for_exception(request, exc)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     return debug.technical_500_response(request, *exc_info)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/views/debug.py", line 53, in technical_500_response
Mar 06 19:39:25 archivebox archivebox[2001]:     return HttpResponse(html, status=status_code, content_type='text/html')
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 298, in __init__
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 6658: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: During handling of the above exception, another exception occurred:
Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/utils/deprecation.py", line 114, in __call__
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response or self.get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 49, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response_for_exception(request, exc)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     return debug.technical_500_response(request, *exc_info)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/views/debug.py", line 53, in technical_500_response
Mar 06 19:39:25 archivebox archivebox[2001]:     return HttpResponse(html, status=status_code, content_type='text/html')
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 298, in __init__
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 6658: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: During handling of the above exception, another exception occurred:
Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/utils/deprecation.py", line 114, in __call__
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response or self.get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 49, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response_for_exception(request, exc)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     return debug.technical_500_response(request, *exc_info)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/views/debug.py", line 53, in technical_500_response
Mar 06 19:39:25 archivebox archivebox[2001]:     return HttpResponse(html, status=status_code, content_type='text/html')
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 298, in __init__
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 6658: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: During handling of the above exception, another exception occurred:
Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/utils/deprecation.py", line 114, in __call__
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response or self.get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 49, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response_for_exception(request, exc)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     return debug.technical_500_response(request, *exc_info)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/views/debug.py", line 53, in technical_500_response
Mar 06 19:39:25 archivebox archivebox[2001]:     return HttpResponse(html, status=status_code, content_type='text/html')
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 298, in __init__
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 6658: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: During handling of the above exception, another exception occurred:
Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/utils/deprecation.py", line 114, in __call__
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response or self.get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 49, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response_for_exception(request, exc)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     return debug.technical_500_response(request, *exc_info)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/views/debug.py", line 53, in technical_500_response
Mar 06 19:39:25 archivebox archivebox[2001]:     return HttpResponse(html, status=status_code, content_type='text/html')
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 298, in __init__
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 6658: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: During handling of the above exception, another exception occurred:
Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/archivebox/core/middleware.py", line 25, in middleware
Mar 06 19:39:25 archivebox archivebox[2001]:     return get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 49, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response_for_exception(request, exc)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     return debug.technical_500_response(request, *exc_info)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/views/debug.py", line 53, in technical_500_response
Mar 06 19:39:25 archivebox archivebox[2001]:     return HttpResponse(html, status=status_code, content_type='text/html')
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 298, in __init__
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 6658: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: During handling of the above exception, another exception occurred:
Mar 06 19:39:25 archivebox archivebox[2001]: Traceback (most recent call last):
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/usr/lib/python3.11/wsgiref/handlers.py", line 137, in run
Mar 06 19:39:25 archivebox archivebox[2001]:     self.result = application(self.environ, self.start_response)
Mar 06 19:39:25 archivebox archivebox[2001]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/contrib/staticfiles/handlers.py", line 76, in __call__
Mar 06 19:39:25 archivebox archivebox[2001]:     return self.application(environ, start_response)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/wsgi.py", line 133, in __call__
Mar 06 19:39:25 archivebox archivebox[2001]:     response = self.get_response(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/base.py", line 130, in get_response
Mar 06 19:39:25 archivebox archivebox[2001]:     response = self._middleware_chain(request)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 49, in inner
Mar 06 19:39:25 archivebox archivebox[2001]:     response = response_for_exception(request, exc)
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
Mar 06 19:39:25 archivebox archivebox[2001]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
Mar 06 19:39:25 archivebox archivebox[2001]:     return debug.technical_500_response(request, *exc_info)
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/views/debug.py", line 53, in technical_500_response
Mar 06 19:39:25 archivebox archivebox[2001]:     return HttpResponse(html, status=status_code, content_type='text/html')
Mar 06 19:39:25 archivebox archivebox[2001]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 298, in __init__
Mar 06 19:39:25 archivebox archivebox[2001]:     self.content = content
Mar 06 19:39:25 archivebox archivebox[2001]:     ^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 328, in content
Mar 06 19:39:25 archivebox archivebox[2001]:     content = self.make_bytes(value)
Mar 06 19:39:25 archivebox archivebox[2001]:               ^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]:   File "/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/django/http/response.py", line 241, in make_bytes
Mar 06 19:39:25 archivebox archivebox[2001]:     return bytes(value.encode(self.charset))
Mar 06 19:39:25 archivebox archivebox[2001]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 06 19:39:25 archivebox archivebox[2001]: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf6' in position 6658: surrogates not allowed
Mar 06 19:39:25 archivebox archivebox[2001]: "GET /public/ HTTP/1.1" 500 59

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (5 by maintainers)

Most upvoted comments

Should be tentatively fixed in https://github.com/ArchiveBox/ArchiveBox/pull/1424

I added workaround logic in wget_output_path() to fallback to the parent dir if the html path contains unprintable unicode.

I’m now running this after moving all directories from archive to broken:

set -euo pipefail
for dir in ../broken/* ; do
        echo $dir
        mv "$dir" . #../_kaputt_/
        pushd ..
        archivebox server --nothreading '[::]:8080' & _pid=$!
        popd
        _exit=0
        sleep 2
        curl -L --fail -o /dev/null -s http://127.0.0.0:8080/public/ || _exit=$?
        if [[ $_exit -eq 7  ]]; then sleep 1 ;
                curl -L --fail -o /dev/null -s http://127.0.0.0:8080/public/ || _exit=$?
        fi
        if [[ $_exit -ne 0 ]]; then echo "$dir broken" ; exit 1 ; fi
        kill $_pid
done

This is one folder I identified:

    "canonical": {
        "archive_org_path": "https://web.archive.org/web/standards.webmasterpro.de/index.html?article=zentriertes+Layout%2C+100%25+H%F6he",
        "dom_path": "output.html",
        "favicon_path": "favicon.ico",
        "git_path": "git/",
        "google_favicon_path": "https://www.google.com/s2/favicons?domain=standards.webmasterpro.de",
        "headers_path": "headers.json",
        "htmltotext_path": "htmltotext.txt",
        "index_path": "index.html",
        "media_path": "media/",
        "mercury_path": "mercury/content.html",
        "pdf_path": "output.pdf",
        "readability_path": "readability/content.html",
        "screenshot_path": "screenshot.png",
        "singlefile_path": "singlefile.html",
        "warc_path": "warc/",
        "wget_path": "standards.webmasterpro.de/index.html@article=zentriertes+Layout,+100%+H\udcf6he.html"
    },

So a grep for literally udcf6 would have shown the issue from the beginning…

I’ll leave the script running and update again if I come across another issue.

I pulled the whole debug block in the settings.py to the bottom of the file and added ERROR_LOG="/tmp/err.log" now the server starts and throws the same 500 error as before w/o any debug toolbar 😄

[ol@archivebox data]$ echo $LANG $LC_ALL $LC_CTYPE
en_US.UTF-8
[ol@archivebox data]$ DEBUG=True DEBUG_TOOLBAR=True archivebox manage shell
[i] [2024-03-26 20:59:05] ArchiveBox v0.7.3: archivebox manage shell
    > /home/ol/data

Python 3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.22.2 -- An enhanced Interactive Python. Type '?' for help.
# ArchiveBox Imports
from archivebox.core.models import Snapshot, ArchiveResult, Tag, User
from archivebox.cli import *
    help
    version
    init
    config
    setup
    add
    remove
    update
    list
    status
    shell
    manage
    server
    oneshot
    schedule

[i] Welcome to the ArchiveBox Shell!
    https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Shell-Usage

    Hint: Example use:
        print(Snapshot.objects.filter(is_archived=True).count())
        Snapshot.objects.get(url="https://example.com").as_json()
        add("https://example.com/some/new/url")

In [1]: import os

In [2]: os.environ
Out[2]:
environ{'DEBUG_TOOLBAR': 'True',
        'DEBUG': 'True',
        'SHELL': '/bin/bash',
        'PWD': '/home/ol/data',
        'LOGNAME': 'ol',
        'XDG_SESSION_TYPE': 'tty',
        'MOTD_SHOWN': 'pam',
        'HOME': '/home/ol',
        'LANG': 'en_US.UTF-8',
        'SSH_CONNECTION': 'xxx 22',
        'XDG_SESSION_CLASS': 'user',
        'TERM': 'xterm-256color',
        'USER': 'ol',
        'SHLVL': '1',
        'XDG_SESSION_ID': '7',
        'XDG_RUNTIME_DIR': '/run/user/1000',
        'SSH_CLIENT': 'x 56510 22',
        'DEBUGINFOD_URLS': 'https://debuginfod.archlinux.org ',
        'PATH': '/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/ol/.local/bin:/home/ol/.local/bin',
        'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus',
        'MAIL': '/var/spool/mail/ol',
        'SSH_TTY': '/dev/pts/3',
        'OLDPWD': '/home/ol',
        '_': '/usr/local/bin/archivebox',
        'TZ': 'UTC',
        'PYTHONSTARTUP': '/home/ol/.local/pipx/venvs/archivebox/lib/python3.11/site-packages/archivebox/core/welcome_message.py',
        'OUTPUT_DIR': '/home/ol/data',
        'DJANGO_SETTINGS_MODULE': 'core.settings'}

Interstingly i did a sqlite3 database.db '.dump' > foo.sql (besides some strace) which lead to not having the issue anymore. I wonder what that did and if something went wrong insside the sqlite file before.

I´d still be interested in getting to know how to debug this 😃

edit: I also moved all archive data back which i suspected to cause issues and it still works.

Looks like you archived a URL that contains unprintable UTF-8 bytes (possibly from a broken emoji/accented character/crylic/chinese/arabic/etc.) and it ended up in a filesystem path, so it’s failing when trying to render the path in the public view.

In the short term you can find/strip all special UTF-8 characters in filenames using this script I wrote: strip_bad_filename_characters.sh or a program like detox (apt install detox; man detox).

In the long term ArchiveBox should fix this by force-normalizing all filenames to UTF-8 form-D on creation so this doesn’t happen in the future.