superset: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4

Make sure these boxes are checked before submitting your issue - thank you!

I have checked the superset logs for python stacktraces and included it here as text if any
I have reproduced the issue with at least the latest released version of superset
I have checked the issue tracker for the same issue and I haven’t found one similar

Superset version

0.15.4

Expected results

change the metadata db from db2 to previous db1 can work

Actual results

after changing to db1, exists the encoding error

Steps to reproduce

add ?charset=utf8 in sqlalchemy_database_uri , and create database db1 character set utf8, use db1 as the metadata db, works
add ?charset=utf8 in sqlalchemy_database_uri , and create database db2 character set utf8, use db2 as the metadata db, works
change the sqlalchemy_database_uri from db2 to db1
view a slice or use SQL Lab, occur the encoding bug

python3.5 superset 0.15.4 mysql 5.7

File "/home/jiajie/project/superset/superset/superset/views.py", line 1588, in explore
    datasources = sorted(datasources, key=lambda ds: ds.full_name)
  File "/home/jiajie/project/superset/superset/superset/views.py", line 1588, in <lambda>
    datasources = sorted(datasources, key=lambda ds: ds.full_name)
  File "/home/jiajie/project/superset/superset/superset/models.py", line 1155, in full_name
    self.database, self.table_name, schema=self.schema)
......................
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/strategies.py", line 557, in _load_for_state
    return self._emit_lazyload(session, state, ident_key, passive)
  File "\<string\>", line 1, in \<lambda\>
    
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/strategies.py", line 603, in _emit_lazyload
    return loading.load_on_ident(q, ident_key)
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/loading.py", line 223, in load_on_ident
    return q.one()
  .......
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/util/compat.py", line 186, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/loading.py", line 75, in instances
    rows = [proc(row) for row in fetch]
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/loading.py", line 75, in <listcomp>
    rows = [proc(row) for row in fetch]
 ....................
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy_utils/types/encrypted.py", line 261, in process_result_value
    decrypted_value = self.engine.decrypt(value)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy_utils/types/encrypted.py", line 89, in decrypt
    decrypted = decrypted.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4: invalid continuation byte

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 16 (13 by maintainers)

Commits related to this issue

setup: Bump sqlalchemy-utils to 0.32.16 Now sqlalchemy-utils will make explicit that we are trying decoding the secret with the wrong key instead of a generic UnicodeDecodeError. Fix #2600 — committed to xrmx/superset by xrmx 7 years ago
setup: Bump sqlalchemy-utils to 0.32.16 (#3405) Now sqlalchemy-utils will make explicit that we are trying decoding the secret with the wrong key instead of a generic UnicodeDecodeError. Fix #26... — committed to apache/superset by xrmx 7 years ago

Most upvoted comments

this is a workaround you can use - convert the utf field to ascii using mysql expression in the edit dimension.

or include ?charset=utf8 in the SQLAlchemy URL

kkalyan on Apr 13, 2017

Here is how I was able to fix this: https://github.com/apache/superset/issues/8538#issuecomment-922061313

jhult on Sep 17, 2021

I’ve been down for some time. I checked the security key and am using same one as with 2.7. I upgraded to 3.6 and tried appending the sqlalchemy to this: SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(DATA_DIR, 'superset.db?charset=utf8') with no luck & I get this error message: 2017-08-15 02:19:26,671:ERROR:flask_appbuilder.security.sqla.manager:DB Creation and initialization failed: 'charset' is an invalid keyword argument for this function

My complete traceback below.

Any help would be appreciated! Traceback (most recent call last): File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app response = self.full_dispatch_request() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request rv = self.handle_user_exception(e) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception reraise(exc_type, exc_value, tb) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise raise value File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/utils.py", line 574, in wraps return f(self, *args, **kwargs) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/views/core.py", line 1691, in dashboard if datasource and not self.datasource_access(datasource): File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/views/base.py", line 94, in datasource_access self.schema_access(datasource, user=user) or File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/views/base.py", line 87, in schema_access self.database_access(datasource.database, user=user) or File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 237, in __get__ return self.impl.get(instance_state(instance), dict_) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 584, in get value = self.callable_(state, passive) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 557, in _load_for_state return self._emit_lazyload(session, state, ident_key, passive) File "<string>", line 1, in <lambda> File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 603, in _emit_lazyload return loading.load_on_ident(q, ident_key) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 223, in load_on_ident return q.one() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2814, in one ret = self.one_or_none() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2784, in one_or_none ret = list(self) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 90, in instances util.raise_from_cause(err) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise raise value File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 75, in instances rows = [proc(row) for row in fetch] File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 75, in <listcomp> rows = [proc(row) for row in fetch] File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 437, in _instance loaded_instance, populate_existing, populators) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 498, in _populate_full dict_[key] = getter(row) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/sql/type_api.py", line 1122, in process return process_value(impl_processor(value), dialect) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy_utils/types/encrypted.py", line 261, in process_result_value decrypted_value = self.engine.decrypt(value) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy_utils/types/encrypted.py", line 89, in decrypt decrypted = decrypted.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 4: invalid continuation byte

jerryhall on Aug 15, 2017

@zhangjiajie023 @timfeirg Here this was caused by having in the config a SECRET_KEY different than the one used to encrypt the password.

xrmx on Jun 23, 2017