sqlx: Unwrap on from_utf8 causes panic with non-UTF-8 database server/client encodings
Environment
- Rust toolchain version: 1.53.0-nightly (74874a690 2021-03-30) x86_64-pc-windows-msvc
- IDE name and version: CLion 2020.3.3 (CL-203.7717.62)
- Operating system: Windows 10 10.0
- sqlx version:
-
- name = “sqlx-core”
-
- version = “0.5.1”
-
- source = “registry+https://github.com/rust-lang/crates.io-index”
-
- checksum = “b1cad9cae4ca8947eba1a90e8ec7d3c59e7a768e2f120dc9013b669c34a90711”
Describing problem
If the postgres database has an encoding other than UTF-8, sqlx cannot connect to it due to an error in converting the message bytes to utf-8 encoding. Sample code:

What is in the dsn does not matter.
When I run this code, I get this:

Full stacktrace is available here - https://gist.github.com/ShagonRU/6bff4316a33895021b3aaed8554be16a
===
Next are my assumptions based on stacktrace.
After getting error (or another response) from database, this ( sqlx-core-0.5.1\src\postgres\message\response.rs:139:26 ) code panics:

Decode uses this trait impl ( sqlx-core-0.5.1\src\io\decode.rs:13:9 ):

Here is called this decode ( sqlx-core-0.5.1\src\postgres\connection\stream.rs:95:48 ):

So. As a result, If the database is not in UTF-8 encoding, then the user will simply get panic and will not be able to do anything about it. There is also no explanation anywhere about this moment. Moreover, as will be shown in the example above, the server can be in UTF-8 encoding, but if the client is not in it, there will be the same panic.
For example - my docker container with postgres:
And i getting the same error as with my local postgres.
I can only connect to a remote server from my system. (fortunately I have remote server, as well as the second non-Windows system, but this is still confusing)
Possible solutions
- Remove
.unwrap()and return correct error about encoding (???, i don’n know what user can do with this error) - Find out what encoding was used and decode messages to utf-8 from it.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 20 (5 by maintainers)
I just hope it’s not lost on anyone how hilarious of a hack this is.
I think I decoded the output. Response is
Decoding as WIN1251
Meaning it’s an auth error which is in WIN1251 for unknown reason
Oh I believe it’s happening. I’m just a little floored that it is. Documentation has been known to lie and postgres docs are quite bad. I’ll see what I can figure out. Thank you for the expanded details.
Found how JDBC fixed this:
https://github.com/zemian/pgjdbc/commit/bd6de1b360c6a72b5513597813bef3773835e84e#diff-92be561701c813957ebca8ca97c2a1e3e0eb9a822c4681dcd2cc0ddfcb34451e
Basically this can only happen for auth errors during connection start-up and will only be FATAL errors. This limits the problem domain a lot. And JDBC then “predicts” the encoding using what it found for
FATAL.