sqlx: Unwrap on from_utf8 causes panic with non-UTF-8 database server/client encodings

Environment

  • Rust toolchain version: 1.53.0-nightly (74874a690 2021-03-30) x86_64-pc-windows-msvc
  • IDE name and version: CLion 2020.3.3 (CL-203.7717.62)
  • Operating system: Windows 10 10.0
  • sqlx version:
    • name = “sqlx-core”
    • version = “0.5.1”
    • source = “registry+https://github.com/rust-lang/crates.io-index”
    • checksum = “b1cad9cae4ca8947eba1a90e8ec7d3c59e7a768e2f120dc9013b669c34a90711”

Describing problem

If the postgres database has an encoding other than UTF-8, sqlx cannot connect to it due to an error in converting the message bytes to utf-8 encoding. Sample code: image

What is in the dsn does not matter. When I run this code, I get this: image

Full stacktrace is available here - https://gist.github.com/ShagonRU/6bff4316a33895021b3aaed8554be16a

===

Next are my assumptions based on stacktrace. After getting error (or another response) from database, this ( sqlx-core-0.5.1\src\postgres\message\response.rs:139:26 ) code panics: image

Decode uses this trait impl ( sqlx-core-0.5.1\src\io\decode.rs:13:9 ): image

Here is called this decode ( sqlx-core-0.5.1\src\postgres\connection\stream.rs:95:48 ): image

So. As a result, If the database is not in UTF-8 encoding, then the user will simply get panic and will not be able to do anything about it. There is also no explanation anywhere about this moment. Moreover, as will be shown in the example above, the server can be in UTF-8 encoding, but if the client is not in it, there will be the same panic. For example - my docker container with postgres: image And i getting the same error as with my local postgres. I can only connect to a remote server from my system. (fortunately I have remote server, as well as the second non-Windows system, but this is still confusing)

Possible solutions

  • Remove .unwrap() and return correct error about encoding (???, i don’n know what user can do with this error)
  • Find out what encoding was used and decode messages to utf-8 from it.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 20 (5 by maintainers)

Most upvoted comments

And JDBC then “predicts” the encoding using what it found for FATAL.

I just hope it’s not lost on anyone how hilarious of a hack this is.

I think I decoded the output. Response is

[83,194,192,198,205,206,0,86,70,65,84,65,76,0,67,50,56,80,48,49,0,77,239,238,235,252,231,238,226,224,242,229,235,252,32,34,112,122,105,120,101,34,32,237,229,32,239,240,238,248,184,235,32,239,240,238,226,229,240,234,243,32,239,238,228,235,232,237,237,238,241,242,232,32,40,239,238,32,239,224,240,238,235,254,41,0,70,100,58,92,112,103,105,110,115,116,97,108,108,101,114,95,49,50,46,97,117,116,111,92,112,111,115,116,103,114,101,115,46,119,105,110,100,111,119,115,45,120,54,52,92,115,114,99,92,98,97,99,107,101,110,100,92,108,105,98,112,113,92,97,117,116,104,46,99,0,76,51,51,51,0,82,97,117,116,104,95,102,97,105,108,101,100,0,0]

Decoding as WIN1251

SВАЖНО00VFATAL00C28P0100Mпользователь “pzixe” не прошёл проверку подлинности (по паролю)00Fd:\pginstaller_12.auto\postgres.windows-x64\src\backend\libpq\auth.c00L33300Rauth_failed0000

Meaning it’s an auth error which is in WIN1251 for unknown reason

Oh I believe it’s happening. I’m just a little floored that it is. Documentation has been known to lie and postgres docs are quite bad. I’ll see what I can figure out. Thank you for the expanded details.

Found how JDBC fixed this:

https://github.com/zemian/pgjdbc/commit/bd6de1b360c6a72b5513597813bef3773835e84e#diff-92be561701c813957ebca8ca97c2a1e3e0eb9a822c4681dcd2cc0ddfcb34451e

Basically this can only happen for auth errors during connection start-up and will only be FATAL errors. This limits the problem domain a lot. And JDBC then “predicts” the encoding using what it found for FATAL.