diesel: Deallocation exception when reading MySQL Time columns with "00:00:00" value

Setup

Versions

  • Rust: 1.42.0
  • Diesel: 1.4.4
  • Database: MySQL/MariaDB
  • Operating System Linux

Feature Flags

  • diesel: features = [“mysql”, “chrono”, “r2d2”]

Problem Description

Whenever a table row contains a Time value of “00:00:00” the application crashes with a “free(): invalid next size (fast)” error.

What are you trying to accomplish?

Read Time columns from MySQL/MariaDB that contain values set at midnight (00:00:00).

What is the expected output?

Normal fetching of the rows without the application crashing.

What is the actual output?

Application crashes with a “free(): invalid next size (fast)”.

Are you seeing any additional errors?

No other details shows even if RUST_BACKTRACE=1.

Steps to reproduce

Read a MySQL table with Time columns (not DateTime) that contain “00:00:00” values. The error happens even if only one of the rows contains a midnight value for Time.

Troubleshooting

I don’t believe that this is related to #1130 since the values are converted correctly to NaiveTime. In fact, I’ve even implemented a “solution” similar to the one mentioned by @frol in that ticket, adapted to NaiveTime, but it didn’t make any difference.

Through lots of tests and debugging sessions I’ve been able to narrow the problem down to the deallocation (drop) of the BindData associated with the connection. At the point of the deallocation the data in the Binds structure (the last processed row) could be one with a midnight (00:00:00) value or it could have a non-midnight value; the error happens in both cases. As long as there was a row with a midnight value processed at some point through the StatementIterator the problem will appear.

Below is a slightly edited call stack at the point when the application crashes (VS Code debugger). Specifically, the error happens when deallocating the first Time column in the bind data (there are two).

__GI_raise (@raise:45)
__GI_abort (@abort:66)
__libc_message (@__libc_message:180)
malloc_printerr (@7ffff73b547c..7ffff73b5503:3)
_int_free (@_int_free:201)
dealloc (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/alloc.rs:103)
dealloc (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/alloc.rs:174)
dealloc_buffer<u8,alloc::alloc::Global> (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/raw_vec.rs:709)
drop<u8,alloc::alloc::Global> (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/raw_vec.rs:719)
drop_in_place<alloc::raw_vec::RawVec<u8, alloc::alloc::Global>> (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:174)
drop_in_place<alloc::vec::Vec<u8>> (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:174)
drop_in_place<diesel::mysql::connection::bind::BindData> (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:174)
drop_in_place<[diesel::mysql::connection::bind::BindData]> (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:174)
drop<diesel::mysql::connection::bind::BindData> (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/vec.rs:2320)
drop_in_place<alloc::vec::Vec<diesel::mysql::connection::bind::BindData>> (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:174)
drop_in_place<diesel::mysql::connection::bind::Binds> (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:174)
drop_in_place<diesel::mysql::connection::stmt::iterator::StatementIterator> (/home/.../.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:174)
map<closure-0,mnemodia_common::models::week_slot::AccWeekSlot> (/home/.../.cargo/registry/src/github.com-1ecc6299db9ec823/diesel-1.4.4/src/mysql/connection/stmt/iterator.rs:37)
query_by_index<...> (/home/.../.cargo/registry/src/github.com-1ecc6299db9ec823/diesel-1.4.4/src/mysql/connection/mod.rs:77)
query_by_index<...> (/home/.../.cargo/registry/src/github.com-1ecc6299db9ec823/diesel-1.4.4/src/r2d2.rs:134)

At the point of raising the exception I can see that at _int_free (@_int_free:201) the nextsize variable has a value of “93825001331200”. This is probably the cause of the specific error message. In a “normal” deallocation this variable has more reasonable values.

If there are no midnight values in any row the deallocation happens without a problem.

Furthermore, I couldn’t see any obvious issues while fetching each row in the method below (/…/diesel-1.4.4/src/mysql/connection/stmt/iterator.rs). binds is populated as expected, as far as I can see.

fn populate_row_buffers(stmt: &Statement, binds: &mut Binds) -> QueryResult<Option<()>> {
    let next_row_result = unsafe { ffi::mysql_stmt_fetch(stmt.stmt.as_ptr()) };
    match next_row_result as libc::c_uint {
        ffi::MYSQL_NO_DATA => Ok(None),
        ffi::MYSQL_DATA_TRUNCATED => binds.populate_dynamic_buffers(stmt).map(Some),
        0 => {
            binds.update_buffer_lengths();
            Ok(Some(()))
        }
        _error => stmt.did_an_error_occur().map(Some),
    }
}

Checklist

  • I have already looked over the issue tracker for similar issues.
  • This issue can be reproduced on Rust’s stable channel. (Your issue will be closed if this is not the case)

I’m not sure if this error is easy to replicate on a different system/DB/table. I’m wondering if it hasn’t been reported before because it’s not common, or because Time columns aren’t used that much (probably DateTime is more common).

Thanks & regards, Alfonso

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 21 (10 by maintainers)

Commits related to this issue

Most upvoted comments

I only ever use Linux, but I can give it a try on that environment. If the tests are successful on Linux then we’d just need somebody to do the Windows part - perhaps an easy change once we see all the differences between the two versions.

That’s interesting. While debugging my error I noticed that the length member of the MYSQL_BIND structure for the Time columns was set to 48, while the length of the buffer (vector) was 40. I thought that could be due to some alignment artefact, but the fact that the length member is supposed to be set by the mysqlclient library made me a bit suspicious.

As this seems to be very suspicious I’ve digged again into the source code of mysql_time.h and it’s horrifying… Compare the definition from the central mysql github repository with the corresponding definition from the mariadb repository. Someone added a int time_zone_displacement in the mysql definition, making both incompatible. Additionally git blame reveals that this happens with the 8.0.19 release. I’ve not found any change log entry anywhere about this…

What now happens is the following:

  • diesel uses static bindings generated for an older mysqlclient version
  • This bindings contain contain a now incomplete definition of MYSQL_TIME that has only a size of 40 bytes.
  • diesel then uses this definition here to calculate the correct binding buffer size for the query result.
  • Newer libmysqlclient versions expect that the buffer has a size of 48 byte and write past the bounds of the buffer. While doing that some probably unrelated thing gets overridden. That then causes the crash in a with an unrelated error message about free.
  • Crashes do not happen with asan/valgrind because they insert padding bytes/red zones to detect such writes. (Don’t know why asan does not detect this write)

That all written comes up the question how to fix this mess. As the new field is in the end of the struct and the c standard does not allow struct field reordering it should be possible to just update our bindings to the new version. That means someone needs to run bindgen on mysql.h on different platforms and submit a PR to the mysqlclient-sys repo linked above. After that we can issue a new release that hopefully fixes this issue for newer versions of libmysqlclient. Older versions should continue to work as we now just allocate a larger buffer for them. They will only populate all but the last field, but as we initialize the buffer to be zeroed we can assume that the field will be then set to zero, which seems to be a sensible default for time_zone_displacement. Beside of that we may want to update the time handling code in diesel/mysql to handle that field as well, but that is something that could be done later. As I’m not using mysql myself and also do not have access to a windows PC I will not work on this issue any time soon. Therefore I mark this issue as help wanted, so for anyone interested feel free to submit a corresponding fix.