ddev: Database corruption on host/docker daemon restart "Failed to start sitename: db service health check timed "

What happened (or feature request):

We sometimes see "Failed to start sitename: db service health check timed " situations. The log often has something like “InnoDB: The log sequence number 1600607 in ibdata file do not match the log sequence number 10696720565 in the ib_logfiles!” and implies a crash.

Perhaps as in https://stackoverflow.com/questions/19378565/log-sequence-number-in-ibdata-files-does-not-match

This may be happening on a host restart; the db container may not have a reasonable shutdown sequence, so we end up with a corrupted db. We could delete the logfiles on start (so they’d be created automatically). There are probably other options.

Note: This is almost always caused by a corrupted database, most often in a larger database. Since v0.16.0, this is generally only caused by docker being shut down in an ungraceful way. Unfortunately, both Docker for WIndows and Docker for Mac shut down without notifying the container during upgrade, with a manual Docker exit, or at system shutdown. To fix this, remove the database with ddev remove --remove-data and then ddev start and re-import the db. It can be avoided by stopping or remove-ing your projects before letting Docker exit.

What you expected to happen:

How to reproduce this:

Version: Please include the output of `ddev version`, `docker version` and the project’s .ddev/config.yaml.

Anything else do we need to know:

Please use a complexity rating of 1-5 (5 is high) for a feature request. (High complexity implies more PR planning)

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 40 (34 by maintainers)

Commits related to this issue

Add innodb_force_recovery = 1, fixes drud/ddev#748 — committed to rfay/mariadb-local by rfay 6 years ago

Most upvoted comments

thanks for this thread.

I had to use this when windows computer locked up and of course nothing i did with dockers cleaned up the crashed mess…

create a file with .ddev/mysql/innodb.cnf [mysqld] innodb_force_recovery = 1

site is now working again.

garske on Jul 11, 2018

Any way I can restore the data on the db image?

Ok, the innodb_force recovery trick worked:

cat .ddev/mysql/innodb.cnf
[mysqld]
innodb_force_recovery = 1

Tuurlijk on Jun 14, 2018

@mac-dev are you using v1.0.0? https://github.com/drud/ddev/issues/748, which went in there, should make things a lot better I’m hoping, would love to hear your experience.

rfay on Jul 23, 2018

I do have some random crashes, recently while adding a field in a drupal content type. I couldn’t see any docker restarts at that moment and the web container was still up and running. Now I guess this might be some kind of db-size limit as our db is about 3GB excluding cash tables and thows can grow a lot too. Thanks for the Info from this thread I can now start debugging this instead of reimporting the db everytime.

mac-dev on Jul 23, 2018