rails: MySQL UTF8MB4 breaks ActiveRecord schema setup
Mysql’s UTF8 support (charset name ‘utf8’) has been limited to 3-byte characters. This was changed in mysql 5.5 by adding a new charset ‘utf8mb4’ and creating an alias for ‘utf8’ of ‘utf8mb3’ (for multibyte 3 chars). It is my understanding that ‘utf8mb4’ is the “real” utf8 support, and it will likely be a popular option amongst developers. I discovered this problem by storing the text of Twitter messages where activerecord would throw an exception because mysql was rejecting the INSERT statements due to ‘invalid’ 4 byte UTF8 characters.
The current mysql2 gem does not support charset utf8mb4 but the github head version of the gem does. After setting up the new character set in database.yml and the mysql server, it is no longer possible to create a new database through the usual rake db creation tasks.
The mysql docs talk about a new limitation on database indexes for string columns when the utf8mb4 charset is in use. The limit is 191 characters for string columns using utf8mb4. (http://dev.mysql.com/doc/refman/5.6/en/charset-unicode-upgrading.html). When the schema_migrations table is created with string length 255, mysql throws an error.
$ rake db:setup
Mysql::Error: Specified key was too long; max key length is 767 bytes:
CREATE UNIQUE INDEX unique_schema_migrations
ON schema_migrations
(version
)
A slightly longer writeup is in http://donpark.org/blog/2013/02/16/rails-3-2-12-not-ready-for-mysql-5-5-utf8mb4
About this issue
- Original URL
- State: closed
- Created 11 years ago
- Comments: 49 (21 by maintainers)
Commits related to this issue
- reduce size of indexed strings so indices created on MySQL [DBA-3014] see https://github.com/rails/rails/issues/9855 — committed to zendesk/samson by jonmoter 9 years ago
- Fix issue with mysql. See https://github.com/rails/rails/issues/9855 for more detail. I just want to run away and scream. — committed to Janfred/dnstrack by Janfred 6 years ago
- Fix issue with mysql. See https://github.com/rails/rails/issues/9855 for more detail. I just want to run away and scream. — committed to Janfred/dnstrack by Janfred 6 years ago
- Fix issue with mysql. See https://github.com/rails/rails/issues/9855 for more detail. I just want to run away and scream. — committed to Janfred/dnstrack by Janfred 6 years ago
- always create table with DYNAMIC row format. see https://github.com/rails/rails/issues/9855, http://3.1415.jp/mgeu6lf5 — committed to sebastianquack/tanzfabrik by retani 6 years ago
this will set default mysql string column length to 191 instead of 255 which is the new index limit on utf8mb4 (aka real utf8).
Update: please note that either updating MySQL config and patching table creation to create dynamic row as mentioned here or using newer MySQL is a better long term solution.
Or just use PostgreSQL.
Solved. mysql 5.6 rails 4.1
in your my.cnf:
and you must create the table with the option ROW_FORMAT=DYNAMIC (by default it’s COMPACT)
here I found a monkey patch http://3.1415.jp/mgeu6lf5
FYI rails 5.2 monkey patch
What I’ve done and what I’ve learned:
This seems to work great without any weird Rails issues - if you can give up schema.rb, which I’ve always found more annoying than helpful in a large production app anyways.
I have worked around this by ALTER DATABASE nnnnn CHARACTER SET utf8mb3 But I would like an out-of-box solution so I don’t need to change the characterset in the database.
For those of you thinking “wasn’t this fixed in 8744632, why isn’t it working?”. The fix was removed in #23168, for some reason…
I think the point of the monkey patch is because it’s easy to forget to always add those options… or new team members replace old ones and no one knows that special options are always required, etc…
Sure you can do it manually, but if you want to do it once and forget it (though document the monkey patch somewhere) the patch makes sense. My solution above has also proven to work well in practice.
I still see the
Mysql2::Error: Specified key was too long; max key length is 767 bytes: CREATE UNIQUE INDEX
when performingrake db:create && rake db:migrate
with Rails 4.