oj: oj gem 3.13.15+ causes an illegal instruction on CPUs that do not support SSE4.2

After the introduction of cmpestri in a455c389c0bd09cb591d1144e9366fc3f3280da0, people with CPUs that do not support SSE4.2 cannot use the gem.

We have multiple reports of ruby abending with an Illegal Instruction from users of Discourse:

https://meta.discourse.org/t/discourse-update-keeps-failing/231862/25?u=supermathie

https://meta.discourse.org/t/discourse-update-keeps-failing/231862/33?u=supermathie

Can we add a runtime check here to ensure the instruction is supported prior to using it?

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 4
Comments: 17 (9 by maintainers)

Links to this issue

Commits related to this issue

FIX: support systems without SSE 4.2 Older CPUs do not support the instruction set used internally by the oj gem starting with version 3.13.15. This pins the version to a working one while oj patch ... — committed to discourse/discourse by SamSaffron 2 years ago
FIX: support systems without SSE 4.2 (#17505) Older CPUs do not support the instruction set used internally by the oj gem starting with version 3.13.15. This pins the version to a working one whi... — committed to discourse/discourse by SamSaffron 2 years ago
Enable the SSE4.2 flag only when the compiled executable file works properly https://github.com/ohler55/oj/issues/789 SSE4.2 instructions will not be used when it causes SEGV during configuration. — committed to Watson1978/oj by Watson1978 2 years ago
Enable the SSE4.2 flag only when the compiled executable file works properly https://github.com/ohler55/oj/issues/789 SSE4.2 instructions will not be used when it causes SEGV during configuration. — committed to Watson1978/oj by Watson1978 2 years ago
Enable the SSE4.2 flag only when the compiled executable file works properly (#791) https://github.com/ohler55/oj/issues/789 SSE4.2 instructions will not be used when it causes SEGV during configu... — committed to ohler55/oj by Watson1978 2 years ago
Enable the SSE4.2 flag only when the compiled executable file works properly (#791) https://github.com/ohler55/oj/issues/789 SSE4.2 instructions will not be used when it causes SEGV during configura... — committed to Watson1978/oj by Watson1978 2 years ago
Provide a way to disable SSE 4.2 instructions We often ship gems compiled on a SSE 4.2 machine, but some users run on older hardware. To avoid an illegal instruction error, disable SSE 4.2 instructio... — committed to stanhu/oj by stanhu 2 years ago
Perform runtime check for SSE 4.2 instructions If SSE 4.2 is not available on the system, don't attempt to use SIMD instructions. Relates to https://github.com/ohler55/oj/issues/789 — committed to stanhu/oj by stanhu 2 years ago
Perform runtime check for SSE 4.2 instructions If SSE 4.2 is not available on the system, don't attempt to use SIMD instructions. Relates to https://github.com/ohler55/oj/issues/789 — committed to stanhu/oj by stanhu 2 years ago
Perform runtime check for SSE 4.2 instructions If SSE 4.2 is not available on the system, don't attempt to use SIMD instructions. Relates to https://github.com/ohler55/oj/issues/789 — committed to stanhu/oj by stanhu 2 years ago
Perform runtime check for SSE 4.2 instructions If SSE 4.2 is not available on the system, don't attempt to use SIMD instructions. Relates to https://github.com/ohler55/oj/issues/789 — committed to stanhu/oj by stanhu 2 years ago
Perform runtime check for SSE 4.2 instructions (#795) If SSE 4.2 is not available on the system, don't attempt to use SIMD instructions. Relates to https://github.com/ohler55/oj/issues/789 — committed to ohler55/oj by stanhu 2 years ago
fix: support systems without SSE 4.2 Upgrade oj to fix https://github.com/ohler55/oj/issues/789 See https://community.openproject.org/wp/44155 — committed to opf/openproject by cbliard 2 years ago
fix: support systems without SSE 4.2 Upgrade oj to fix https://github.com/ohler55/oj/issues/789 See https://community.openproject.org/wp/44155 — committed to opf/openproject by cbliard 2 years ago
DEV: Bump oj from 3.13.14 to 3.15.0 https://github.com/ohler55/oj/issues/789 has been fixed and SSE4.2 is disabled by default. — committed to discourse/discourse by gschlager a year ago
DEV: Bump oj from 3.13.14 to 3.15.0 (#22352) https://github.com/ohler55/oj/issues/789 has been fixed and SSE4.2 is disabled by default. — committed to discourse/discourse by gschlager a year ago

Most upvoted comments

v3.13.20 is out now and disables this by default: https://github.com/ohler55/oj/pull/806

If you need this optimization, compile the gem with the --with-sse42 flag.

stanhu on Aug 8, 2022

We are also seeing many reports of this upon upgrading to 3.13.16: https://gitlab.com/gitlab-org/gitlab/-/issues/368656

eduardobonet on Jul 25, 2022

I may have spoken too soon. It seems that with the -msse42 flag, the .so contains other SSE instructions that may not be available. Using https://github.com/pkgw/elfx86exts/releases, we see:

$ ./elfx86exts /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/oj-3.13.19/lib/oj/oj.so
MODE64 (call)
CMOV (cmovne)
SSE2 (movq)
SSE41 (pinsrq)
SSE1 (movups)
SSSE3 (pshufb)
SSE3 (movddup)
SSE42 (pcmpestri)
CPU Generation: Unknown

Without the -msse4.2 flag, we get:

MODE64 (call)
SSE2 (pxor)
SSE1 (movups)
CMOV (cmovl)
CPU Generation: Intel Core

So it seems we may need to have a flag that disables -msse42 outright here.

stanhu on Aug 2, 2022

I think we can close this issue. I’ve received confirmation from users with x86 hardware without SSE instructions that 3.3.18 solves the problem.

I’ve also verified that 3.3.18 properly enables SSE 4.2 for x86 on Linux and macOS, while leaves it off for macOS arm64 platforms.

stanhu on Jul 26, 2022

How about an environment variable that is checked in extconf.rb to block use of SSE4.2?

ohler55 on Jul 21, 2022

@Watson1978 can you look into this?

ohler55 on Jul 15, 2022