mini_racer: mini_racer 0.12.0 Segmentation fault on x86_64-linux with libv8-node 21.7.2.0
Hello the team
Thank you for the update to libv8-node 21.7.2.0 yesterday. I wanted to report an error. Setup:
- mini_racer 0.12.0
- libv8-node 21.7.2.0
- ruby 3.2.3
- OS: Ubuntu 22.04.3 LTS (from CircleCI cimg/ruby:3.2.3-node)
- Rails: 6.1.7.7
- Sprockets: 4.2.1
- The command:
RAILS_ENV=production SECRET_KEY_BASE=1 bin/rails assets:clobber assets:precompile
/home/circleci/ec/vendor/ruby/3.2.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:228: [BUG] Segmentation fault at 0x00007f3f10a31008
ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0048 p:---- s:0301 e:000300 CFUNC :eval_unsafe
c:0047 p:0009 s:0295 e:000294 BLOCK /home/circleci/repo/vendor/ruby/3.2.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:228
c:0046 p:0010 s:0292 e:000291 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:348
c:0045 p:0008 s:0281 e:000280 BLOCK /home/circleci/repo/vendor/ruby/3.2.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:227 [FINISH]
c:0044 p:---- s:0278 e:000277 CFUNC :synchronize
c:0043 p:0045 s:0274 e:000273 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:225
c:0042 p:0007 s:0267 e:000266 BLOCK /home/circleci/repo/vendor/ruby/3.2.0/gems/execjs-2.9.1/lib/execjs/mini_racer_runtime.rb:11
c:0041 p:0003 s:0264 e:000262 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/execjs-2.9.1/lib/execjs/mini_racer_runtime.rb:67
c:0040 p:0035 s:0256 e:000255 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/execjs-2.9.1/lib/execjs/mini_racer_runtime.rb:10 [FINISH]
c:0039 p:---- s:0249 e:000248 CFUNC :new
c:0038 p:0038 s:0242 e:000241 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/execjs-2.9.1/lib/execjs/runtime.rb:68
c:0037 p:0013 s:0236 e:000235 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/execjs-2.9.1/lib/execjs/module.rb:27
c:0036 p:0020 s:0230 e:000229 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/terser-1.2.2/lib/terser.rb:189
c:0035 p:0065 s:0223 e:000221 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/terser-1.2.2/lib/terser.rb:229
SEGV received in SEGV handler
c:0034 p:0014 s:0212 e:000211 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/terser-1.2.2/lib/terser.rb:181
c:0033 p:0029 s:0206 e:000205 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/terser-1.2.2/lib/terser/compressor.rb:36
c:0032 p:0007 s:0198 e:000197 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/terser-1.2.2/lib/terser/compressor.rb:23
c:0031 p:0047 s:0193 e:000192 METHOD /home/circleci/repo/vendor/ruby/3.2.0/gems/sprockets-4.2.1/lib/sprockets/processor_utils.rb:84
c:0030 p:0013 s:0185 e:000184 BLOCK /home/circleci/repo/vendor/ruby/3.2.0/gems/sprockets-4.2.1/lib/sprockets/processor_utils.rb:66 [FINISH]
c:0029 p:---- s:0180 e:000179 CFUNC :reverse_each
c:0028 p:0035
Received "aborted" signal
It does work on Darwin Kernel Version 23.2.0 arm64 (Mac on arm) that compile from source.
The error is from eval -> eval_unsafe, so I suspect it is an issue with the libv8-node
https://github.com/rubyjs/mini_racer/blob/5b2a25cdb4c84c7500ed47639ef8c664f5bac26d/lib/mini_racer.rb#L228
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 9
- Comments: 105
Commits related to this issue
- Stick to mini_racer 0.9.0 for now https://github.com/rubyjs/mini_racer/issues/300 — committed to tomhughes/openstreetmap-website by tomhughes 3 years ago
- fix: downgrade mini_racer to prevent segfaults Currently when I run `rails assets:precompile` the application segfaults. This seems to be related to mini_racer. The easiest solution is to downgrade ... — committed to elasticspoon/human-essentials by elasticspoon 2 years ago
- fix: remove mini_racer to prevent segfaults (and because we don't use it) (#4478) * fix: downgrade mini_racer to prevent segfaults Currently when I run `rails assets:precompile` the application s... — committed to rubyforgood/human-essentials by elasticspoon 2 years ago
- Fix segfault, disable V8's thread-isolated allocator Fixes: https://github.com/rubyjs/mini_racer/issues/300 — committed to bnoordhuis/mini_racer by bnoordhuis 2 years ago
- Fix segfault, disable V8's thread-isolated allocator Fixes: https://github.com/rubyjs/mini_racer/issues/300 — committed to bnoordhuis/mini_racer by bnoordhuis 2 years ago
- Disable use of Linux memory protection keys V8's default thread-isolated allocator has a bug on x64 Linux. It uses memory protection keys (see `man 7 pkeys`) to write-protect JIT code memory but in ... — committed to bnoordhuis/libv8-node by bnoordhuis 2 years ago
- Fix segfault, disable V8's thread-isolated allocator Fixes: https://github.com/rubyjs/mini_racer/issues/300 — committed to bnoordhuis/mini_racer by bnoordhuis 2 years ago
This does reproduce it in docker:
I looked at the files and there is nothing special about them, the backtrace for the segfault is on cleanup.
specifically miniracer calling deallocate from the finalize_list in ruby and
free_context_rawleading to a segfault.@lloeki is this enough for you to debug, or do you need more help here? One possibility here is that somehow autoprefixer is forking processes, I looked at the code and it does not look like it.
The crash looks like is is from
if (isolate_info) { isolate_info->release(); }
@SamSaffron Yeah I think I can fit that in this week’s schedule.
It’s not so much for the Ruby version than the glibc one.
thanks so much for the instruction … CPU is running hot now 😃
If this is any help, @SamSaffron’s repro did work for me and I traced the segfault a bit further down to this line:
https://github.com/rubyjs/mini_racer/blob/5b2a25cdb4c84c7500ed47639ef8c664f5bac26d/ext/mini_racer_extension/mini_racer_extension.cc#L1434
I forced libv8-node down to 20.12.1.0 and the issue went away, so perhaps it’s possible that it’s either a bug in 21.7.2.0 or a breaking change requiring an update?
Edit: for those who don’t know (like me before yesterday!) that
isolatein the line I think is a v8::Isolate, i.e. part of V8 proper and notmini_racer.LD_PRELOADwouldn’t changeRbConfig::CONFIG['MAINLIBS']output, it’d just forcefully inject jemalloc symbols into it.In any case it appears that
RbConfig::CONFIG['MAINLIBS']doesn’t contain anything about jemalloc anyway, even when statically built against jemalloc.I couldn’t reproduce it either inside that
x86_64image: test suite passes, even when artificially stressing libv8 by throwing copious amounts of quickly hacked JS code processing to it (granted it was a real quick test).We are now seeing different and new issues:
It is going to take us a bit longer to get this v8 upgrade completed
@D-system does https://github.com/rubyjs/mini_racer/issues/300#issuecomment-2121520163 still reproduce the issue for you?
Sadly https://github.com/rubyjs/mini_racer/issues/300#issuecomment-2202876857 no longer crashes… I need a new consistent repro
oh no, and yes we just saw some of these too cc @lloeki 😢
will try debugging more this week
@tisba I think you meant 0.13.0. Yes, I just tried with 0.13.0 without success. It still seg fault in the unsafe block.
@SamSaffron apologies for the lack of updates, I started building but hit some local snags. I should have them sorted out over the weekend.
I have good news @lloeki
No longer crashes, any chance you can push this out? This may be the best possible resolution here…
I am sure this is resolvable…
I am on glibc here yeah lets see I am building now with gcc to see if it resolves before going too crazy.
Repro:
eval.js is eval.js.txt
@lloeki FYI my above sample also repros outside of docker… it crashes with a segfault in the
isolate->Dispose()line. I checked and the isolate is not in use, I guess the next step here is to compile this somehow on local with symbols so we can step into v8 and see where in Dispose this is crashing. will try a bit more to figure out if I can somehow make repro script, nothing complex is happening here just creating a context and then GC is clearing it up.@george-gca entirely unrelated. see https://github.com/rubyjs/libv8-node/pull/36#issuecomment-2045274661
I’m still not able to reproduce it in a local Docker even when I use the Docker image of the CI.
I continue to investigate on the CI directly. SSH and vi for the win!
Anyway, I track down the issue. On a brand new Rails 6.1 app:
autoprefixer-railshas onlyexecjsas dependency. In my very case,autoprefixer-railsmight have some code that seg fault in the CI only.I don’t know what else I could search.
I am indeed using puma in clustered mode, but it was working fine on 0.8.0, and starting puma in single mode gives me the same segfault in the same place. I have the suggested fork safety line in an initializer, and removing this makes no difference in any scenario.
@lloeki this looks urgent , should we roll out a release that puts us back at an earlier version while we work this out ?
Hey @D-system. A couple of question hopefully helping to narrow the issue down
Can you reproduce it with a freshly created rails application? On
aarch64-linuxmy basic checks works, as well asrails new foo && cd foo && bundle install && RAILS_ENV=production SECRET_KEY_BASE=1 bin/rails assets:clobber assets:precompileusingcimg/ruby:3.2.3-node.Can you reproduce the issue with Ruby 3.3.0?
Can you confirm if you’re using jemalloc or not? Looks like the image is at least installing libjemalloc, I haven’t checked if they build Ruby against it, or using
LD_PRELOAD, or using this up to the user. There have been various issues with jemalloc and mini_racer/v8. In case jemalloc is used, can you try without?