rails: The listen gem breaks my laptop

@ahoward it might not be great, but lets try to stay civil. 🙇

@vipulnsward Yes. In fact, there’s a command to do so:

rails new thisbreaksmylaptop

One thing: please don’t spread groundless FUD about Listen. (At least first make sure Listen really is the culprit).

TL;DR - saying that Listen is shitty because of this issue (without knowing the root cause here) is like saying Rails is shitty because of a Listen issue. It only obscures the problem and discourages maintainers from putting in more personal time to try and improve things.

@ahoward - while personally I’m not offended by your comments, they’re just misleading. Listen is just a layer of workarounds that makes file monitoring portable, so if this really was a Listen issue, it would likely be reproduceable on Linux, Windows and BSD. If it isn’t, it likely isn’t a Listen issue.

Also, Listen works as reliably as the backends let it, e.g. rb-fsevent - which I’m not even a contributor of.

Sure, Listen does need another overhaul (for 4.x) - but that’s mostly unrelated to this.

Ironically though, Listen’s acceptance tests are the best way to test backends like rb-fsevent. So if there’s something messed up, I can maybe just replicate it in Listen and detect bad scenarios and blacklist certain gem version combinations (which I’ve done before).

(But I’d need some way to turn it into a Travis tests - e.g. with shell code to create the problem and detect orphaned processes).

My point: removing Listen would just mean that some poor contributors would have to waste hours to reinvent the same fixes/workarounds that Listen already has.

So it’s not constructive to create a movement of “lets drop Listen”. Simply because it’s not the root problem here. Listen is more like an API interface, so dropping it wouldn’t make sense unless the API was bad.

And just to give a sense:

Both Listen and the backend drivers started as hacks as early as 2010: https://github.com/thibaudgg/rb-fsevent/commits/d803637f19d3d1450105b5e0d4f8ed1060f98acb/lib/rb-fsevent/fsevent.rb - and there’s a long history of OSX nightmares that brave contributors have been trying to debug and patch ever since.

And many original contributors have simply moved on with their lives. If anyone wants to make the code “less shitty” - there are probably no other obstacles than being unable to contact the maintainers.

I’m not the maintainer of rb-fsevent, but I’d be glad to add a better OSX backend implementation in Listen. Or even just bumping the Listen dependency on rb-fsevent (if a fix becomes available).

I know it’s easy to jump to conclusions when we’re frustrated - sometimes it’s better to just reach out and ask first. If something works shitty, there may be a bigger reason other than “incompetence”.

Sorry for the length - just felt like a wrong I had to fix.

Still a problem!

Hi everyone, Listen maintainer here…

TL;DR - skip to end…

First, I haven’t used a Mac for over 5 years. I only use Linux, so unless I can reproduce a problem on the Mac - at least on Travis (OSX builds), I can’t fix it.

I probably can’t even reasonably run OSX in a Virtual Machine (AFAIK), so if I can’t reproduce it or write a unit/acceptance test for it, I won’t even know how something is broke.

Especially if nothing is ever reported.

Otherwise, I do release bugfixed-versions of Listen VERY quickly.

With that said …

Listen supports multiple backends. rb-fsevent is the problem child here.

Basically, most of Listen is a giant workaround for OSX-related issues - by design. So surprising issues on OSX are … not surprising. And they’re often OSX version-dependent.

You can blame Apple for a lack of decent file/dir monitoring.

One solution I’d suggest is replacing the current Darwin backend with fs-event listener from https://github.com/burke/zeus.

I’d do that myself, but 1) I don’t have time nowadays (but PRs are encouraged!) 2) I wouldn’t be able to test it, since I don’t own a Mac.

One thing that Listen does that’s special for OSX is: it runs each watched directory in a separate thread. (This was mostly to allow mapping a file back to a given watcher). I don’t know if that’s the issue here, but I’d be glad to fix+release. There are acceptance tests for OSX on Travis, so if you can reproduce it somehow, it can likely be added to the build to be resolved forever.

There are lots of hideous details (many related to OSX), so feel free to ask, open issues, etc.

As for alternatives I once researched and created a list of “watch-and-do-something” tools here: https://github.com/guard/guard/wiki/Guard-alternatives

Meanwhile, there’s polling mode, which sounds bad, but actually may perform much better than expected.

One good/bad thing about Listen is that it’s “directory based” and not “file based” (like many other file watching implementations). This means Listen has better support of removing/adding whole trees of files and editor support (especially useful for Rails users).

The downside is: it needs to scan whole trees of changes on OSX, because the current backend has no hints about which files were changed. And HFS filesystem has 1-second resolution, which makes things worse. Especially when combined with file upper/lower case OSX-specific issues.

I really considered facebook/watchman, but it didn’t solve all the problems that Listen was designed to solve (but it was pretty close). I also considered the listeners from Zeus, but again, I don’t have/use/want OSX. (I only use Linux).

TL;DR

OSX file monitoring is (was?) shitty. It also was the most problematic and often the slowest.

I doubt those things can be fixed by “a gem” instead of fixing OSX.

Trying to switch implementations is like trading one can of “known” worms with another can of “unknown” worms.

My best bet would be to transplant this:

https://github.com/burke/zeus/blob/master/go/filemonitor/filemonitor_darwin.go or this: https://github.com/burke/zeus/blob/v0.15.5/ext/fsevents/main.m to replace this: https://github.com/guard/listen/blob/master/lib/listen/adapter/darwin.rb

Feel free to mention me (@e2) if I can help/advise/etc.

A fix has been merged into the Listen gem to resolve this issue.

Ok, I came up with a workaround. Looks like it’s not only the spring-watch-listen gem, and while removing it helps slightly (I think it might have just been a placebo), it doesn’t fully solve the problem.

First I removed the listen and spring-watcher-listen gems. Then, in my environments/development.rb file, I commented out this line:

 # Use an evented file watcher to asynchronously detect changes in source code,
 # routes, locales, etc. This feature depends on the listen gem.
 # config.file_watcher = ActiveSupport::EventedFileUpdateChecker

Then, Rails seems to work fine.

I very much recommend making using the listen gem opt-in, not opt-out, as it’s very hard to debug. Also, laptop breaking probably turns many developers away from rails (it almost did for me).

The listen gem was added in #22254.

+1

Now I’m killing manually. pkill -9 fsevent_wat*

@here rb-fsevent 0.10.3 is out with a potential fix for this issue.

It’s been months that I encountered this issue and it’s only now that I found out that it’s related to my new Rails project. 😐 I ❤️ Rails but I reckon this can be a dealbreaker for new people coming into Rails.

With that said, I’m not knowledgeable enough yet to fix it, but here’s a workaround that doesn’t have to disable listen/rb-fsevent/spring altogether: we can throttle down applications (including Rails-related processes) in OSX using renice:

Fire and forget way:

$ sudo su -
$ `while true; do renice +15 -p `ps ax | grep 'rb-fsevent' | grep -v grep | awk '{print $1}' | tr 'n' ' '`; done`

… or a command line alternative: make an executable renice-rails in your $PATH with the following content:

#!/bin/sh
# filename: renice-rails
renice +15 -p `ps ax | grep 'rb-fsevent' | grep -v grep | awk '{print $1}' | tr 'n' ' '`

I quickly put up a 1 minute blog about it https://padi.github.io/articles/throttle-down-rails-processes/

Hi @benaubin, we’re discussing a complete rewrite of rb-fsevent that could potentially fix your issue.

I would also recommend upgrading to the latest macOS release (Sierra) as Apple fix some scary bugs around fsevents.

@tenderlove already addressed the tone of the previous feedback; I don’t think it’s productive to continue discussing same.

saying that Listen is shitty because of this issue (without knowing the root cause here) is like saying Rails is shitty because of a Listen issue

For the record: Rails is shitty because of this issue, regardless of which gem is the root cause – and we will consequently explore any reasonable option to make the problem go away.

@penne12 I don’t think so. We need to either fix listen, or find an alternative. If we can find a suitable alternative then we’d remove listen as default. This seems like it might have potential

the guys that are working on that gem know next to nothing about process management. a disaster.

amateur hour: https://github.com/thibaudgg/rb-fsevent/blob/d803637/lib/rb-fsevent/fsevent.rb#L55

i’m seeing this too. it is way too easy to orphan processes. the listen gem is a piece of shit and should be removed.

I’m facing this issue since a while. ruby 2.7.2 rails 6.1.0 macos 11.1

In my CPU monitor I can see ruby 99% fseventsd 0.1%

remove config.file_watcher = ActiveSupport::EventedFileUpdateChecker helps a lot. after this change ruby process goes up to 60% maximum while the browser is loading, then it goes below 10%

This issue has been automatically marked as stale because it has not been commented on for at least three months. The resources of the Rails team are limited, and so we are asking for your help. If you can still reproduce this error on the 5-2-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open. Thank you for all your contributions.

I have problems similar to those described here on macOS Mojave (10.14.4), Ruby 2 (ruby 2.6.3p62) Rails 5 (5.2.3). I could solve them by reinstalling the gems listen and rb-fsevent:

gem uninstall listen rb-fsevent
gem install listen rb-fsevent

However, I need to do this on a regular basis…

I would also agree that monitoring file events on MacOS has been miserable, at least until 10.12’s MASSIVE HFS+ metadata corruption fix, and that for end users some of that frustration is actually my fault with some unrelated bugs in rb-fsevent. Most of those issues should be resolved now, with the last being fully resolved in the rewrite (such as the ability to ignore filesystem events originating from your own process, which will be very useful).

@benaubin above and beyond the rewrite, I understand that the current version of rb-fsevent isn’t going away for quite some time and have made quite a few large changes in the latest version on master branch. It’s not released as a gem yet mostly so I can add a post install message mentioning the new gem… which doesn’t exist yet. I should probably just release and not wait. One of the fixes included makes sure the fsevent_watch doesn’t stick around longer than its parent process in the rare case it does… thus wasting resources, especially for people who don’t shut down their laptops and simply put them to sleep. EDIT: This change was not mine, and made it into the code, but doesn’t appear to have made it into a pre-compiled binary release… and thus has potentially never been used. Ouch.

Like… the changes are technically too large to just force on people as-is possibly… It’s 100% backwards compatible but the communication protocol is completely different and the callback now accepts 2 parameters instead of just 1, with the second including some seriously detailed metadata about each and every change (like telling you ahead of time, without having to check yourself, whether the change refers to a directory, file, hard link… and whether the change was simply metadata, which can be safely ignored by most use cases of something like Listen… the API provides this data for free since 10.10 and we’ve just been ignoring it until now).

If you can upgrade to sierra, please do. There’s a massive filesystem metadata corruption bug that completely breaks fsevents and had to be worked around in the most brittle and broken way possible. The code implementing the workaround legitimately scares me. We have to replace the system implementation of realpath() at runtime and it’s seriously bad juju in every way.

I’m just going to put it out there, removing all references to listen in the Gemfile fixed the problem on my computer. I recently got a new laptop with an i7, so I’ve haven’t noticed any more problems.

I have disabled spring system wide (DISABLE_SPRING=1 in my profile) and haven’t run into this issue since. It’s been weeks now. This specifically seems related to the interplay between spring and listen/rb-fsevent/fseventsd