runtime: FileSystemWatcher may cause problems in containers - inotify limits and incorrect error message
MOVED FROM https://github.com/aspnet/Home/issues/3475
Looking around the web I’m seeing years of issues with FileSystemWatcher saying “The configured user limit (n) on the number of inotify instances has been reached.”
UPDATE: Looks like https://github.com/dotnet/corefx/blob/a10890f4ffe0fadf090c922578ba0e606ebdd16c/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs#L371 will assume when inotify_add_watch fails with an ENOSPEC it must but an issue with inotify instances being out of range. In fact, ENOSPEC can also mean “the kernel failed to allocate a needed resource.” We had no way to know it was anything other than “too many files open.” The error message is misleading.
From the Man Page - The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource
Phrased differently. There’s two Error Cases and we throw a message that implies there’s just One.
This is becoming more prevalent in container situations in constrained sandboxes. I’m trying to deploy https://github.com/shanselman/superzeit (just clone and “now --public” or run locally with docker) to Zeit.co and I’m hitting this regularly. I don’t think I’m hitting a limit. I think Zeit (and others) are blocking the syscall.
I think there are two issues here:
1 We should return a different error message if inotify_add_watch fails, and then circuit break so that FileSystemWatcher doesn’t prevent the app from starting. If we CAN startup without a watch successfully, we should.
2 It seems DOTNET_USE_POLLING_FILE_WATCHER=1 is used in dotnet-watch and the aspnet file providers but the base System.IO FileSystemWatcher class doesn’t support DOTNET_USE_POLLING_FILE_WATCHER? We should probably be consistent.
If I change reloadOnChange: false in Program.cs to bypass the first watch that is set on AppSettings.json, I end up hitting it later when Razor/MVC sets up its FileWatchers. We need at a minimum, to have DOTNET_USE_POLLING_FILE_WATCHER respected everywhere. Another idea would be for a way to have FileSystemWatcher “fail gracefully.” We need to test on systems with
Unhandled Exception: System.IO.IOException: The configured user limit (8192) on the number of inotify instances has been reached.
> [0] at System.IO.FileSystemWatcher.StartRaisingEvents()
> [0] at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
> [0] at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
> [0] at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
> [0] at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
> [0] at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
> [0] at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__0_0()
> [0] at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
> [0] at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
> [0] at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
> [0] at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
> [0] at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
> [0] at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
> [0] at superzeit.Program.Main(String[] args) in /app/superzeit/Program.cs:line 17
Related issues?
- https://github.com/dotnet/corefx/issues/5660
- https://github.com/dotnet/corefx/search?q=inotify&type=Issues
- This issue may have been related https://github.com/aspnet/Mvc/issues/8299 because the FileWatcher failed to watch.
@stephentoub @natemcmaster @muratg @pranavkm
dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.1.401
Commit: 91b1c13032
Runtime Environment:
OS Name: Windows
OS Version: 10.0.17134
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.1.401\
Host (useful for support):
Version: 2.1.3-servicing-26724-03
Commit: 124038c13e
.NET Core SDKs installed:
2.1.400 [C:\Program Files\dotnet\sdk]
2.1.401 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 3
- Comments: 31 (13 by maintainers)
Hey all, specific to aspnetcore instances that are experiencing this exception, I have isolated a cause and steps to mitigate until a better solution comes along. Adding the fixes described below in a brand new project reduced the watchers used from around 93 to around 26. This only pertains to aspnetcore and I’ll not rehash the other tools to mitigate the issue like
reloadOnChange: true
etc.tl;dr version: Taghelpers that use
asp-append-version="true"
add watchers, removing this or setting tofalse
prevents the watcher creation. You can also dump watchers on startup to keep the value low, by clearing fileproviders and preventing view file recompiling (only affects Development env) like this:Without the above snippet, I noticed that there were about 54-58 watchers being used at startup.
To get the number of watchers being used (+ ~4 watchers that are not used by dotnet), you can add the following to your Dockerfile in the
aspnetcore-runtime
build step:RUN apt-get update -y && apt-get install -y procps lsof
Following that, we can either check via the terminal for the docker container by running
lsof | grep inotify | wc -l
or we can add a ShellHelper class.If going the ShellHelper route, adding the following JsonResult to a controller, we can verify in the browser:
With the asp-append-version properties, you’ll see the watches jumping by about 40 on a page request that uses the
<link ..>
or<script ..>
assets. After removing the properties, the value should not jump.YMMV but I hope that this will help anyone else that has been banging their head on their desk for months trying to solve for this issue. I will add a full repro if desired.
@kylef000 Thanks again -
By running the script that specific container does have privileged access to the host (in my case a Linux VM) which is needed to be able to alter the hosts inotify limits, however the container immediately stops and is discarded (actually just noticed I am missing the --rm flag to make this auto discard) the change to the host remains though.
My service containers however run in normal access (none privileged) but they inherit the inotify limit of the host which is now increased, they dont run in privileged mode.
Granted it is definitely a work around and I think either the Linux VM that Docker fires up should by default have a higher inotify limit and/or the ASP.NET Core FileWatchers need some attention to see why they are using so many of them.
Cheers