aspire: Docker Networks are not being cleaned up (again)
Is there an existing issue for this?
- I have searched the existing issues
Describe the bug
I followed https://github.com/dotnet/aspire/issues/7215 and applied the fix when it was released. Everything seemed better.
Now using dotnet 9 + aspire 9.3, the problem appears to be back. I had zero problems on a project with just sql server, but once I added redis, I’m getting “failed to create network” at least once a day.
The ironic part is i’m not even using redis yet past setting it up in Program.cs.
fail: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
could not create a network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 2, "Network": "default-aspire-network-1b9lq4ehg4", "error": "object not found\nnetwork not found\ndocker command 'CreateNetwork' returned with non-zero exit code 1: command output: Stdout: '' Stderr: 'Error response from daemon: all predefined address pools have been fully subnetted\n'"}
Expected Behavior
Restarting AppHost should not exhaust docker networks.
Steps To Reproduce
app host:
<PackageReference Include="Aspire.Hosting.Redis" Version="9.3.0" />
var redis = builder.AddRedis("redis")
.WithImageTag("8.0.2")
.WithRedisInsight()
.WithDataVolume("redis-data")
.WithPersistence(TimeSpan.FromSeconds(1));
web:
<PackageReference Include="Aspire.StackExchange.Redis" Version="9.3.0" />
builder.AddRedisClient(connectionName: "redis");
Exceptions (if any)
No response
.NET Version info
No response
Anything else?
No response
About this issue
- Original URL
- State: open
- Created a year ago
- Reactions: 1
- Comments: 40 (22 by maintainers)
In the next Aspire release (or current main), we’ll create an intentionally persistent network if you have any persistent container resources. While it does mean we’ll intentionally leave a network running, we’ll also re-use it on the next run, meaning we won’t keep growing the number of networks if we’re not able to clean up resources properly.
It seems to be the persistent containers that retain the aspire networks. I just cleaned up a few days ago (10 aspire networks lying around) and now I’m back to 3 after a day or two. But this time I looked a bit closer and I can see that all five of my persistent containers are each connected to all the aspire networks, including the bridge.
@davidfowl @karolz-ms
dotnet watchon Windows isn’t actually fixed; I’ve confirmed that ctrl+r is still killing the DCP processes.@MeikelLP are you able to add any additional subnet ranges to the list of pools you have configured? The configured ranges don’t have to be contiguous, so you could add any sub range available to you from the private network ranges (
10.0.0.0/8,172.16.0.0/12, or192.168.0.0/16). If you can’t add additional subnet ranges, you might consider shrinking the size of networks allocated from the pool to25instead of24. That’d let you run up to 8 custom networks with 128 IPs each instead of 4 networks with 256 IPs (obviously the viability of using smaller networks depends on your use cases and how many containers you expect to need to connect to a network).I believe the current main branch will improve the experience for you (we’ve made changes to be more aggressive about cleaning up abandoned container resources from past runs as well as the change to use a persistent network if there are persistent container resources reducing the overall potential for churning a new network every run), but I’d still suggest a slightly higher maximum capacity will give a better overall experience when interacting with container based solutions such as Aspire or Docker Compose.
As expected, MacBook handles the network cleanup much better.
I got as high as two networks, but it went back down to one.
I’ve confirmed that Ctrl+R works as expected on Mac with signal sending. I definitely appreciate how much of a pain it is to get graceful shutdown working on Windows; DCP was just killing Windows processes on shutdown until very recently as well.
@kijanawoodard ironically, you may have a better overall experience with a change we’ve recently made in Aspire main that will intentionally leave a persistent network running if you have any persistent container resources. Since we re-use the same persistent network if it exists, we won’t keep creating new ones each run.
We’ll still potentially leak container resources, but we also have a cleanup step that runs on startup and looks for abandoned container resources, so they should get removed after the AppHost restarts.
@danegsta Feel free to send PR if you have a solution on mind that works.
I have personally had no issues with this on my MacBook pro M3. Not sure if that helps. Back in 9.0 it was a huge issue