core: Add net.isr.maxthreads, net.isr.bindthreads, net.isr.dispatch to Tunables and adjust defaults
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
- I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
- I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue
Is your feature request related to a problem? Please describe.
- Hardware: Compulab fitlet2 - Atom E3950 CPU, dual Intel gigabit NICs.
- Gigabit (almost) PPPoE WAN.
When running https://www.speedtest.net/ to my ISP’s server, I noticed that OPNsense shows 20-30% packet loss on WAN.
It turns out that all PPP packets are handled on only one CPU core (with my hardware and default OPNsense configuration), and the single-core performance of E3950 is just not enough.
The issue (and the solution) is described in pfSense docs, FreeBSD Bugzilla
Describe the solution you like
I solved the issue by adding the following tunables:
net.isr.dispatch
:deferred
(hybrid
works too)net.isr.maxthreads
:-1
net.isr.bindthreads
:1
(not sure if it’s required, enabled just because it “makes sense”)
- I think OPNsense UI should show these tunables by default.
- I think the default values for
net.isr.maxthreads
andnet.isr.bindthreads
should be adjusted. - Maybe OPNsense should show a warning on PPP configuration page.
net.isr.maxthreads
: I think on a router/firewall it should be set to “all cores” (-1
) by default. Also, as far as I understand, this tunable has no effect when net.isr.dispatch
== direct
(the default).
net.isr.bindthreads
: I haven’t tested its performance impact. But if a thread is created for every CPU core, I think it makes sense to bind every thread to its own core.
net.isr.maxthreads
and net.isr.bindthreads
can be set only on boot, so a good default value is a bit more important than usual.
net.isr.dispatch
: I’ve read in multiple places that you should avoid changing it unless you absolutely need to. Still don’t understand the details. But at least it can be changed without a reboot, so:
- Maybe it should be “more visible” than a “tunable”? I. e. maybe it should be added to “Interfaces: Settings” page, after various hardware offload options?
- Maybe a script could detect that PPPoE is used, together with an affected NIC, and set the tunable automatically?
- Comments in netisr.c suggest that dispatch policy can be set per-protocol. Maybe it could be set for PPP only? (although I haven’t found how)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (8 by maintainers)
Just to add the UK uses PPPoE as well, it is more common than you think. Would be an idea to ask people on twitter maybe and in the forum and point people to a survey to get better understanding maybe? Or as @amezin mentioned detect that PPPoE is used, when you set your wan interface to it and then apply the changes in question? Mind you not sure what these changes would do if you are using RSS though. Just some food for thought, but docs for this would be a good start and if you pick PPPoE for your WAN interface, could have some help text in the UI that says you might want to check this url to the docs page in question to help get better throughput with this connection type you have as well which would be good.
I‘m sure you can make this easier by sharing a comparison table across hardware with throughput and sysctl combinations. It’ll make changing the defaults that much easier. Thanks!
Then most of the core dev’s are not able to reproduce which makes it even harder to change defaults.
IMHO the best way would be:
It should be easy to see which knobs would gain more throughput and at first can be added to the docs. Then let this evolve over time, let other with less throughput or evern other WAN uplink use against these values. Then, at some time, defaults can be changed.
Just my 2 cents 😃
Maybe is because freebsd is not designed to be a firewall only, I don’t know
Men you asked for documentation (all the settings are explained with pro and cons), evidence and tests, now you have it, do as you wish.
If I was rude in my first comment accept my apologies
Yeah, not really. 😉