core: Add net.isr.maxthreads, net.isr.bindthreads, net.isr.dispatch to Tunables and adjust defaults

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

Is your feature request related to a problem? Please describe.

  • Hardware: Compulab fitlet2 - Atom E3950 CPU, dual Intel gigabit NICs.
  • Gigabit (almost) PPPoE WAN.

When running https://www.speedtest.net/ to my ISP’s server, I noticed that OPNsense shows 20-30% packet loss on WAN.

It turns out that all PPP packets are handled on only one CPU core (with my hardware and default OPNsense configuration), and the single-core performance of E3950 is just not enough.

The issue (and the solution) is described in pfSense docs, FreeBSD Bugzilla

Describe the solution you like

I solved the issue by adding the following tunables:

  • net.isr.dispatch: deferred (hybrid works too)
  • net.isr.maxthreads: -1
  • net.isr.bindthreads: 1 (not sure if it’s required, enabled just because it “makes sense”)
  1. I think OPNsense UI should show these tunables by default.
  2. I think the default values for net.isr.maxthreads and net.isr.bindthreads should be adjusted.
  3. Maybe OPNsense should show a warning on PPP configuration page.

net.isr.maxthreads: I think on a router/firewall it should be set to “all cores” (-1) by default. Also, as far as I understand, this tunable has no effect when net.isr.dispatch == direct (the default).

net.isr.bindthreads: I haven’t tested its performance impact. But if a thread is created for every CPU core, I think it makes sense to bind every thread to its own core.

net.isr.maxthreads and net.isr.bindthreads can be set only on boot, so a good default value is a bit more important than usual.

net.isr.dispatch: I’ve read in multiple places that you should avoid changing it unless you absolutely need to. Still don’t understand the details. But at least it can be changed without a reboot, so:

  • Maybe it should be “more visible” than a “tunable”? I. e. maybe it should be added to “Interfaces: Settings” page, after various hardware offload options?
  • Maybe a script could detect that PPPoE is used, together with an affected NIC, and set the tunable automatically?
  • Comments in netisr.c suggest that dispatch policy can be set per-protocol. Maybe it could be set for PPP only? (although I haven’t found how)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (8 by maintainers)

Most upvoted comments

Just to add the UK uses PPPoE as well, it is more common than you think. Would be an idea to ask people on twitter maybe and in the forum and point people to a survey to get better understanding maybe? Or as @amezin mentioned detect that PPPoE is used, when you set your wan interface to it and then apply the changes in question? Mind you not sure what these changes would do if you are using RSS though. Just some food for thought, but docs for this would be a good start and if you pick PPPoE for your WAN interface, could have some help text in the UI that says you might want to check this url to the docs page in question to help get better throughput with this connection type you have as well which would be good.

I‘m sure you can make this easier by sharing a comparison table across hardware with throughput and sysctl combinations. It’ll make changing the defaults that much easier. Thanks!

Then most of the core dev’s are not able to reproduce which makes it even harder to change defaults.

IMHO the best way would be:

  • iperf to external IPerf server, single stream, and a linux firewall (e.g. ipfire) between to check if you reach full speeds. If you have full speed on 3 times of the day this is good.
  • second would be the same test at the same time with default opnsense installation, compare the results in a sheet
  • third would be to adjust sysctl’s one by one, every time with a reboot in between and add to the sheet.

It should be easy to see which knobs would gain more throughput and at first can be added to the docs. Then let this evolve over time, let other with less throughput or evern other WAN uplink use against these values. Then, at some time, defaults can be changed.

Just my 2 cents 😃

So what’s the reason FreeBSD has this default and doesn’t listen to “authorities” on the subject?

Maybe is because freebsd is not designed to be a firewall only, I don’t know

Men you asked for documentation (all the settings are explained with pro and cons), evidence and tests, now you have it, do as you wish.

If I was rude in my first comment accept my apologies

Yeah, not really. 😉