* [Intel-wired-lan] Does the 'igb` kernel module support setting 2-Tuple filters (aka `--config-ntuple`) on a i210 NIC? @ 2020-04-30 19:22 Dan Williams 2020-04-30 23:48 ` Vinicius Costa Gomes 0 siblings, 1 reply; 5+ messages in thread From: Dan Williams @ 2020-04-30 19:22 UTC (permalink / raw) To: intel-wired-lan Does the 'igb` kernel module support setting "2-Tuple filters" (aka `--config-ntuple`, aka RFS) on a I-210IC NIC? - Is this the appropriate mailing list? - If not, which module *should* we be using instead? - If so, how do we enable it in the 'igb' driver? *.1. Context: * Hey, all, we're running into a very perplexing configuration issue, while trying to tune our 'igb' driver, and the documentation out there is sparse. All the examples we've found come up dry. (either by throwing errors with our setup, or emitting nothing but opaque error messages: "Operation not supported" "invalid argument") Hopefully, someone on the list can point us in the right direction. We have a computer logging a high rate of ethernet packets ( 25k packets/sec ~70 Mb/sec); But we're having trouble convincing the hardware to receive all of these packets, at a sustained rate -- specifically we're dropping packets while processing through the kernel layers. We're currently attempting to optimize the network stack, but we're having trouble setting the driver parameters... which is what this message is all about. *.2. Platform Summary:* Hardware: Advantech 3500 <https://www.advantech.com/products/1-2jkd2d/ark-3500/mod_adb8f9a9-4b1b-4cf5-84ba-9e135c099c43> CPU ($ lscpu): Architecture: x86_64 CPU family: 6 Model: 58 Model name: Intel(R) Core(TM) i7-3610QE CPU @ 2.30GHz NIC ($ lspci -vs 02:00.0) 02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) Flags: bus master, fast devsel, latency 0, IRQ 18 OS ($ lsb_release -a) Ubuntu 16.04.6 LTS Kernel (`uname -r`): 4.15.0-88-lowlatency Kernel Module ($ modinfo igb) filename: /lib/modules/4.15.0-88-lowlatency/kernel/drivers/net/ethernet/intel/igb/igb.ko version: 5.4.0-k license: GPL Ethtool Version ($ ethtool --version) ethtool version 4.5 *.3. What have we tried so far:* .3.a. The NIC supports what we want to do: The data sheet, <https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=2ahUKEwj_nona25DpAhWPoHIEHfvYBWcQFjAAegQIARAB&url=https%3A%2F%2Fwww.intel.com%2Fcontent%2Fwww%2Fus%2Fen%2Fembedded%2Fproducts%2Fnetworking%2Fi210-ethernet-controller-datasheet.html&usg=AOvVaw1N7hqg0JJAaqXsomLAUhfB> in section 7.1.2.4 "2-Tuple Filters", says: > The 2-tuple filters are configured via the TTQF (See Section 8.11.3), IMIR > (See Section 8.11.1) and > IMIR_EXT (See Section 8.11.2) registers as follows (per filter): > Am I correct in assuming these are the mechanisms the 'igb' driver is interfacing with? .3.b. What is the flow table currenttly? > # ethtool --show-rxfh enp2s0 > RX flow hash indirection table for enp2s0 with 4 RX ring(s): > 0: 0 0 0 0 0 0 0 0 > 8: 0 0 0 0 0 0 0 0 > 16: 0 0 0 0 0 0 0 0 > 24: 0 0 0 0 0 0 0 0 > 32: 0 0 0 0 0 0 0 0 > 40: 0 0 1 1 1 1 1 1 > 48: 1 1 1 1 1 1 1 1 > 56: 1 1 1 1 1 1 1 1 > 64: 1 1 1 1 1 1 1 1 > 72: 1 1 1 1 1 1 1 1 > 80: 1 1 1 1 1 2 2 2 > 88: 2 2 2 2 2 2 2 2 > 96: 2 2 2 2 2 2 2 2 > 104: 2 2 2 2 2 2 2 2 > 112: 2 2 2 2 2 2 2 2 > 120: 2 2 2 2 2 2 2 2 > RSS hash key: > Operation not supported > "Operation not supported" -- does this mean the NIC has RSS routing hard-coded? And we cannot change the hash-function? Or is RSS just hard-coded? .3.c. Current "Hash flow" settings: > # ethtool -n enp2s0 rx-flow-hash udp4 > UDP over IPV4 flows use these fields for computing Hash flow key: > IP SA > IP DA > L4 bytes 0 & 1 [TCP/UDP src port] > L4 bytes 2 & 3 [TCP/UDP dst port] > > # sudo ethtool -n enp2s0 > 4 RX rings available > Total 0 rules > .3.d. Enable ntuple features: > # ethtool --show-features enp2s0 | grep ntuple > ntuple-filters: on > .3.e. Add ntuple rule: Commands Tried: > # ethtool -U enp2s0 flow-type udp4 src-ip 192.168.3.43 dst-ip 0.0.0.0 > src-port 555 dst-port 2368 action -1 > rmgr: Cannot insert RX class rule: Invalid argument > # ethtool -U enp2s0 flow-type udp4 src-ip 192.168.3.43 action 1 > rmgr: Cannot insert RX class rule: Invalid argument > # ethtool -U enp2s0 flow-type ip4 src-ip 192.168.3.43 action 1 > rmgr: Cannot insert RX class rule: Invalid argument > # ethtool -U enp2s0 flow-type ip4 src-ip 192.168.3.43 action 1 loc 4 > rmgr: Cannot insert RX class rule: Invalid argument > .3.f. More Interface info: > # ethtool -i enp2s0 > driver: igb > version: 5.4.0-k > firmware-version: 3.16, 0x800004ad > expansion-rom-version: > bus-info: 0000:02:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > # ethtool -t enp2s0 > The test result is PASS > The test extra info: > Register test (offline) 0 > Eeprom test (offline) 0 > Interrupt test (offline) 0 > Loopback test (offline) 0 > Link test (on/offline) 0 > Daniel Williams | Software Engineer dwilliams at nextdroid.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20200430/d221c541/attachment.html> ^ permalink raw reply [flat|nested] 5+ messages in thread
* [Intel-wired-lan] Does the 'igb` kernel module support setting 2-Tuple filters (aka `--config-ntuple`) on a i210 NIC? 2020-04-30 19:22 [Intel-wired-lan] Does the 'igb` kernel module support setting 2-Tuple filters (aka `--config-ntuple`) on a i210 NIC? Dan Williams @ 2020-04-30 23:48 ` Vinicius Costa Gomes 2020-05-04 13:58 ` Dan Williams 0 siblings, 1 reply; 5+ messages in thread From: Vinicius Costa Gomes @ 2020-04-30 23:48 UTC (permalink / raw) To: intel-wired-lan Dan Williams <dwilliams@nextdroid.com> writes: > Does the 'igb` kernel module support setting "2-Tuple filters" (aka > `--config-ntuple`, aka RFS) on a I-210IC NIC? > - Is this the appropriate mailing list? > - If not, which module *should* we be using instead? > - If so, how do we enable it in the 'igb' driver? > > > *.1. Context: * > Hey, all, we're running into a very perplexing configuration issue, while > trying to tune our 'igb' driver, and the documentation out there is > sparse. All the examples we've found come up dry. (either by throwing > errors with our setup, or emitting nothing but opaque error messages: > "Operation not supported" "invalid argument") Hopefully, someone on the > list can point us in the right direction. > > We have a computer logging a high rate of ethernet packets ( 25k > packets/sec ~70 Mb/sec); But we're having trouble convincing the hardware > to receive all of these packets, at a sustained rate -- specifically we're > dropping packets while processing through the kernel layers. We're > currently attempting to optimize the network stack, but we're having > trouble setting the driver parameters... which is what this message is all > about. That's weird. That packet rate is not *that* high, the Linux kernel should be able to handle that fine. Can you give more details of the workload you are running? > > *.2. Platform Summary:* > Hardware: > Advantech 3500 > <https://www.advantech.com/products/1-2jkd2d/ark-3500/mod_adb8f9a9-4b1b-4cf5-84ba-9e135c099c43> > CPU ($ lscpu): > Architecture: x86_64 > CPU family: 6 > Model: 58 > Model name: Intel(R) Core(TM) i7-3610QE CPU @ 2.30GHz > NIC ($ lspci -vs 02:00.0) > 02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network > Connection (rev 03) > Flags: bus master, fast devsel, latency 0, IRQ 18 > OS ($ lsb_release -a) > Ubuntu 16.04.6 LTS > Kernel (`uname -r`): > 4.15.0-88-lowlatency > Kernel Module ($ modinfo igb) > filename: > /lib/modules/4.15.0-88-lowlatency/kernel/drivers/net/ethernet/intel/igb/igb.ko > version: 5.4.0-k > license: GPL > Ethtool Version ($ ethtool --version) > ethtool version 4.5 There had been a lot of improvements in the network stack in the last 4 years, trying with a recent kernel, if possible, would be useful to know if the issue you are seeing persists. > > *.3. What have we tried so far:* > .3.a. The NIC supports what we want to do: > The data sheet, > <https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=2ahUKEwj_nona25DpAhWPoHIEHfvYBWcQFjAAegQIARAB&url=https%3A%2F%2Fwww.intel.com%2Fcontent%2Fwww%2Fus%2Fen%2Fembedded%2Fproducts%2Fnetworking%2Fi210-ethernet-controller-datasheet.html&usg=AOvVaw1N7hqg0JJAaqXsomLAUhfB> > in section 7.1.2.4 "2-Tuple Filters", says: > >> The 2-tuple filters are configured via the TTQF (See Section 8.11.3), IMIR >> (See Section 8.11.1) and >> IMIR_EXT (See Section 8.11.2) registers as follows (per filter): >> The problem is that the NIC supports the 2-tuple filters, but support for using them in Linux was never added. > > Am I correct in assuming these are the mechanisms the 'igb' driver is > interfacing with? You are right. > > .3.b. What is the flow table currenttly? > >> # ethtool --show-rxfh enp2s0 >> RX flow hash indirection table for enp2s0 with 4 RX ring(s): >> 0: 0 0 0 0 0 0 0 0 >> 8: 0 0 0 0 0 0 0 0 >> 16: 0 0 0 0 0 0 0 0 >> 24: 0 0 0 0 0 0 0 0 >> 32: 0 0 0 0 0 0 0 0 >> 40: 0 0 1 1 1 1 1 1 >> 48: 1 1 1 1 1 1 1 1 >> 56: 1 1 1 1 1 1 1 1 >> 64: 1 1 1 1 1 1 1 1 >> 72: 1 1 1 1 1 1 1 1 >> 80: 1 1 1 1 1 2 2 2 >> 88: 2 2 2 2 2 2 2 2 >> 96: 2 2 2 2 2 2 2 2 >> 104: 2 2 2 2 2 2 2 2 >> 112: 2 2 2 2 2 2 2 2 >> 120: 2 2 2 2 2 2 2 2 >> RSS hash key: >> Operation not supported >> > > "Operation not supported" -- does this mean the NIC has RSS routing > hard-coded? And we cannot change the hash-function? > Or is RSS just hard-coded? IIRC the function and hash key cannot be changed, the only thing that can be changed is the indirection table, i.e. assigning different "target" queues to different hash values. > > > .3.c. Current "Hash flow" settings: > >> # ethtool -n enp2s0 rx-flow-hash udp4 >> UDP over IPV4 flows use these fields for computing Hash flow key: >> IP SA >> IP DA >> L4 bytes 0 & 1 [TCP/UDP src port] >> L4 bytes 2 & 3 [TCP/UDP dst port] >> > > >> # sudo ethtool -n enp2s0 >> 4 RX rings available >> Total 0 rules >> > > > .3.d. Enable ntuple features: > >> # ethtool --show-features enp2s0 | grep ntuple >> ntuple-filters: on >> > > .3.e. Add ntuple rule: Commands Tried: > >> # ethtool -U enp2s0 flow-type udp4 src-ip 192.168.3.43 dst-ip 0.0.0.0 >> src-port 555 dst-port 2368 action -1 >> rmgr: Cannot insert RX class rule: Invalid argument >> # ethtool -U enp2s0 flow-type udp4 src-ip 192.168.3.43 action 1 >> rmgr: Cannot insert RX class rule: Invalid argument >> > # ethtool -U enp2s0 flow-type ip4 src-ip 192.168.3.43 action 1 >> rmgr: Cannot insert RX class rule: Invalid argument >> # ethtool -U enp2s0 flow-type ip4 src-ip 192.168.3.43 action 1 loc 4 >> rmgr: Cannot insert RX class rule: Invalid argument >> Right now, only filters for ethernet addresses, ethtype, VLAN ID and PCP are implemented. I agree that returning -EINVAL is not helpful. > > .3.f. More Interface info: > >> # ethtool -i enp2s0 >> driver: igb >> version: 5.4.0-k >> firmware-version: 3.16, 0x800004ad >> expansion-rom-version: >> bus-info: 0000:02:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: yes >> > > # ethtool -t enp2s0 >> The test result is PASS >> The test extra info: >> Register test (offline) 0 >> Eeprom test (offline) 0 >> Interrupt test (offline) 0 >> Loopback test (offline) 0 >> Link test (on/offline) 0 >> > > > > > > > Daniel Williams | Software Engineer > dwilliams at nextdroid.com > _______________________________________________ > Intel-wired-lan mailing list > Intel-wired-lan at osuosl.org > https://lists.osuosl.org/mailman/listinfo/intel-wired-lan -- Vinicius ^ permalink raw reply [flat|nested] 5+ messages in thread
* [Intel-wired-lan] Does the 'igb` kernel module support setting 2-Tuple filters (aka `--config-ntuple`) on a i210 NIC? 2020-04-30 23:48 ` Vinicius Costa Gomes @ 2020-05-04 13:58 ` Dan Williams 2020-05-04 14:20 ` Paul Menzel 2020-05-04 21:14 ` Alexander Duyck 0 siblings, 2 replies; 5+ messages in thread From: Dan Williams @ 2020-05-04 13:58 UTC (permalink / raw) To: intel-wired-lan > > > We have a computer logging a high rate of ethernet packets ( 25k >> > packets/sec ~70 Mb/sec); But we're having trouble convincing the >> hardware >> > to receive all of these packets, at a sustained rate -- specifically >> we're >> > dropping packets while processing through the kernel layers. We're >> > currently attempting to optimize the network stack, but we're having >> > trouble setting the driver parameters... which is what this message is >> all >> > about. >> > > That's weird. That packet rate is not *that* high, the Linux kernel > should be able to handle that fine. > Can you give more details of the workload you are running? > Okay, in more detail: we have two groups of incoming streams: (for the minimum setup to cause a problem) - 4x Camera Streams each transmits a 3.2mb image every .1 s, split into jumbo frames (mtu is set to the full 9000) - Constant stream of data from a Lidar at 18k packets / sec. Each packet is 1206 bytes long. - Both streams continue steady-state, indefinitely (we have verified behavior out to 4 hours so far) We receive all of these over ethernet, and routed to a single network port on a single NIC. The driver is the 'igb' kernel module, as supplied from ubuntu. The OS is Ubuntu 16.04 LTS with a 4.15.0-88-lowlatency kernel. ---- Biggest Problem: Over time decay of packet processing. We've been working on this for a couple of weeks; when the processes start we're logging the full data rate (~24kpps) but over time, something slows down, and our logging rate shrinks. (on the order of 20 packets / second / minute, consistently falling over hours. ... after the first hour, we've lost 500 pps, after the second hour, 1kpps... etc.) Our user-land process simply isn't seeing the full count of packets -- we have debug code that reads it from the OS, and then immediately drops the buffer on the floor. Generally, we see drops in netstat, but not in the driver. (i.e. from `ethtool -S | grep rx_*`) So, our tentative guess is that we want to tune some parameters, somewhere in the kernel or network driver to help out the kernel. Ideas welcome, of course :) ( We are *also* dropping from the ring buffer, when both the lidar stream and a camera stream are assigned to the same queue, but that looks like a releated but separate issue) Things we've tried / checked: - irq alignment -- they're already reasonable set - cpu assignment (via `taskset`) - change processor task scheduling / priority (no effect) Ideas: - what is the default hash algorithm? if we can't change it, maybe we can work around it? - is there any other way to set the queue settings on this network card? Debug tool? Rebuilding the kernel module with custom settings? - Our hardware also has an 82579 NIC as well -- would you guys recommend we use that NIC, instead? - Do other network cards / chipsets have better support under linux? Particularly when tuning input queues? Daniel Williams | Software Engineer dwilliams at nextdroid.com On Thu, Apr 30, 2020 at 7:48 PM Vinicius Costa Gomes < vinicius.gomes@intel.com> wrote: > Dan Williams <dwilliams@nextdroid.com> writes: > > > Does the 'igb` kernel module support setting "2-Tuple filters" (aka > > `--config-ntuple`, aka RFS) on a I-210IC NIC? > > - Is this the appropriate mailing list? > > - If not, which module *should* we be using instead? > > - If so, how do we enable it in the 'igb' driver? > > > > > > *.1. Context: * > > Hey, all, we're running into a very perplexing configuration issue, while > > trying to tune our 'igb' driver, and the documentation out there is > > sparse. All the examples we've found come up dry. (either by throwing > > errors with our setup, or emitting nothing but opaque error messages: > > "Operation not supported" "invalid argument") Hopefully, someone on > the > > list can point us in the right direction. > > > > We have a computer logging a high rate of ethernet packets ( 25k > > packets/sec ~70 Mb/sec); But we're having trouble convincing the > hardware > > to receive all of these packets, at a sustained rate -- specifically > we're > > dropping packets while processing through the kernel layers. We're > > currently attempting to optimize the network stack, but we're having > > trouble setting the driver parameters... which is what this message is > all > > about. > > That's weird. That packet rate is not *that* high, the Linux kernel > should be able to handle that fine. > > Can you give more details of the workload you are running? > > > > > *.2. Platform Summary:* > > Hardware: > > Advantech 3500 > > < > https://www.advantech.com/products/1-2jkd2d/ark-3500/mod_adb8f9a9-4b1b-4cf5-84ba-9e135c099c43 > > > > CPU ($ lscpu): > > Architecture: x86_64 > > CPU family: 6 > > Model: 58 > > Model name: Intel(R) Core(TM) i7-3610QE CPU @ 2.30GHz > > NIC ($ lspci -vs 02:00.0) > > 02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network > > Connection (rev 03) > > Flags: bus master, fast devsel, latency 0, IRQ 18 > > OS ($ lsb_release -a) > > Ubuntu 16.04.6 LTS > > Kernel (`uname -r`): > > 4.15.0-88-lowlatency > > Kernel Module ($ modinfo igb) > > filename: > > > /lib/modules/4.15.0-88-lowlatency/kernel/drivers/net/ethernet/intel/igb/igb.ko > > version: 5.4.0-k > > license: GPL > > Ethtool Version ($ ethtool --version) > > ethtool version 4.5 > > There had been a lot of improvements in the network stack in the last 4 > years, trying with a recent kernel, if possible, would be useful to know > if the issue you are seeing persists. > > > > > *.3. What have we tried so far:* > > .3.a. The NIC supports what we want to do: > > The data sheet, > > < > https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=2ahUKEwj_nona25DpAhWPoHIEHfvYBWcQFjAAegQIARAB&url=https%3A%2F%2Fwww.intel.com%2Fcontent%2Fwww%2Fus%2Fen%2Fembedded%2Fproducts%2Fnetworking%2Fi210-ethernet-controller-datasheet.html&usg=AOvVaw1N7hqg0JJAaqXsomLAUhfB > > > > in section 7.1.2.4 "2-Tuple Filters", says: > > > >> The 2-tuple filters are configured via the TTQF (See Section 8.11.3), > IMIR > >> (See Section 8.11.1) and > >> IMIR_EXT (See Section 8.11.2) registers as follows (per filter): > >> > > The problem is that the NIC supports the 2-tuple filters, but support > for using them in Linux was never added. > > > > > Am I correct in assuming these are the mechanisms the 'igb' driver is > > interfacing with? > > You are right. > > > > > .3.b. What is the flow table currenttly? > > > >> # ethtool --show-rxfh enp2s0 > >> RX flow hash indirection table for enp2s0 with 4 RX ring(s): > >> 0: 0 0 0 0 0 0 0 0 > >> 8: 0 0 0 0 0 0 0 0 > >> 16: 0 0 0 0 0 0 0 0 > >> 24: 0 0 0 0 0 0 0 0 > >> 32: 0 0 0 0 0 0 0 0 > >> 40: 0 0 1 1 1 1 1 1 > >> 48: 1 1 1 1 1 1 1 1 > >> 56: 1 1 1 1 1 1 1 1 > >> 64: 1 1 1 1 1 1 1 1 > >> 72: 1 1 1 1 1 1 1 1 > >> 80: 1 1 1 1 1 2 2 2 > >> 88: 2 2 2 2 2 2 2 2 > >> 96: 2 2 2 2 2 2 2 2 > >> 104: 2 2 2 2 2 2 2 2 > >> 112: 2 2 2 2 2 2 2 2 > >> 120: 2 2 2 2 2 2 2 2 > >> RSS hash key: > >> Operation not supported > >> > > > > "Operation not supported" -- does this mean the NIC has RSS routing > > hard-coded? And we cannot change the hash-function? > > Or is RSS just hard-coded? > > IIRC the function and hash key cannot be changed, the only thing that > can be changed is the indirection table, i.e. assigning different > "target" queues to different hash values. > > > > > > > .3.c. Current "Hash flow" settings: > > > >> # ethtool -n enp2s0 rx-flow-hash udp4 > >> UDP over IPV4 flows use these fields for computing Hash flow key: > >> IP SA > >> IP DA > >> L4 bytes 0 & 1 [TCP/UDP src port] > >> L4 bytes 2 & 3 [TCP/UDP dst port] > >> > > > > > >> # sudo ethtool -n enp2s0 > >> 4 RX rings available > >> Total 0 rules > >> > > > > > > .3.d. Enable ntuple features: > > > >> # ethtool --show-features enp2s0 | grep ntuple > >> ntuple-filters: on > >> > > > > .3.e. Add ntuple rule: Commands Tried: > > > >> # ethtool -U enp2s0 flow-type udp4 src-ip 192.168.3.43 dst-ip 0.0.0.0 > >> src-port 555 dst-port 2368 action -1 > >> rmgr: Cannot insert RX class rule: Invalid argument > >> # ethtool -U enp2s0 flow-type udp4 src-ip 192.168.3.43 action 1 > >> rmgr: Cannot insert RX class rule: Invalid argument > >> > > # ethtool -U enp2s0 flow-type ip4 src-ip 192.168.3.43 action 1 > >> rmgr: Cannot insert RX class rule: Invalid argument > >> # ethtool -U enp2s0 flow-type ip4 src-ip 192.168.3.43 action 1 loc 4 > >> rmgr: Cannot insert RX class rule: Invalid argument > >> > > Right now, only filters for ethernet addresses, ethtype, VLAN ID and PCP > are implemented. I agree that returning -EINVAL is not helpful. > > > > > .3.f. More Interface info: > > > >> # ethtool -i enp2s0 > >> driver: igb > >> version: 5.4.0-k > >> firmware-version: 3.16, 0x800004ad > >> expansion-rom-version: > >> bus-info: 0000:02:00.0 > >> supports-statistics: yes > >> supports-test: yes > >> supports-eeprom-access: yes > >> supports-register-dump: yes > >> supports-priv-flags: yes > >> > > > > # ethtool -t enp2s0 > >> The test result is PASS > >> The test extra info: > >> Register test (offline) 0 > >> Eeprom test (offline) 0 > >> Interrupt test (offline) 0 > >> Loopback test (offline) 0 > >> Link test (on/offline) 0 > >> > > > > > > > > > > > > > > Daniel Williams | Software Engineer > > dwilliams at nextdroid.com > > _______________________________________________ > > Intel-wired-lan mailing list > > Intel-wired-lan at osuosl.org > > https://lists.osuosl.org/mailman/listinfo/intel-wired-lan > > -- > Vinicius > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20200504/d47ddb4a/attachment-0001.html> ^ permalink raw reply [flat|nested] 5+ messages in thread
* [Intel-wired-lan] Does the 'igb` kernel module support setting 2-Tuple filters (aka `--config-ntuple`) on a i210 NIC? 2020-05-04 13:58 ` Dan Williams @ 2020-05-04 14:20 ` Paul Menzel 2020-05-04 21:14 ` Alexander Duyck 1 sibling, 0 replies; 5+ messages in thread From: Paul Menzel @ 2020-05-04 14:20 UTC (permalink / raw) To: intel-wired-lan Dear Dan, Please only send plain text messages with no HTML part to mailing lists. Am 04.05.20 um 15:58 schrieb Dan Williams: >>> We have a computer logging a high rate of ethernet packets ( 25k >>>> packets/sec ~70 Mb/sec); But we're having trouble convincing >>>> the hardware to receive all of these packets, at a sustained >>>> rate -- specifically we're dropping packets while processing >>>> through the kernel layers. We're currently attempting to >>>> optimize the network stack, but we're having trouble setting >>>> the driver parameters... which is what this message is all >>>> about. >> >> That's weird. That packet rate is not *that* high, the Linux kernel >> should be able to handle that fine. > >> Can you give more details of the workload you are running? > > Okay, in more detail: we have two groups of incoming streams: (for > the minimum setup to cause a problem) - 4x Camera Streams each > transmits a 3.2mb image every .1 s, split into jumbo frames (mtu is > set to the full 9000) - Constant stream of data from a Lidar at 18k > packets / sec. Each packet is 1206 bytes long. - Both streams > continue steady-state, indefinitely (we have verified behavior out > to 4 hours so far) > > We receive all of these over ethernet, and routed to a single > network port on a single NIC. The driver is the 'igb' kernel > module, as supplied from ubuntu. The OS is Ubuntu 16.04 LTS with a > 4.15.0-88-lowlatency kernel. To debug already fixed problems, please try to reproduce this with current software, for example Ubuntu 20.04 or a mainline kernel build. [?] Kind regards, Paul [1]: https://wiki.ubuntu.com/Kernel/MainlineBuilds ^ permalink raw reply [flat|nested] 5+ messages in thread
* [Intel-wired-lan] Does the 'igb` kernel module support setting 2-Tuple filters (aka `--config-ntuple`) on a i210 NIC? 2020-05-04 13:58 ` Dan Williams 2020-05-04 14:20 ` Paul Menzel @ 2020-05-04 21:14 ` Alexander Duyck 1 sibling, 0 replies; 5+ messages in thread From: Alexander Duyck @ 2020-05-04 21:14 UTC (permalink / raw) To: intel-wired-lan On Mon, May 4, 2020 at 6:58 AM Dan Williams <dwilliams@nextdroid.com> wrote: > > We have a computer logging a high rate of ethernet packets ( 25k >>> > packets/sec ~70 Mb/sec); But we're having trouble convincing the >>> hardware >>> > to receive all of these packets, at a sustained rate -- specifically >>> we're >>> > dropping packets while processing through the kernel layers. We're >>> > currently attempting to optimize the network stack, but we're having >>> > trouble setting the driver parameters... which is what this message is >>> all >>> > about. >>> >> >> That's weird. That packet rate is not *that* high, the Linux kernel >> should be able to handle that fine. > > >> Can you give more details of the workload you are running? >> > > Okay, in more detail: we have two groups of incoming streams: (for the > minimum setup to cause a problem) > - 4x Camera Streams each transmits a 3.2mb image every .1 s, split into > jumbo frames (mtu is set to the full 9000) > - Constant stream of data from a Lidar at 18k packets / sec. Each packet > is 1206 bytes long. > - Both streams continue steady-state, indefinitely (we have verified > behavior out to 4 hours so far) > So you mentioned before that your rate was ~70mb/s or so. As far as the streams is anything changing in terms of the sources or are the streams coming from the same source throughout the run? I ask because if the source/destination ports and IP addresses are not changing then the hash will not change so the work on the queues should be constant as well. > We receive all of these over ethernet, and routed to a single network port > on a single NIC. The driver is the 'igb' kernel module, as supplied from > ubuntu. > The OS is Ubuntu 16.04 LTS with a 4.15.0-88-lowlatency kernel. > > ---- > Biggest Problem: > > Over time decay of packet processing. > > We've been working on this for a couple of weeks; when the processes start > we're logging the full data rate (~24kpps) but over time, something slows > down, and our logging rate shrinks. > (on the order of 20 packets / second / minute, consistently falling over > hours. ... after the first hour, we've lost 500 pps, after the second hour, > 1kpps... etc.) > It might be useful to provide "ethtool -S <iface>" output for the interface at the start, then one hour in, and then two hours in. That would allow us to check and see if there are any other indicators that are changing over time such as flow control frames. > Our user-land process simply isn't seeing the full count of packets -- we > have debug code that reads it from the OS, and then immediately drops the > buffer on the floor. Generally, we see drops in netstat, but not in the > driver. (i.e. from `ethtool -S | grep rx_*`) > So, our tentative guess is that we want to tune some parameters, somewhere > in the kernel or network driver to help out the kernel. Ideas welcome, > of course :) > The fact that you are seeing drops in the netstat would imply that your application isn't able to keep up with the data rates being provided by the network. Some of that can be addressed by trying to smooth out the burstiness of the traffic by increasing the interrupt rate as I describe below. > ( We are *also* dropping from the ring buffer, when both the lidar stream > and a camera stream are assigned to the same queue, but that looks like a > releated but separate issue) > So in terms of the ring buffer there are essentially two knobs you have to control some of that. The first is the size of the ring buffer, controlled via "ethtool -G <iface> rx <ring size>", the default is 256, you may want to try increasing it to 512 and see if that helps. The other item you could look at adjusting is the interrupt moderation, that is controlled by "ethtool -C <iface> rx-usecs <irq delay>". For your workload something like an IRQ delay of 50usec might make sense. That should limit the interrupts to about 20K per second which should mean that you only have one or two frames in the ring per interrupt, assuming the queue doesn't get stuck in NAPI polling mode. > Things we've tried / checked: > - irq alignment -- they're already reasonable set > - cpu assignment (via `taskset`) > - change processor task scheduling / priority (no effect) > > Ideas: > - what is the default hash algorithm? if we can't change it, maybe we can > work around it? > It is a Toeplitz hash, and there is not a way to change that. > - is there any other way to set the queue settings on this network card? > Debug tool? Rebuilding the kernel module with custom settings? > On the network card itself there isn't an option there. One option you could look at would be receive packet steering, or receive flow steering. You can find more information on those here: https://www.kernel.org/doc/html/v5.1/networking/scaling.html > - Our hardware also has an 82579 NIC as well -- would you guys recommend > we use that NIC, instead? > The 82579 would likely have fewer options than the i210 you are currently using. In addition it would support fewer queues. > - Do other network cards / chipsets have better support under linux? > Particularly when tuning input queues? > In the 1Gb space there usually isn't much out there for tuning individual queues. Normally high queue counts aren't that common on 1Gb devices so it doesn't make much sense to usually optimize for that. In the 10Gb space the drivers tend to have many more options when it comes to queue specific tuning, but I don't know if you would want to head in that direction. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20200504/6ba1d33b/attachment-0001.html> ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-05-04 21:14 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-04-30 19:22 [Intel-wired-lan] Does the 'igb` kernel module support setting 2-Tuple filters (aka `--config-ntuple`) on a i210 NIC? Dan Williams 2020-04-30 23:48 ` Vinicius Costa Gomes 2020-05-04 13:58 ` Dan Williams 2020-05-04 14:20 ` Paul Menzel 2020-05-04 21:14 ` Alexander Duyck
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.