Re: Cannot run multiple 'xdpsock' concurrently?

Re: Cannot run multiple 'xdpsock' concurrently?

[Maciej Fijalkowski]

On Mon, Feb 03, 2020 at 10:53:40AM +0100, Magnus Karlsson wrote:
> On Fri, Jan 31, 2020 at 3:14 AM Cameron Elliott
> <cameron@cameronelliott.com> wrote:
> >
> > Hello, I am trying to measure the maximum mpps I can push using AF_XDP on a 40G X710
> >
> > I can do 22 mpps  after resolving a few bumbles I made with drivers, etc., (Thanks Magnus!)
> > when using a single instance of 'xdpsock'
> >
> >
> > Apparently the way to upto 50, 60 or 70? mpps is to use multiple cores...
> > And apparently the simple way to do that, is multiple instances of xdpsock on different queues.
> >
> > But, my attempts with multiple instances fail. :(
> >
> >
> >
> > First, I checked my channel setup:
> >
> > $ sudo ethtool --set-channels enp1s0f0
> > no channel parameters changed.
> > current values: rx 0 tx 0 other 1 combined 4
> >
> > I presume that is okay...
> >
> > Then I run these two commands in two different windows:
> >
> > sudo  /home/c/bpf-next/samples/bpf/xdpsock -i enp1s0f0 -t -N -z -q 0
> > sudo  /home/c/bpf-next/samples/bpf/xdpsock -i enp1s0f0 -t -N -z -q 1
> >
> > With the only difference being the queue id.
> >
> > The first will start and show ~22 mpps tx rate.
> > When I start the second, both instances die:
> >
> > The first instace dies with:
> > /home/c/bpf-next/samples/bpf/xdpsock_user.c:kick_tx:794: errno: 100/"Network is down"
> >
> > The second instance dies with:
> > /home/c/bpf-next/samples/bpf/xdpsock_user.c:kick_tx:794: errno: 6/"No such device or address"
> >
> >
> > Do I understand correctly I should be able to run two instances like this concurrently?
> 
> This is probably not supported by the current xdpsock application.
> What is likely happening is that it tries to load the XDP program
> multiple times. As the XDP program is per netdev, not per queue, it
> should only be loaded once. When the second process then fails, it
> probably removes the XDP program (as it think it has loaded it) which
> crashes the first process you started.

Of course it *was* supported. Program is loaded only on the first xdpsock
instance since libbpf is checking whether xdp resources are there. On the
removal part you're right, we still haven't figured it out how to properly
exit the xdpsock when there are other xdpsocks running.

Actually commit b3873a5be757 ("net/i40e: Fix concurrency issues between
config flow and XSK") (CCing Maxim, Bjorn) broke the xdpsock multiple
instances support. If we drop the check against busy bit in PF state in
the i40e_xsk_wakeup, then I can run many xdpsocks on same ifindex.

I'm not really sure that is the right approach since we are explicitly
setting the __I40E_CONFIG_BUSY bit in i40e_queue_pair_disable which is
used when ndo_bpf is called with XDP_SETUP_XSK_UMEM command.

> 
> So, the application needs to get extended to support this. Maybe you
> want to do this :-)? Could be a good exercise. You could add a new
> commend line option that takes the number of instances you would like
> to create. Look at the -M option for some inspiration as it does some
> of the things you need. Maybe you can reuse that code to support your
> use case.
> 
> /Magnus
> 
> >
> > Thank you for any ideas, input.
> >
> >
> >
> > # ethtool dump / i40e driver from recent bpf-next clone
> > c@lumen ~> ethtool -i enp1s0f0
> > driver: i40e
> > version: 2.8.20-k
> > firmware-version: 7.10 0x80006456 1.2527.0
> > expansion-rom-version:
> > bus-info: 0000:01:00.0
> > supports-statistics: yes
> > supports-test: yes
> > supports-eeprom-access: yes
> > supports-register-dump: yes
> > supports-priv-flags: yes
> >
> >

Re: Cannot run multiple 'xdpsock' concurrently?

[Magnus Karlsson]

On Fri, Jan 31, 2020 at 3:14 AM Cameron Elliott
<cameron@cameronelliott.com> wrote:
>
> Hello, I am trying to measure the maximum mpps I can push using AF_XDP on a 40G X710
>
> I can do 22 mpps  after resolving a few bumbles I made with drivers, etc., (Thanks Magnus!)
> when using a single instance of 'xdpsock'
>
>
> Apparently the way to upto 50, 60 or 70? mpps is to use multiple cores...
> And apparently the simple way to do that, is multiple instances of xdpsock on different queues.
>
> But, my attempts with multiple instances fail. :(
>
>
>
> First, I checked my channel setup:
>
> $ sudo ethtool --set-channels enp1s0f0
> no channel parameters changed.
> current values: rx 0 tx 0 other 1 combined 4
>
> I presume that is okay...
>
> Then I run these two commands in two different windows:
>
> sudo  /home/c/bpf-next/samples/bpf/xdpsock -i enp1s0f0 -t -N -z -q 0
> sudo  /home/c/bpf-next/samples/bpf/xdpsock -i enp1s0f0 -t -N -z -q 1
>
> With the only difference being the queue id.
>
> The first will start and show ~22 mpps tx rate.
> When I start the second, both instances die:
>
> The first instace dies with:
> /home/c/bpf-next/samples/bpf/xdpsock_user.c:kick_tx:794: errno: 100/"Network is down"
>
> The second instance dies with:
> /home/c/bpf-next/samples/bpf/xdpsock_user.c:kick_tx:794: errno: 6/"No such device or address"
>
>
> Do I understand correctly I should be able to run two instances like this concurrently?

This is probably not supported by the current xdpsock application.
What is likely happening is that it tries to load the XDP program
multiple times. As the XDP program is per netdev, not per queue, it
should only be loaded once. When the second process then fails, it
probably removes the XDP program (as it think it has loaded it) which
crashes the first process you started.

So, the application needs to get extended to support this. Maybe you
want to do this :-)? Could be a good exercise. You could add a new
commend line option that takes the number of instances you would like
to create. Look at the -M option for some inspiration as it does some
of the things you need. Maybe you can reuse that code to support your
use case.

/Magnus

>
> Thank you for any ideas, input.
>
>
>
> # ethtool dump / i40e driver from recent bpf-next clone
> c@lumen ~> ethtool -i enp1s0f0
> driver: i40e
> version: 2.8.20-k
> firmware-version: 7.10 0x80006456 1.2527.0
> expansion-rom-version:
> bus-info: 0000:01:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
>

Re: Cannot run multiple 'xdpsock' concurrently?

[Maxim Mikityanskiy]

On 2020-02-03 05:11, Maciej Fijalkowski wrote:
> On Mon, Feb 03, 2020 at 10:53:40AM +0100, Magnus Karlsson wrote:
>> On Fri, Jan 31, 2020 at 3:14 AM Cameron Elliott
>> <cameron@cameronelliott.com> wrote:
>>>
>>> Hello, I am trying to measure the maximum mpps I can push using AF_XDP on a 40G X710
>>>
>>> I can do 22 mpps  after resolving a few bumbles I made with drivers, etc., (Thanks Magnus!)
>>> when using a single instance of 'xdpsock'
>>>
>>>
>>> Apparently the way to upto 50, 60 or 70? mpps is to use multiple cores...
>>> And apparently the simple way to do that, is multiple instances of xdpsock on different queues.
>>>
>>> But, my attempts with multiple instances fail. :(
>>>
>>>
>>>
>>> First, I checked my channel setup:
>>>
>>> $ sudo ethtool --set-channels enp1s0f0
>>> no channel parameters changed.
>>> current values: rx 0 tx 0 other 1 combined 4
>>>
>>> I presume that is okay...
>>>
>>> Then I run these two commands in two different windows:
>>>
>>> sudo  /home/c/bpf-next/samples/bpf/xdpsock -i enp1s0f0 -t -N -z -q 0
>>> sudo  /home/c/bpf-next/samples/bpf/xdpsock -i enp1s0f0 -t -N -z -q 1
>>>
>>> With the only difference being the queue id.
>>>
>>> The first will start and show ~22 mpps tx rate.
>>> When I start the second, both instances die:
>>>
>>> The first instace dies with:
>>> /home/c/bpf-next/samples/bpf/xdpsock_user.c:kick_tx:794: errno: 100/"Network is down"
>>>
>>> The second instance dies with:
>>> /home/c/bpf-next/samples/bpf/xdpsock_user.c:kick_tx:794: errno: 6/"No such device or address"
>>>
>>>
>>> Do I understand correctly I should be able to run two instances like this concurrently?
>>
>> This is probably not supported by the current xdpsock application.
>> What is likely happening is that it tries to load the XDP program
>> multiple times. As the XDP program is per netdev, not per queue, it
>> should only be loaded once. When the second process then fails, it
>> probably removes the XDP program (as it think it has loaded it) which
>> crashes the first process you started.
> 
> Of course it *was* supported. Program is loaded only on the first xdpsock
> instance since libbpf is checking whether xdp resources are there. On the
> removal part you're right, we still haven't figured it out how to properly
> exit the xdpsock when there are other xdpsocks running.
> 
> Actually commit b3873a5be757 ("net/i40e: Fix concurrency issues between
> config flow and XSK") (CCing Maxim, Bjorn) broke the xdpsock multiple
> instances support.

That was surprising, but now that I took a look at that code in the 
context of this issue, it got clear to me why it happens.

> If we drop the check against busy bit in PF state in
> the i40e_xsk_wakeup, then I can run many xdpsocks on same ifindex.
> 
> I'm not really sure that is the right approach since we are explicitly
> setting the __I40E_CONFIG_BUSY bit in i40e_queue_pair_disable which is
> used when ndo_bpf is called with XDP_SETUP_XSK_UMEM command.

Right, we shouldn't drop this check, at least it's needed to sync on 
xsk_wakeup when closing a socket. I think there are two issues here, and 
I suggest this way of solving them:

1. __I40E_CONFIG_BUSY is set while a single QP is being recreated, e.g., 
when opening/closing an XSK. I considered it as a flag for global 
configuration changes, but apparently it's also needed when a single XSK 
is being configured. Probably xsk_wakeup shouldn't return ENETDOWN when 
this flag is set, but something lighter like EAGAIN. xdpsock will need 
to repeat the syscall instead of bailing out.

2. Should xdpsock stop on ENETDOWN? Normal network sockets don't close 
as soon as the carrier is lost, so why should xdpsock fail on ENETDOWN? 
IMO, xdpsock should handle it more gracefully (wait with some timeout?).

Does it sound like the right direction?

>>
>> So, the application needs to get extended to support this. Maybe you
>> want to do this :-)? Could be a good exercise. You could add a new
>> commend line option that takes the number of instances you would like
>> to create. Look at the -M option for some inspiration as it does some
>> of the things you need. Maybe you can reuse that code to support your
>> use case.
>>
>> /Magnus
>>
>>>
>>> Thank you for any ideas, input.
>>>
>>>
>>>
>>> # ethtool dump / i40e driver from recent bpf-next clone
>>> c@lumen ~> ethtool -i enp1s0f0
>>> driver: i40e
>>> version: 2.8.20-k
>>> firmware-version: 7.10 0x80006456 1.2527.0
>>> expansion-rom-version:
>>> bus-info: 0000:01:00.0
>>> supports-statistics: yes
>>> supports-test: yes
>>> supports-eeprom-access: yes
>>> supports-register-dump: yes
>>> supports-priv-flags: yes
>>>
>>>

Re: Cannot run multiple 'xdpsock' concurrently?

[Maciej Fijalkowski]

On Mon, Feb 03, 2020 at 01:49:11PM +0200, Maxim Mikityanskiy wrote:
> On 2020-02-03 05:11, Maciej Fijalkowski wrote:
> > On Mon, Feb 03, 2020 at 10:53:40AM +0100, Magnus Karlsson wrote:
> > > On Fri, Jan 31, 2020 at 3:14 AM Cameron Elliott
> > > <cameron@cameronelliott.com> wrote:
> > > > 
> > > > Hello, I am trying to measure the maximum mpps I can push using AF_XDP on a 40G X710
> > > > 
> > > > I can do 22 mpps  after resolving a few bumbles I made with drivers, etc., (Thanks Magnus!)
> > > > when using a single instance of 'xdpsock'
> > > > 
> > > > 
> > > > Apparently the way to upto 50, 60 or 70? mpps is to use multiple cores...
> > > > And apparently the simple way to do that, is multiple instances of xdpsock on different queues.
> > > > 
> > > > But, my attempts with multiple instances fail. :(
> > > > 
> > > > 
> > > > 
> > > > First, I checked my channel setup:
> > > > 
> > > > $ sudo ethtool --set-channels enp1s0f0
> > > > no channel parameters changed.
> > > > current values: rx 0 tx 0 other 1 combined 4
> > > > 
> > > > I presume that is okay...
> > > > 
> > > > Then I run these two commands in two different windows:
> > > > 
> > > > sudo  /home/c/bpf-next/samples/bpf/xdpsock -i enp1s0f0 -t -N -z -q 0
> > > > sudo  /home/c/bpf-next/samples/bpf/xdpsock -i enp1s0f0 -t -N -z -q 1
> > > > 
> > > > With the only difference being the queue id.
> > > > 
> > > > The first will start and show ~22 mpps tx rate.
> > > > When I start the second, both instances die:
> > > > 
> > > > The first instace dies with:
> > > > /home/c/bpf-next/samples/bpf/xdpsock_user.c:kick_tx:794: errno: 100/"Network is down"
> > > > 
> > > > The second instance dies with:
> > > > /home/c/bpf-next/samples/bpf/xdpsock_user.c:kick_tx:794: errno: 6/"No such device or address"
> > > > 
> > > > 
> > > > Do I understand correctly I should be able to run two instances like this concurrently?
> > > 
> > > This is probably not supported by the current xdpsock application.
> > > What is likely happening is that it tries to load the XDP program
> > > multiple times. As the XDP program is per netdev, not per queue, it
> > > should only be loaded once. When the second process then fails, it
> > > probably removes the XDP program (as it think it has loaded it) which
> > > crashes the first process you started.
> > 
> > Of course it *was* supported. Program is loaded only on the first xdpsock
> > instance since libbpf is checking whether xdp resources are there. On the
> > removal part you're right, we still haven't figured it out how to properly
> > exit the xdpsock when there are other xdpsocks running.
> > 
> > Actually commit b3873a5be757 ("net/i40e: Fix concurrency issues between
> > config flow and XSK") (CCing Maxim, Bjorn) broke the xdpsock multiple
> > instances support.
> 
> That was surprising, but now that I took a look at that code in the context
> of this issue, it got clear to me why it happens.
> 
> > If we drop the check against busy bit in PF state in
> > the i40e_xsk_wakeup, then I can run many xdpsocks on same ifindex.
> > 
> > I'm not really sure that is the right approach since we are explicitly
> > setting the __I40E_CONFIG_BUSY bit in i40e_queue_pair_disable which is
> > used when ndo_bpf is called with XDP_SETUP_XSK_UMEM command.
> 
> Right, we shouldn't drop this check, at least it's needed to sync on
> xsk_wakeup when closing a socket. I think there are two issues here, and I
> suggest this way of solving them:
> 
> 1. __I40E_CONFIG_BUSY is set while a single QP is being recreated, e.g.,
> when opening/closing an XSK. I considered it as a flag for global
> configuration changes, but apparently it's also needed when a single XSK is
> being configured. Probably xsk_wakeup shouldn't return ENETDOWN when this
> flag is set, but something lighter like EAGAIN. xdpsock will need to repeat
> the syscall instead of bailing out.
> 
Haven't yet checked the l2fwd case since I broke my hw setup, but do we
really need the syscall repetition? For tx only scenario skipping the
syscall would mean that we wouldn't post entries to completion queue and
in case tx ring gets full we would call complete_tx_only() again. So there
we have syscall being repeated?

> 2. Should xdpsock stop on ENETDOWN? Normal network sockets don't close as
> soon as the carrier is lost, so why should xdpsock fail on ENETDOWN? IMO,
> xdpsock should handle it more gracefully (wait with some timeout?).
> 
> Does it sound like the right direction?

It does! :) But in general I would allow ENETDOWN to the set of errnos
checked in kick_tx(). Not sure about repeating syscall though.

> 
> > > 
> > > So, the application needs to get extended to support this. Maybe you
> > > want to do this :-)? Could be a good exercise. You could add a new
> > > commend line option that takes the number of instances you would like
> > > to create. Look at the -M option for some inspiration as it does some
> > > of the things you need. Maybe you can reuse that code to support your
> > > use case.
> > > 
> > > /Magnus
> > > 
> > > > 
> > > > Thank you for any ideas, input.
> > > > 
> > > > 
> > > > 
> > > > # ethtool dump / i40e driver from recent bpf-next clone
> > > > c@lumen ~> ethtool -i enp1s0f0
> > > > driver: i40e
> > > > version: 2.8.20-k
> > > > firmware-version: 7.10 0x80006456 1.2527.0
> > > > expansion-rom-version:
> > > > bus-info: 0000:01:00.0
> > > > supports-statistics: yes
> > > > supports-test: yes
> > > > supports-eeprom-access: yes
> > > > supports-register-dump: yes
> > > > supports-priv-flags: yes
> > > > 
> > > > 
> 

Re: Cannot run multiple 'xdpsock' concurrently?

[Maciej Fijalkowski]

On Tue, Feb 04, 2020 at 04:41:31PM +0100, Björn Töpel wrote:
> On Tue, 4 Feb 2020 at 14:58, Maciej Fijalkowski
> <maciej.fijalkowski@intel.com> wrote:
> >
> > On Mon, Feb 03, 2020 at 01:49:11PM +0200, Maxim Mikityanskiy wrote:
> [...]
> > > Right, we shouldn't drop this check, at least it's needed to sync on
> > > xsk_wakeup when closing a socket. I think there are two issues here, and I
> > > suggest this way of solving them:
> > >
> > > 1. __I40E_CONFIG_BUSY is set while a single QP is being recreated, e.g.,
> > > when opening/closing an XSK. I considered it as a flag for global
> > > configuration changes, but apparently it's also needed when a single XSK is
> > > being configured. Probably xsk_wakeup shouldn't return ENETDOWN when this
> > > flag is set, but something lighter like EAGAIN. xdpsock will need to repeat
> > > the syscall instead of bailing out.
> > >
> > Haven't yet checked the l2fwd case since I broke my hw setup, but do we
> > really need the syscall repetition? For tx only scenario skipping the
> > syscall would mean that we wouldn't post entries to completion queue and
> > in case tx ring gets full we would call complete_tx_only() again. So there
> > we have syscall being repeated?
> >
> 
> I sort of like EAGAIN, as Max suggested, is better. Kind of nice for a
> user to know that "hey, for some reason the driver couldn't service
> you ATM, please try again".
> 
> > > 2. Should xdpsock stop on ENETDOWN? Normal network sockets don't close as
> > > soon as the carrier is lost, so why should xdpsock fail on ENETDOWN? IMO,
> > > xdpsock should handle it more gracefully (wait with some timeout?).
> > >
> > > Does it sound like the right direction?
> >
> > It does! :) But in general I would allow ENETDOWN to the set of errnos
> > checked in kick_tx(). Not sure about repeating syscall though.
> >
> 
> The fact that xdpsock die() on errors are a bit harsh, can't argue
> there. So, I'm all in for simplifying the sample error checking for
> kick_tx().

So to summarize that:
- return EAGAIN instead of ENETDOWN within ndo_xsk_wakeup implementations
  in i40e/ixgbe,
- check for ENETDOWN in kick_tx() as well

Do you guys agree?

What about timeout and syscall repetition? That's the last question.

> 
> 
> Björn

Re: Cannot run multiple 'xdpsock' concurrently?

[Björn Töpel]

On Tue, 4 Feb 2020 at 14:58, Maciej Fijalkowski
<maciej.fijalkowski@intel.com> wrote:
>
> On Mon, Feb 03, 2020 at 01:49:11PM +0200, Maxim Mikityanskiy wrote:
[...]
> > Right, we shouldn't drop this check, at least it's needed to sync on
> > xsk_wakeup when closing a socket. I think there are two issues here, and I
> > suggest this way of solving them:
> >
> > 1. __I40E_CONFIG_BUSY is set while a single QP is being recreated, e.g.,
> > when opening/closing an XSK. I considered it as a flag for global
> > configuration changes, but apparently it's also needed when a single XSK is
> > being configured. Probably xsk_wakeup shouldn't return ENETDOWN when this
> > flag is set, but something lighter like EAGAIN. xdpsock will need to repeat
> > the syscall instead of bailing out.
> >
> Haven't yet checked the l2fwd case since I broke my hw setup, but do we
> really need the syscall repetition? For tx only scenario skipping the
> syscall would mean that we wouldn't post entries to completion queue and
> in case tx ring gets full we would call complete_tx_only() again. So there
> we have syscall being repeated?
>

I sort of like EAGAIN, as Max suggested, is better. Kind of nice for a
user to know that "hey, for some reason the driver couldn't service
you ATM, please try again".

> > 2. Should xdpsock stop on ENETDOWN? Normal network sockets don't close as
> > soon as the carrier is lost, so why should xdpsock fail on ENETDOWN? IMO,
> > xdpsock should handle it more gracefully (wait with some timeout?).
> >
> > Does it sound like the right direction?
>
> It does! :) But in general I would allow ENETDOWN to the set of errnos
> checked in kick_tx(). Not sure about repeating syscall though.
>

The fact that xdpsock die() on errors are a bit harsh, can't argue
there. So, I'm all in for simplifying the sample error checking for
kick_tx().


Björn

Re: Cannot run multiple 'xdpsock' concurrently?

[Björn Töpel]

On 2020-02-04 10:46, Maciej Fijalkowski wrote:
> On Tue, Feb 04, 2020 at 04:41:31PM +0100, Björn Töpel wrote:
[...]
> 
> So to summarize that:
> - return EAGAIN instead of ENETDOWN within ndo_xsk_wakeup implementations
>    in i40e/ixgbe,
> - check for ENETDOWN in kick_tx() as well
> 
> Do you guys agree?
>

Well, start with latter. From some perspective, changing the return 
value is ABI breakage, no?

> What about timeout and syscall repetition? That's the last question.
>

Hmm, timeout in what context? When to redo the syscall?

Re: Cannot run multiple 'xdpsock' concurrently?

[Magnus Karlsson]

On Tue, Feb 4, 2020 at 6:20 PM Björn Töpel <bjorn.topel@intel.com> wrote:
>
>
>
> On 2020-02-04 10:46, Maciej Fijalkowski wrote:
> > On Tue, Feb 04, 2020 at 04:41:31PM +0100, Björn Töpel wrote:
> [...]
> >
> > So to summarize that:
> > - return EAGAIN instead of ENETDOWN within ndo_xsk_wakeup implementations
> >    in i40e/ixgbe,
> > - check for ENETDOWN in kick_tx() as well
> >
> > Do you guys agree?
> >
>
> Well, start with latter. From some perspective, changing the return
> value is ABI breakage, no?
>
> > What about timeout and syscall repetition? That's the last question.
> >
>
> Hmm, timeout in what context? When to redo the syscall?

Cameron,

FYI: https://lore.kernel.org/netdev/20200205212147.GA5358@pc-9.home/T/#r715e0a78ae13c0fc1ce67b3219e9a532b2bfd011

/Magnus

Cannot run multiple 'xdpsock' concurrently?

[Cameron Elliott]

Hello, I am trying to measure the maximum mpps I can push using AF_XDP
on a 40G X710

I can do ~22 mpps after resolving a few bumbles I made with drivers,
etc., (Thanks Magnus!)
when using a single instance of 'xdpsock'


Apparently the way to upto 50, 60 or 70? mpps is to use multiple cores...
And apparently the simple way to do that, is multiple instances of
xdpsock on different queues.

But, my attempts with multiple instances fail. :(



First, I checked my channel setup:

$ sudo ethtool --set-channels enp1s0f0
no channel parameters changed.
current values: rx 0 tx 0 other 1 combined 4

I presume that is okay...

Then I run these two commands in two different windows:

sudo  /home/c/bpf-next/samples/bpf/xdpsock -i enp1s0f0 -t -N -z -q 0
sudo  /home/c/bpf-next/samples/bpf/xdpsock -i enp1s0f0 -t -N -z -q 1

With the only difference being the queue id.

The first will start and show ~22 mpps tx rate.
When I start the second, both instances die:

The first instace dies with:
/home/c/bpf-next/samples/bpf/xdpsock_user.c:kick_tx:794: errno:
100/"Network is down"

The second instance dies with:
/home/c/bpf-next/samples/bpf/xdpsock_user.c:kick_tx:794: errno: 6/"No
such device or address"


Do I understand correctly I should be able to run two instances like
this concurrently?



Thank you for any ideas, input.



# ethtool dump / i40e driver from recent bpf-next clone
c@lumen ~> ethtool -i enp1s0f0
driver: i40e
version: 2.8.20-k
firmware-version: 7.10 0x80006456 1.2527.0
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes