error loading xdp program on virtio nic

error loading xdp program on virtio nic

[David Ahern]

Hi:

Trying to load an XDP program on a virtio based nic is failing with:

virtio_net: XDP expects header/data in single page, any_header_sg required

I have not encountered this error before and not able to find what is
missing. Any tips?

Thanks,
David

Re: error loading xdp program on virtio nic

[Jason Wang]

On 2019/11/21 上午1:52, David Ahern wrote:
> Hi:
>
> Trying to load an XDP program on a virtio based nic is failing with:
>
> virtio_net: XDP expects header/data in single page, any_header_sg required
>
> I have not encountered this error before and not able to find what is
> missing. Any tips?


Hi David:

What qemu + guest kernel version did you use? And could you share you 
qemu cli?

Old qemu requires vnet header to be placed into a separate sg which 
breaks the assumption of XDP. Recent qemu doesn't have such limitation 
(any_header_sg feature).

Thanks


>
> Thanks,
> David
>

Re: error loading xdp program on virtio nic

[David Ahern]

On 11/20/19 8:26 PM, Jason Wang wrote:
> 
> On 2019/11/21 上午1:52, David Ahern wrote:
>> Hi:
>>
>> Trying to load an XDP program on a virtio based nic is failing with:
>>
>> virtio_net: XDP expects header/data in single page, any_header_sg
>> required
>>
>> I have not encountered this error before and not able to find what is
>> missing. Any tips?
> 
> 
> Hi David:
> 
> What qemu + guest kernel version did you use? And could you share you
> qemu cli?
> 
> Old qemu requires vnet header to be placed into a separate sg which
> breaks the assumption of XDP. Recent qemu doesn't have such limitation
> (any_header_sg feature).
> 
>

Hi Jason,

When I run qemu via my older vm-tools scripts XDP works fine. This is
the first time I am trying to use XDP with guests started by libvirt.

We isolated it to a libvirt xml file using an old machine type
(pc-i440fx-1.5) - basically any machine with VIRTIO_F_VERSION_1 not set.
Using a newer one move the problem forward.

The current error message is:
  virtio_net: Too few free TX rings available
again, looking for some libvirt setting for the vm create.

Re: error loading xdp program on virtio nic

[Jason Wang]

On 2019/11/21 上午11:35, David Ahern wrote:
> On 11/20/19 8:26 PM, Jason Wang wrote:
>> On 2019/11/21 上午1:52, David Ahern wrote:
>>> Hi:
>>>
>>> Trying to load an XDP program on a virtio based nic is failing with:
>>>
>>> virtio_net: XDP expects header/data in single page, any_header_sg
>>> required
>>>
>>> I have not encountered this error before and not able to find what is
>>> missing. Any tips?
>>
>> Hi David:
>>
>> What qemu + guest kernel version did you use? And could you share you
>> qemu cli?
>>
>> Old qemu requires vnet header to be placed into a separate sg which
>> breaks the assumption of XDP. Recent qemu doesn't have such limitation
>> (any_header_sg feature).
>>
>>
> Hi Jason,
>
> When I run qemu via my older vm-tools scripts XDP works fine. This is
> the first time I am trying to use XDP with guests started by libvirt.
>
> We isolated it to a libvirt xml file using an old machine type
> (pc-i440fx-1.5) - basically any machine with VIRTIO_F_VERSION_1 not set.
> Using a newer one move the problem forward.


Yes, if VERSION_1 implies ANY_HEADER_SG, and if you're using old qemu, 
make sure any_header_sg is enabled in qemu cli.


>
> The current error message is:
>    virtio_net: Too few free TX rings available
> again, looking for some libvirt setting for the vm create.
>

Make sure you have sufficient queues, e.g if you N vcpus with multiqueue 
enabled, you need 2*N queues for virtio-net.

Thanks

Re: error loading xdp program on virtio nic

[David Ahern]

On 11/20/19 8:54 PM, Jason Wang wrote:
>>
>> The current error message is:
>>    virtio_net: Too few free TX rings available
>> again, looking for some libvirt setting for the vm create.
>>
> 
> Make sure you have sufficient queues, e.g if you N vcpus with multiqueue
> enabled, you need 2*N queues for virtio-net.

yep, that did the trick and now I can attach xdp programs. Thanks for
the help.

Re: error loading xdp program on virtio nic

[Jesper Dangaard Brouer]

On Wed, 20 Nov 2019 21:05:48 -0700
David Ahern <dsahern@gmail.com> wrote:

> On 11/20/19 8:54 PM, Jason Wang wrote:
> >>
> >> The current error message is:
> >>    virtio_net: Too few free TX rings available
> >> again, looking for some libvirt setting for the vm create.
> >>  
> > 
> > Make sure you have sufficient queues, e.g if you N vcpus with multiqueue
> > enabled, you need 2*N queues for virtio-net.  
> 
> yep, that did the trick and now I can attach xdp programs. Thanks for
> the help.

How did you configure number of queues in libbvirt? 

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: error loading xdp program on virtio nic

[Jason Wang]

On 2019/11/21 下午2:26, Jesper Dangaard Brouer wrote:
> On Wed, 20 Nov 2019 21:05:48 -0700
> David Ahern <dsahern@gmail.com> wrote:
>
>> On 11/20/19 8:54 PM, Jason Wang wrote:
>>>> The current error message is:
>>>>     virtio_net: Too few free TX rings available
>>>> again, looking for some libvirt setting for the vm create.
>>>>   
>>> Make sure you have sufficient queues, e.g if you N vcpus with multiqueue
>>> enabled, you need 2*N queues for virtio-net.
>> yep, that did the trick and now I can attach xdp programs. Thanks for
>> the help.
> How did you configure number of queues in libbvirt?
>

By specifying queues property like:

<devices>

   <interface type='network'>
     <source network='default'/>
     <target dev='vnet1'/>
     <model type='virtio'/>
     <driver name='vhost' txmode='iothread' ioeventfd='on' 
event_idx='off' queues='5' rx_queue_size='256' tx_queue_size='256'>
       <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' 
ufo='off' mrg_rxbuf='off'/>
       <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/>
     </driver>
     </interface>
</devices>


More information is here: https://libvirt.org/formatdomain.html

Thanks

Re: error loading xdp program on virtio nic

[David Ahern]

On 11/21/19 12:02 AM, Jason Wang wrote:
> By specifying queues property like:
> 
> <devices>
> 
>   <interface type='network'>
>     <source network='default'/>
>     <target dev='vnet1'/>
>     <model type='virtio'/>
>     <driver name='vhost' txmode='iothread' ioeventfd='on'
> event_idx='off' queues='5' rx_queue_size='256' tx_queue_size='256'>

I can not check this because the 3.0 version of libvirt does not support
tx_queue_size. It is multiqueue (queues=5 in the example) setting that
needs to be set to 2*Nvcpus for XDP, correct?

>       <host csum='off' gso='off' tso4='off' tso6='off' ecn='off'
> ufo='off' mrg_rxbuf='off'/>
>       <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/>
>     </driver>
>     </interface>
> </devices>
> 
> 

The virtio_net driver suggests the queues are needed for XDP_TX:

       /* XDP requires extra queues for XDP_TX */
        if (curr_qp + xdp_qp > vi->max_queue_pairs) {
                NL_SET_ERR_MSG_MOD(extack, "Too few free TX rings
available");
                netdev_warn(dev, "request %i queues but max is %i\n",
                            curr_qp + xdp_qp, vi->max_queue_pairs);
                return -ENOMEM;
        }

Doubling the number of queues for each tap device adds overhead to the
hypervisor if you only want to allow XDP_DROP or XDP_DIRECT. Am I
understanding that correctly?

Re: error loading xdp program on virtio nic

[Jason Wang]

On 2019/11/21 下午11:49, David Ahern wrote:
> On 11/21/19 12:02 AM, Jason Wang wrote:
>> By specifying queues property like:
>>
>> <devices>
>>
>>    <interface type='network'>
>>      <source network='default'/>
>>      <target dev='vnet1'/>
>>      <model type='virtio'/>
>>      <driver name='vhost' txmode='iothread' ioeventfd='on'
>> event_idx='off' queues='5' rx_queue_size='256' tx_queue_size='256'>
> I can not check this because the 3.0 version of libvirt does not support
> tx_queue_size. It is multiqueue (queues=5 in the example) setting that
> needs to be set to 2*Nvcpus for XDP, correct?


Yes.


>
>>        <host csum='off' gso='off' tso4='off' tso6='off' ecn='off'
>> ufo='off' mrg_rxbuf='off'/>
>>        <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/>
>>      </driver>
>>      </interface>
>> </devices>
>>
>>
> The virtio_net driver suggests the queues are needed for XDP_TX:
>
>         /* XDP requires extra queues for XDP_TX */
>          if (curr_qp + xdp_qp > vi->max_queue_pairs) {
>                  NL_SET_ERR_MSG_MOD(extack, "Too few free TX rings
> available");
>                  netdev_warn(dev, "request %i queues but max is %i\n",
>                              curr_qp + xdp_qp, vi->max_queue_pairs);
>                  return -ENOMEM;
>          }
>
> Doubling the number of queues for each tap device adds overhead to the
> hypervisor if you only want to allow XDP_DROP or XDP_DIRECT. Am I
> understanding that correctly?


Yes, but there's almost impossible to know whether or not XDP_TX will be 
used by the program. If we don't use per CPU TX queue, it must be 
serialized through locks, not sure it's worth try that (not by default, 
of course).

Thanks

>

Re: error loading xdp program on virtio nic

[David Ahern]

On 11/21/19 11:09 PM, Jason Wang wrote:
>> Doubling the number of queues for each tap device adds overhead to the
>> hypervisor if you only want to allow XDP_DROP or XDP_DIRECT. Am I
>> understanding that correctly?
> 
> 
> Yes, but there's almost impossible to know whether or not XDP_TX will be
> used by the program. If we don't use per CPU TX queue, it must be
> serialized through locks, not sure it's worth try that (not by default,
> of course).
> 

This restriction is going to prevent use of XDP in VMs in general cloud
hosting environments. 2x vhost threads for vcpus is a non-starter.

If one XDP feature has high resource needs, then we need to subdivide
the capabilities to let some work and others fail. For example, a flag
can be added to xdp_buff / xdp_md that indicates supported XDP features.
If there are insufficient resources for XDP_TX, do not show support for
it. If a program returns XDP_TX anyways, packets will be dropped.

Re: error loading xdp program on virtio nic

[Jakub Kicinski]

On Fri, 22 Nov 2019 08:43:50 -0700, David Ahern wrote:
> On 11/21/19 11:09 PM, Jason Wang wrote:
> >> Doubling the number of queues for each tap device adds overhead to the
> >> hypervisor if you only want to allow XDP_DROP or XDP_DIRECT. Am I
> >> understanding that correctly?  
> > 
> > Yes, but there's almost impossible to know whether or not XDP_TX will be
> > used by the program. If we don't use per CPU TX queue, it must be
> > serialized through locks, not sure it's worth try that (not by default,
> > of course). 
> 
> This restriction is going to prevent use of XDP in VMs in general cloud
> hosting environments. 2x vhost threads for vcpus is a non-starter.
> 
> If one XDP feature has high resource needs, then we need to subdivide
> the capabilities to let some work and others fail. For example, a flag
> can be added to xdp_buff / xdp_md that indicates supported XDP features.
> If there are insufficient resources for XDP_TX, do not show support for
> it. If a program returns XDP_TX anyways, packets will be dropped.

This is covered by the always planned but never completed work on
richer queue configuration ABI by yours truly, Magnus and others.

Once we have better control over queues we can do some "manual mode"
config where queues are not automatically provisioned, are provisioned
even when program is not bound or are provisioned on a subset of cores
(if all NICs' IRQs/NAPIs are pinned to those core all should be gut).

🤷‍♂️

Re: error loading xdp program on virtio nic

[Jesper Dangaard Brouer]

On Fri, 22 Nov 2019 08:43:50 -0700
David Ahern <dsahern@gmail.com> wrote:

> On 11/21/19 11:09 PM, Jason Wang wrote:
> >> Doubling the number of queues for each tap device adds overhead to the
> >> hypervisor if you only want to allow XDP_DROP or XDP_DIRECT. Am I
> >> understanding that correctly?  
> > 
> > 
> > Yes, but there's almost impossible to know whether or not XDP_TX will be
> > used by the program. If we don't use per CPU TX queue, it must be
> > serialized through locks, not sure it's worth try that (not by default,
> > of course).
> >   
> 
> This restriction is going to prevent use of XDP in VMs in general cloud
> hosting environments. 2x vhost threads for vcpus is a non-starter.
> 
> If one XDP feature has high resource needs, then we need to subdivide
> the capabilities to let some work and others fail. For example, a flag
> can be added to xdp_buff / xdp_md that indicates supported XDP features.
> If there are insufficient resources for XDP_TX, do not show support for
> it. If a program returns XDP_TX anyways, packets will be dropped.
> 

This sounds like concrete use-case and solid argument why we need XDP
feature detection and checks. (Last part of LPC talk[1] were about
XDP features).

An interesting perspective you bring up, is that XDP features are not
static per device driver.  It actually needs to be dynamic, as your
XDP_TX feature request depend on the queue resources available.

Implementation wise, I would not add flags to xdp_buff / xdp_md.
Instead I propose in[1] slide 46, that the verifier should detect the
XDP features used by a BPF-prog.  If you XDP prog doesn't use e.g.
XDP_TX, then you should be allowed to run it on a virtio_net device
with less queue configured, right?


[1] http://people.netfilter.org/hawk/presentations/LinuxPlumbers2019/xdp-distro-view.pdf
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: error loading xdp program on virtio nic

[David Ahern]

On 11/22/19 9:57 AM, Jesper Dangaard Brouer wrote:
> Implementation wise, I would not add flags to xdp_buff / xdp_md.
> Instead I propose in[1] slide 46, that the verifier should detect the
> XDP features used by a BPF-prog.  If you XDP prog doesn't use e.g.
> XDP_TX, then you should be allowed to run it on a virtio_net device
> with less queue configured, right?

Thanks for the reference and yes, that is the goal: allow XDP in the
most use cases possible. e.g., Why limit XDP_DROP which requires no
resources because XDP_TX does not work?

I agree a flag in the api is an ugly way to allow it. For the verifier
approach, you mean add an internal flag (e.g., bitmask of return codes)
that the program uses and the NIC driver can check at attach time?

Re: error loading xdp program on virtio nic

[Toke Høiland-Jørgensen]

David Ahern <dsahern@gmail.com> writes:

> On 11/22/19 9:57 AM, Jesper Dangaard Brouer wrote:
>> Implementation wise, I would not add flags to xdp_buff / xdp_md.
>> Instead I propose in[1] slide 46, that the verifier should detect the
>> XDP features used by a BPF-prog.  If you XDP prog doesn't use e.g.
>> XDP_TX, then you should be allowed to run it on a virtio_net device
>> with less queue configured, right?
>
> Thanks for the reference and yes, that is the goal: allow XDP in the
> most use cases possible. e.g., Why limit XDP_DROP which requires no
> resources because XDP_TX does not work?
>
> I agree a flag in the api is an ugly way to allow it. For the verifier
> approach, you mean add an internal flag (e.g., bitmask of return codes)
> that the program uses and the NIC driver can check at attach time?

Yes, that's more or less what we've discussed. With the actual set of
flags, and the API for the driver (new ndo?) TBD. Suggestions welcome; I
anticipate this is something Jesper and I need to circle back to soonish
in any case (unless someone beats us to it!).

-Toke

Re: error loading xdp program on virtio nic

[Toke Høiland-Jørgensen]

David Ahern <dsahern@gmail.com> writes:

> On 11/22/19 9:57 AM, Jesper Dangaard Brouer wrote:
>> Implementation wise, I would not add flags to xdp_buff / xdp_md.
>> Instead I propose in[1] slide 46, that the verifier should detect the
>> XDP features used by a BPF-prog.  If you XDP prog doesn't use e.g.
>> XDP_TX, then you should be allowed to run it on a virtio_net device
>> with less queue configured, right?
>
> Thanks for the reference and yes, that is the goal: allow XDP in the
> most use cases possible. e.g., Why limit XDP_DROP which requires no
> resources because XDP_TX does not work?
>
> I agree a flag in the api is an ugly way to allow it. For the verifier
> approach, you mean add an internal flag (e.g., bitmask of return codes)
> that the program uses and the NIC driver can check at attach time?

Yes, that's more or less what we've discussed. With the actual set of
flags, and the API for the driver (new ndo?) TBD. Suggestions welcome; I
anticipate this is something Jesper and I need to circle back to soonish
in any case (unless someone beats us to it!).

-Toke

Re: error loading xdp program on virtio nic

[Jason Wang]

On 2019/11/23 上午12:57, Jesper Dangaard Brouer wrote:
> On Fri, 22 Nov 2019 08:43:50 -0700
> David Ahern <dsahern@gmail.com> wrote:
>
>> On 11/21/19 11:09 PM, Jason Wang wrote:
>>>> Doubling the number of queues for each tap device adds overhead to the
>>>> hypervisor if you only want to allow XDP_DROP or XDP_DIRECT. Am I
>>>> understanding that correctly?
>>>
>>> Yes, but there's almost impossible to know whether or not XDP_TX will be
>>> used by the program. If we don't use per CPU TX queue, it must be
>>> serialized through locks, not sure it's worth try that (not by default,
>>> of course).
>>>    
>> This restriction is going to prevent use of XDP in VMs in general cloud
>> hosting environments. 2x vhost threads for vcpus is a non-starter.
>>
>> If one XDP feature has high resource needs, then we need to subdivide
>> the capabilities to let some work and others fail. For example, a flag
>> can be added to xdp_buff / xdp_md that indicates supported XDP features.
>> If there are insufficient resources for XDP_TX, do not show support for
>> it. If a program returns XDP_TX anyways, packets will be dropped.
>>
> This sounds like concrete use-case and solid argument why we need XDP
> feature detection and checks. (Last part of LPC talk[1] were about
> XDP features).
>
> An interesting perspective you bring up, is that XDP features are not
> static per device driver.  It actually needs to be dynamic, as your
> XDP_TX feature request depend on the queue resources available.
>
> Implementation wise, I would not add flags to xdp_buff / xdp_md.
> Instead I propose in[1] slide 46, that the verifier should detect the
> XDP features used by a BPF-prog.  If you XDP prog doesn't use e.g.
> XDP_TX, then you should be allowed to run it on a virtio_net device
> with less queue configured, right?


Yes, I think so. But I remember we used to have something like 
header_adjust in the past but finally removed ...

Thanks


>
>
> [1] http://people.netfilter.org/hawk/presentations/LinuxPlumbers2019/xdp-distro-view.pdf

Re: error loading xdp program on virtio nic

[Jason Wang]

On 2019/11/22 下午11:43, David Ahern wrote:
> On 11/21/19 11:09 PM, Jason Wang wrote:
>>> Doubling the number of queues for each tap device adds overhead to the
>>> hypervisor if you only want to allow XDP_DROP or XDP_DIRECT. Am I
>>> understanding that correctly?
>>
>> Yes, but there's almost impossible to know whether or not XDP_TX will be
>> used by the program. If we don't use per CPU TX queue, it must be
>> serialized through locks, not sure it's worth try that (not by default,
>> of course).
>>
> This restriction is going to prevent use of XDP in VMs in general cloud
> hosting environments. 2x vhost threads for vcpus is a non-starter.
>
> If one XDP feature has high resource needs, then we need to subdivide
> the capabilities to let some work and others fail. For example, a flag
> can be added to xdp_buff / xdp_md that indicates supported XDP features.
> If there are insufficient resources for XDP_TX, do not show support for
> it. If a program returns XDP_TX anyways, packets will be dropped.
>

Or we can just:
- If queues is sufficient, using per-cpu TX queue
- If not, synchronize through spinlocks (like what TAP did right now, 
since there's no easy way to have more queues on the fly)

Thanks