XDP Native mode with public cloud (GCP)

XDP Native mode with public cloud (GCP)

[John McDowall]

Hi,

I am running the XDP redirect example on Google Cloud Platform using
the GCP provided Ubuntu 20.04 image as my host. The code works fine in
SKB_MODE (xdpgeneric). When I go to DRV_MODE I get the following error
message:   "libbpf: Kernel error message: virtio_net: XDP expects
header/data in single page, any_header_sg required".

For all images on GCP that I have looked at the number of RX and TX
channels are set to 0 and the combined is the number of cores (ethtool
-l dev). My question is then, is there any way to run XDP native on
GCP images or do I need to try and create a custom image with re and
tx queues set?

Second question, does anyone have experience running native XDP on AWS
and Azure?

Regards

John

Re: XDP Native mode with public cloud (GCP)

[David Ahern]

On 4/28/20 11:11 AM, John McDowall wrote:
> Hi,
> 
> I am running the XDP redirect example on Google Cloud Platform using
> the GCP provided Ubuntu 20.04 image as my host. The code works fine in
> SKB_MODE (xdpgeneric). When I go to DRV_MODE I get the following error
> message:   "libbpf: Kernel error message: virtio_net: XDP expects
> header/data in single page, any_header_sg required".

The most likely explanation is an old machine model for the VM.

But that is just the first step. Once the VM model is updated to
something modern, the cloud provider needs to enable multiqueue for the
tap device with the number of queues = 2 * vcpus for each tap device.
e.g., a 4 vcpu VM needs 8 queues per tap device that xdp is to be allowed.

> 
> For all images on GCP that I have looked at the number of RX and TX
> channels are set to 0 and the combined is the number of cores (ethtool
> -l dev). My question is then, is there any way to run XDP native on
> GCP images or do I need to try and create a custom image with re and
> tx queues set?
> 
> Second question, does anyone have experience running native XDP on AWS
> and Azure?
> 

last I looked AWS does not support XDP either. Never tried Azure.

Re: XDP Native mode with public cloud (GCP)

[John McDowall]

David,

Thanks for the quick reply - I assumed that this was the answer. The
good news is that I see almost 8Gbps in xdpgeneric mode.

Regards

John

On Tue, Apr 28, 2020 at 10:19 AM David Ahern <dsahern@gmail.com> wrote:
>
> On 4/28/20 11:11 AM, John McDowall wrote:
> > Hi,
> >
> > I am running the XDP redirect example on Google Cloud Platform using
> > the GCP provided Ubuntu 20.04 image as my host. The code works fine in
> > SKB_MODE (xdpgeneric). When I go to DRV_MODE I get the following error
> > message:   "libbpf: Kernel error message: virtio_net: XDP expects
> > header/data in single page, any_header_sg required".
>
> The most likely explanation is an old machine model for the VM.
>
> But that is just the first step. Once the VM model is updated to
> something modern, the cloud provider needs to enable multiqueue for the
> tap device with the number of queues = 2 * vcpus for each tap device.
> e.g., a 4 vcpu VM needs 8 queues per tap device that xdp is to be allowed.
>
> >
> > For all images on GCP that I have looked at the number of RX and TX
> > channels are set to 0 and the combined is the number of cores (ethtool
> > -l dev). My question is then, is there any way to run XDP native on
> > GCP images or do I need to try and create a custom image with re and
> > tx queues set?
> >
> > Second question, does anyone have experience running native XDP on AWS
> > and Azure?
> >
>
> last I looked AWS does not support XDP either. Never tried Azure.

Re: XDP Native mode with public cloud (GCP)

[Zvi Effron]

On Tue, Apr 28, 2020 at 12:19 PM David Ahern <dsahern@gmail.com> wrote:
> last I looked AWS does not support XDP either. Never tried Azure.

XDP just recently started supporting XDP. You will need their
out-of-tree driver from https://github.com/amzn/amzn-drivers/. It does
not support XDP_REDIRECT, and I don't think it supports AF_XDP, but it
does support XDP_PASS, XDP_DROP, and XDP_TX.

I have verified this today on a Fedora 31 based image, though it does
require a few modifications to get the RPM to build.

--Zvi

Re: XDP Native mode with public cloud (GCP)

[John McDowall]

Zvi,

Thanks, this looks promising, I will take a deeper look and see what I
can get working.

Really appreciate the pointer

Regards

John

On Tue, Apr 28, 2020 at 6:52 PM Zvi Effron <zeffron@riotgames.com> wrote:
>
> XDP just recently started supporting XDP. You will need their out-of-tree driver from https://github.com/amzn/amzn-drivers/. It does not support XDP_REDIRECT, and I don't think it supports AF_XDP, but it does support XDP_PASS, XDP_DROP, and XDP_TX.
>
> I have verified this today on a Fedora 31 based image, though it does require a few modifications to get the RPM to build.
>
> --Zvi
>
> On Tue, Apr 28, 2020 at 7:52 PM John McDowall <jmcdowall@paloaltonetworks.com> wrote:
>>
>> David,
>>
>> Thanks for the quick reply - I assumed that this was the answer. The
>> good news is that I see almost 8Gbps in xdpgeneric mode.
>>
>> Regards
>>
>> John
>>
>> On Tue, Apr 28, 2020 at 10:19 AM David Ahern <dsahern@gmail.com> wrote:
>> >
>> > On 4/28/20 11:11 AM, John McDowall wrote:
>> > > Hi,
>> > >
>> > > I am running the XDP redirect example on Google Cloud Platform using
>> > > the GCP provided Ubuntu 20.04 image as my host. The code works fine in
>> > > SKB_MODE (xdpgeneric). When I go to DRV_MODE I get the following error
>> > > message:   "libbpf: Kernel error message: virtio_net: XDP expects
>> > > header/data in single page, any_header_sg required".
>> >
>> > The most likely explanation is an old machine model for the VM.
>> >
>> > But that is just the first step. Once the VM model is updated to
>> > something modern, the cloud provider needs to enable multiqueue for the
>> > tap device with the number of queues = 2 * vcpus for each tap device.
>> > e.g., a 4 vcpu VM needs 8 queues per tap device that xdp is to be allowed.
>> >
>> > >
>> > > For all images on GCP that I have looked at the number of RX and TX
>> > > channels are set to 0 and the combined is the number of cores (ethtool
>> > > -l dev). My question is then, is there any way to run XDP native on
>> > > GCP images or do I need to try and create a custom image with re and
>> > > tx queues set?
>> > >
>> > > Second question, does anyone have experience running native XDP on AWS
>> > > and Azure?
>> > >
>> >
>> > last I looked AWS does not support XDP either. Never tried Azure.