How to debug packet sends to virtual functions

How to debug packet sends to virtual functions

[Mats Liljegren]

I'm trying to get a modified version of the l2fwd example running, and
have problems with packets being silently thrown away. I can receive
packets, and my printf's indicates that the packets are being sent to
the correct port, using correct MAC address as source address. And
still, the packets are lost.

Since the port is a virtual function, it seems like I cannot use
tcpdump on it to see the network traffic. There is nothing coming out
of the cable (activity light not flashing, the receiving end running
tcpdump does not see any traffic).

I'm using two X540 with two virtual functions each. The application
runs in a KVM/Qemu environmen.

Any suggestions how to debug this?

Regards,
Mats

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

I have a follow-up on this:

ixgbe version 3.13.10-k
ixgbevf version 2.7.12-k

(These are what was provided by Ubuntu 13.10)

I tried the following sequence on the host, before starting the guest:
1) sudo rmmod ixgbe
2) sudo modprobe ixgbe max_vfs=2
3) sudo ifconfig em1 up  # This is the physical function
4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
5) ping 192.168.2.1

I can see that the ping request reaches its target, and a reply is
sent back. But this reply is not received by the ping shell command.

Why?

Regards,
Mats

On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
<liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I'm trying to get a modified version of the l2fwd example running, and
> have problems with packets being silently thrown away. I can receive
> packets, and my printf's indicates that the packets are being sent to
> the correct port, using correct MAC address as source address. And
> still, the packets are lost.
>
> Since the port is a virtual function, it seems like I cannot use
> tcpdump on it to see the network traffic. There is nothing coming out
> of the cable (activity light not flashing, the receiving end running
> tcpdump does not see any traffic).
>
> I'm using two X540 with two virtual functions each. The application
> runs in a KVM/Qemu environmen.
>
> Any suggestions how to debug this?
>
> Regards,
> Mats

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

Never mind, I was hit by the infamous MAC spoofing... I got it working
on both the host and the guest using ixgbevf driver, so apparently the
cables are correctly attached.

Using DPDK is still no-go. It can receive packets, but when sending
the packets the function returns success, but the driver reports
nothing (i.e. no errors, no sent packets, no nothing, except for
received packets of course).

What could cause this behavior?

Regards
Mats

On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
<liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I have a follow-up on this:
>
> ixgbe version 3.13.10-k
> ixgbevf version 2.7.12-k
>
> (These are what was provided by Ubuntu 13.10)
>
> I tried the following sequence on the host, before starting the guest:
> 1) sudo rmmod ixgbe
> 2) sudo modprobe ixgbe max_vfs=2
> 3) sudo ifconfig em1 up  # This is the physical function
> 4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
> 5) ping 192.168.2.1
>
> I can see that the ping request reaches its target, and a reply is
> sent back. But this reply is not received by the ping shell command.
>
> Why?
>
> Regards,
> Mats
>
> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> I'm trying to get a modified version of the l2fwd example running, and
>> have problems with packets being silently thrown away. I can receive
>> packets, and my printf's indicates that the packets are being sent to
>> the correct port, using correct MAC address as source address. And
>> still, the packets are lost.
>>
>> Since the port is a virtual function, it seems like I cannot use
>> tcpdump on it to see the network traffic. There is nothing coming out
>> of the cable (activity light not flashing, the receiving end running
>> tcpdump does not see any traffic).
>>
>> I'm using two X540 with two virtual functions each. The application
>> runs in a KVM/Qemu environmen.
>>
>> Any suggestions how to debug this?
>>
>> Regards,
>> Mats

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

This is my fourth mail in my desperate attempt to get DPDK running in
KVM and no comments so far, not even any "it works for me". Am I the
only one crazy enough to believe that this can be done?

Anyway, out of desperation I tried to get it running without having
ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed
the instructions in the Programmer's Guide, chapter "Setting Up a KVM
Virtual Machine Monitor", using the PMD version of the instructions.

I get as far as being able to see my four virtual functions in the
guest using "lspci". But starting the DPDK application gives me the
following error:

PMD:    The MAC address is not valid.
        The most likely cause of this error is that the VM host
        has not assigned a valid MAC address to this VF device.
        Please consult the DPDK Release Notes (FAQ section) for
        a possible solution to this problem.

This may be true, but without any kernel modules loaded, how am I
supposed to change any MAC addresses? Can this be done from within
DPDK?

As a side-note, I did try to load ixgbevf in the guest, but it
produced no interfaces. There was no error messages in the syslog
though.

Is it possible to get X540 working in a guest or should I switch hardware?

Since the instructions assumes I know the command line to KVM to start
my guest (which I do not), I cannot followed them precisely. I use
virsh and XML file, and maybe I've misunderstood how to translate the
pci-assign parameter to XML code. I currently use a <hostdev> entry,
but I've also tried <interface type='hostdev'>. Neither has been
working for me so far, though the <interface> version got me as far as
being able to receive packets at least, but not transmitting.

Regards
Mats


On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren
<liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Never mind, I was hit by the infamous MAC spoofing... I got it working
> on both the host and the guest using ixgbevf driver, so apparently the
> cables are correctly attached.
>
> Using DPDK is still no-go. It can receive packets, but when sending
> the packets the function returns success, but the driver reports
> nothing (i.e. no errors, no sent packets, no nothing, except for
> received packets of course).
>
> What could cause this behavior?
>
> Regards
> Mats
>
> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> I have a follow-up on this:
>>
>> ixgbe version 3.13.10-k
>> ixgbevf version 2.7.12-k
>>
>> (These are what was provided by Ubuntu 13.10)
>>
>> I tried the following sequence on the host, before starting the guest:
>> 1) sudo rmmod ixgbe
>> 2) sudo modprobe ixgbe max_vfs=2
>> 3) sudo ifconfig em1 up  # This is the physical function
>> 4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
>> 5) ping 192.168.2.1
>>
>> I can see that the ping request reaches its target, and a reply is
>> sent back. But this reply is not received by the ping shell command.
>>
>> Why?
>>
>> Regards,
>> Mats
>>
>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> I'm trying to get a modified version of the l2fwd example running, and
>>> have problems with packets being silently thrown away. I can receive
>>> packets, and my printf's indicates that the packets are being sent to
>>> the correct port, using correct MAC address as source address. And
>>> still, the packets are lost.
>>>
>>> Since the port is a virtual function, it seems like I cannot use
>>> tcpdump on it to see the network traffic. There is nothing coming out
>>> of the cable (activity light not flashing, the receiving end running
>>> tcpdump does not see any traffic).
>>>
>>> I'm using two X540 with two virtual functions each. The application
>>> runs in a KVM/Qemu environmen.
>>>
>>> Any suggestions how to debug this?
>>>
>>> Regards,
>>> Mats

Re: How to debug packet sends to virtual functions

[jigsaw]

Hi Mats,

I've tried vf with 82599EB and it works fine. As long as the VF is
visible in guest, DPDK's VF driver should work just as ixgbevf, which
shares more or less the same code.

I don't understand why you would expect DPDK@guest to work as VF,
while the host has no ixgbe loaded.
To make further debug I'd suggest compile ixgbe driver with your own
syslogs. At least you will be able to see the signalling between vf
and ixgbe drivers.

-Qinglai


On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren
<liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> This is my fourth mail in my desperate attempt to get DPDK running in
> KVM and no comments so far, not even any "it works for me". Am I the
> only one crazy enough to believe that this can be done?
>
> Anyway, out of desperation I tried to get it running without having
> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed
> the instructions in the Programmer's Guide, chapter "Setting Up a KVM
> Virtual Machine Monitor", using the PMD version of the instructions.
>
> I get as far as being able to see my four virtual functions in the
> guest using "lspci". But starting the DPDK application gives me the
> following error:
>
> PMD:    The MAC address is not valid.
>         The most likely cause of this error is that the VM host
>         has not assigned a valid MAC address to this VF device.
>         Please consult the DPDK Release Notes (FAQ section) for
>         a possible solution to this problem.
>
> This may be true, but without any kernel modules loaded, how am I
> supposed to change any MAC addresses? Can this be done from within
> DPDK?
>
> As a side-note, I did try to load ixgbevf in the guest, but it
> produced no interfaces. There was no error messages in the syslog
> though.
>
> Is it possible to get X540 working in a guest or should I switch hardware?
>
> Since the instructions assumes I know the command line to KVM to start
> my guest (which I do not), I cannot followed them precisely. I use
> virsh and XML file, and maybe I've misunderstood how to translate the
> pci-assign parameter to XML code. I currently use a <hostdev> entry,
> but I've also tried <interface type='hostdev'>. Neither has been
> working for me so far, though the <interface> version got me as far as
> being able to receive packets at least, but not transmitting.
>
> Regards
> Mats
>
>
> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren
> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Never mind, I was hit by the infamous MAC spoofing... I got it working
>> on both the host and the guest using ixgbevf driver, so apparently the
>> cables are correctly attached.
>>
>> Using DPDK is still no-go. It can receive packets, but when sending
>> the packets the function returns success, but the driver reports
>> nothing (i.e. no errors, no sent packets, no nothing, except for
>> received packets of course).
>>
>> What could cause this behavior?
>>
>> Regards
>> Mats
>>
>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> I have a follow-up on this:
>>>
>>> ixgbe version 3.13.10-k
>>> ixgbevf version 2.7.12-k
>>>
>>> (These are what was provided by Ubuntu 13.10)
>>>
>>> I tried the following sequence on the host, before starting the guest:
>>> 1) sudo rmmod ixgbe
>>> 2) sudo modprobe ixgbe max_vfs=2
>>> 3) sudo ifconfig em1 up  # This is the physical function
>>> 4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
>>> 5) ping 192.168.2.1
>>>
>>> I can see that the ping request reaches its target, and a reply is
>>> sent back. But this reply is not received by the ping shell command.
>>>
>>> Why?
>>>
>>> Regards,
>>> Mats
>>>
>>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> I'm trying to get a modified version of the l2fwd example running, and
>>>> have problems with packets being silently thrown away. I can receive
>>>> packets, and my printf's indicates that the packets are being sent to
>>>> the correct port, using correct MAC address as source address. And
>>>> still, the packets are lost.
>>>>
>>>> Since the port is a virtual function, it seems like I cannot use
>>>> tcpdump on it to see the network traffic. There is nothing coming out
>>>> of the cable (activity light not flashing, the receiving end running
>>>> tcpdump does not see any traffic).
>>>>
>>>> I'm using two X540 with two virtual functions each. The application
>>>> runs in a KVM/Qemu environmen.
>>>>
>>>> Any suggestions how to debug this?
>>>>
>>>> Regards,
>>>> Mats

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

Hi Qinglai,

Thanks for the response!

My previous attempt was with ixgbe loaded in the host. I also needed
to load ixgbevf, but this seems to be because of a short-coming in
libvirt. Maybe loading ixgbevf and then unbind it when running the
guest is what causes my problems. I could receive packets with this
setup, but not transmit. I used extra debug to syslog, and it showed
that packets was placed in the transmit queue, but these packets was
never sent.

I'll see if I can get this working without loading ixgbevf in the host.

What instructions did you follow to get this working? Did you do any
deviation from the instructions?

Regards
Mats

On Tue, Feb 4, 2014 at 11:26 AM, jigsaw <jigsaw-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi Mats,
>
> I've tried vf with 82599EB and it works fine. As long as the VF is
> visible in guest, DPDK's VF driver should work just as ixgbevf, which
> shares more or less the same code.
>
> I don't understand why you would expect DPDK@guest to work as VF,
> while the host has no ixgbe loaded.
> To make further debug I'd suggest compile ixgbe driver with your own
> syslogs. At least you will be able to see the signalling between vf
> and ixgbe drivers.
>
> -Qinglai
>
>
> On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren
> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> This is my fourth mail in my desperate attempt to get DPDK running in
>> KVM and no comments so far, not even any "it works for me". Am I the
>> only one crazy enough to believe that this can be done?
>>
>> Anyway, out of desperation I tried to get it running without having
>> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed
>> the instructions in the Programmer's Guide, chapter "Setting Up a KVM
>> Virtual Machine Monitor", using the PMD version of the instructions.
>>
>> I get as far as being able to see my four virtual functions in the
>> guest using "lspci". But starting the DPDK application gives me the
>> following error:
>>
>> PMD:    The MAC address is not valid.
>>         The most likely cause of this error is that the VM host
>>         has not assigned a valid MAC address to this VF device.
>>         Please consult the DPDK Release Notes (FAQ section) for
>>         a possible solution to this problem.
>>
>> This may be true, but without any kernel modules loaded, how am I
>> supposed to change any MAC addresses? Can this be done from within
>> DPDK?
>>
>> As a side-note, I did try to load ixgbevf in the guest, but it
>> produced no interfaces. There was no error messages in the syslog
>> though.
>>
>> Is it possible to get X540 working in a guest or should I switch hardware?
>>
>> Since the instructions assumes I know the command line to KVM to start
>> my guest (which I do not), I cannot followed them precisely. I use
>> virsh and XML file, and maybe I've misunderstood how to translate the
>> pci-assign parameter to XML code. I currently use a <hostdev> entry,
>> but I've also tried <interface type='hostdev'>. Neither has been
>> working for me so far, though the <interface> version got me as far as
>> being able to receive packets at least, but not transmitting.
>>
>> Regards
>> Mats
>>
>>
>> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren
>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> Never mind, I was hit by the infamous MAC spoofing... I got it working
>>> on both the host and the guest using ixgbevf driver, so apparently the
>>> cables are correctly attached.
>>>
>>> Using DPDK is still no-go. It can receive packets, but when sending
>>> the packets the function returns success, but the driver reports
>>> nothing (i.e. no errors, no sent packets, no nothing, except for
>>> received packets of course).
>>>
>>> What could cause this behavior?
>>>
>>> Regards
>>> Mats
>>>
>>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> I have a follow-up on this:
>>>>
>>>> ixgbe version 3.13.10-k
>>>> ixgbevf version 2.7.12-k
>>>>
>>>> (These are what was provided by Ubuntu 13.10)
>>>>
>>>> I tried the following sequence on the host, before starting the guest:
>>>> 1) sudo rmmod ixgbe
>>>> 2) sudo modprobe ixgbe max_vfs=2
>>>> 3) sudo ifconfig em1 up  # This is the physical function
>>>> 4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
>>>> 5) ping 192.168.2.1
>>>>
>>>> I can see that the ping request reaches its target, and a reply is
>>>> sent back. But this reply is not received by the ping shell command.
>>>>
>>>> Why?
>>>>
>>>> Regards,
>>>> Mats
>>>>
>>>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
>>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>> I'm trying to get a modified version of the l2fwd example running, and
>>>>> have problems with packets being silently thrown away. I can receive
>>>>> packets, and my printf's indicates that the packets are being sent to
>>>>> the correct port, using correct MAC address as source address. And
>>>>> still, the packets are lost.
>>>>>
>>>>> Since the port is a virtual function, it seems like I cannot use
>>>>> tcpdump on it to see the network traffic. There is nothing coming out
>>>>> of the cable (activity light not flashing, the receiving end running
>>>>> tcpdump does not see any traffic).
>>>>>
>>>>> I'm using two X540 with two virtual functions each. The application
>>>>> runs in a KVM/Qemu environmen.
>>>>>
>>>>> Any suggestions how to debug this?
>>>>>
>>>>> Regards,
>>>>> Mats

Re: How to debug packet sends to virtual functions

[jigsaw]

Hi Mats,

I didn't have any deviation. What I did is just loading ixgbe (with
extra params for vf, as you mentioned in your first email),
and DPDK is up and running in guest. Of course I also followed section
8.10 of the DPDK release notes.

I can switch between DPDK and ixgebvf in guest at runtime and
everything works fine.

Sorry I can't help to debug coz I have only 82599EB at hand.

-Qinglai

On Tue, Feb 4, 2014 at 12:45 PM, Mats Liljegren
<liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi Qinglai,
>
> Thanks for the response!
>
> My previous attempt was with ixgbe loaded in the host. I also needed
> to load ixgbevf, but this seems to be because of a short-coming in
> libvirt. Maybe loading ixgbevf and then unbind it when running the
> guest is what causes my problems. I could receive packets with this
> setup, but not transmit. I used extra debug to syslog, and it showed
> that packets was placed in the transmit queue, but these packets was
> never sent.
>
> I'll see if I can get this working without loading ixgbevf in the host.
>
> What instructions did you follow to get this working? Did you do any
> deviation from the instructions?
>
> Regards
> Mats
>
> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw <jigsaw-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Hi Mats,
>>
>> I've tried vf with 82599EB and it works fine. As long as the VF is
>> visible in guest, DPDK's VF driver should work just as ixgbevf, which
>> shares more or less the same code.
>>
>> I don't understand why you would expect DPDK@guest to work as VF,
>> while the host has no ixgbe loaded.
>> To make further debug I'd suggest compile ixgbe driver with your own
>> syslogs. At least you will be able to see the signalling between vf
>> and ixgbe drivers.
>>
>> -Qinglai
>>
>>
>> On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren
>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> This is my fourth mail in my desperate attempt to get DPDK running in
>>> KVM and no comments so far, not even any "it works for me". Am I the
>>> only one crazy enough to believe that this can be done?
>>>
>>> Anyway, out of desperation I tried to get it running without having
>>> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed
>>> the instructions in the Programmer's Guide, chapter "Setting Up a KVM
>>> Virtual Machine Monitor", using the PMD version of the instructions.
>>>
>>> I get as far as being able to see my four virtual functions in the
>>> guest using "lspci". But starting the DPDK application gives me the
>>> following error:
>>>
>>> PMD:    The MAC address is not valid.
>>>         The most likely cause of this error is that the VM host
>>>         has not assigned a valid MAC address to this VF device.
>>>         Please consult the DPDK Release Notes (FAQ section) for
>>>         a possible solution to this problem.
>>>
>>> This may be true, but without any kernel modules loaded, how am I
>>> supposed to change any MAC addresses? Can this be done from within
>>> DPDK?
>>>
>>> As a side-note, I did try to load ixgbevf in the guest, but it
>>> produced no interfaces. There was no error messages in the syslog
>>> though.
>>>
>>> Is it possible to get X540 working in a guest or should I switch hardware?
>>>
>>> Since the instructions assumes I know the command line to KVM to start
>>> my guest (which I do not), I cannot followed them precisely. I use
>>> virsh and XML file, and maybe I've misunderstood how to translate the
>>> pci-assign parameter to XML code. I currently use a <hostdev> entry,
>>> but I've also tried <interface type='hostdev'>. Neither has been
>>> working for me so far, though the <interface> version got me as far as
>>> being able to receive packets at least, but not transmitting.
>>>
>>> Regards
>>> Mats
>>>
>>>
>>> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren
>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> Never mind, I was hit by the infamous MAC spoofing... I got it working
>>>> on both the host and the guest using ixgbevf driver, so apparently the
>>>> cables are correctly attached.
>>>>
>>>> Using DPDK is still no-go. It can receive packets, but when sending
>>>> the packets the function returns success, but the driver reports
>>>> nothing (i.e. no errors, no sent packets, no nothing, except for
>>>> received packets of course).
>>>>
>>>> What could cause this behavior?
>>>>
>>>> Regards
>>>> Mats
>>>>
>>>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
>>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>> I have a follow-up on this:
>>>>>
>>>>> ixgbe version 3.13.10-k
>>>>> ixgbevf version 2.7.12-k
>>>>>
>>>>> (These are what was provided by Ubuntu 13.10)
>>>>>
>>>>> I tried the following sequence on the host, before starting the guest:
>>>>> 1) sudo rmmod ixgbe
>>>>> 2) sudo modprobe ixgbe max_vfs=2
>>>>> 3) sudo ifconfig em1 up  # This is the physical function
>>>>> 4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
>>>>> 5) ping 192.168.2.1
>>>>>
>>>>> I can see that the ping request reaches its target, and a reply is
>>>>> sent back. But this reply is not received by the ping shell command.
>>>>>
>>>>> Why?
>>>>>
>>>>> Regards,
>>>>> Mats
>>>>>
>>>>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
>>>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>>> I'm trying to get a modified version of the l2fwd example running, and
>>>>>> have problems with packets being silently thrown away. I can receive
>>>>>> packets, and my printf's indicates that the packets are being sent to
>>>>>> the correct port, using correct MAC address as source address. And
>>>>>> still, the packets are lost.
>>>>>>
>>>>>> Since the port is a virtual function, it seems like I cannot use
>>>>>> tcpdump on it to see the network traffic. There is nothing coming out
>>>>>> of the cable (activity light not flashing, the receiving end running
>>>>>> tcpdump does not see any traffic).
>>>>>>
>>>>>> I'm using two X540 with two virtual functions each. The application
>>>>>> runs in a KVM/Qemu environmen.
>>>>>>
>>>>>> Any suggestions how to debug this?
>>>>>>
>>>>>> Regards,
>>>>>> Mats

Re: How to debug packet sends to virtual functions

[Burakov, Anatoly]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Mats Liljegren
> Sent: Tuesday, February 04, 2014 10:45 AM
> To: jigsaw
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] How to debug packet sends to virtual functions
> 
> Hi Qinglai,
> 
> Thanks for the response!
> 
> My previous attempt was with ixgbe loaded in the host. I also needed to load
> ixgbevf, but this seems to be because of a short-coming in libvirt. Maybe
> loading ixgbevf and then unbind it when running the guest is what causes my
> problems. I could receive packets with this setup, but not transmit. I used
> extra debug to syslog, and it showed that packets was placed in the transmit
> queue, but these packets was never sent.
> 
> I'll see if I can get this working without loading ixgbevf in the host.
> 
> What instructions did you follow to get this working? Did you do any
> deviation from the instructions?
> 
> Regards
> Mats
> 
> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw <jigsaw@gmail.com> wrote:
> > Hi Mats,
> >
> > I've tried vf with 82599EB and it works fine. As long as the VF is
> > visible in guest, DPDK's VF driver should work just as ixgbevf, which
> > shares more or less the same code.
> >
> > I don't understand why you would expect DPDK@guest to work as VF,
> > while the host has no ixgbe loaded.
> > To make further debug I'd suggest compile ixgbe driver with your own
> > syslogs. At least you will be able to see the signalling between vf
> > and ixgbe drivers.
> >
> > -Qinglai
> >
> >
> > On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren
> > <liljegren.mats2@gmail.com> wrote:
> >> This is my fourth mail in my desperate attempt to get DPDK running in
> >> KVM and no comments so far, not even any "it works for me". Am I the
> >> only one crazy enough to believe that this can be done?
> >>
> >> Anyway, out of desperation I tried to get it running without having
> >> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed
> >> the instructions in the Programmer's Guide, chapter "Setting Up a KVM
> >> Virtual Machine Monitor", using the PMD version of the instructions.
> >>
> >> I get as far as being able to see my four virtual functions in the
> >> guest using "lspci". But starting the DPDK application gives me the
> >> following error:
> >>
> >> PMD:    The MAC address is not valid.
> >>         The most likely cause of this error is that the VM host
> >>         has not assigned a valid MAC address to this VF device.
> >>         Please consult the DPDK Release Notes (FAQ section) for
> >>         a possible solution to this problem.
> >>
> >> This may be true, but without any kernel modules loaded, how am I
> >> supposed to change any MAC addresses? Can this be done from within
> >> DPDK?
> >>
> >> As a side-note, I did try to load ixgbevf in the guest, but it
> >> produced no interfaces. There was no error messages in the syslog
> >> though.
> >>
> >> Is it possible to get X540 working in a guest or should I switch hardware?
> >>
> >> Since the instructions assumes I know the command line to KVM to
> >> start my guest (which I do not), I cannot followed them precisely. I
> >> use virsh and XML file, and maybe I've misunderstood how to translate
> >> the pci-assign parameter to XML code. I currently use a <hostdev>
> >> entry, but I've also tried <interface type='hostdev'>. Neither has
> >> been working for me so far, though the <interface> version got me as
> >> far as being able to receive packets at least, but not transmitting.
> >>
> >> Regards
> >> Mats
> >>
> >>
> >> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren
> >> <liljegren.mats2@gmail.com> wrote:
> >>> Never mind, I was hit by the infamous MAC spoofing... I got it
> >>> working on both the host and the guest using ixgbevf driver, so
> >>> apparently the cables are correctly attached.
> >>>
> >>> Using DPDK is still no-go. It can receive packets, but when sending
> >>> the packets the function returns success, but the driver reports
> >>> nothing (i.e. no errors, no sent packets, no nothing, except for
> >>> received packets of course).
> >>>
> >>> What could cause this behavior?
> >>>
> >>> Regards
> >>> Mats
> >>>
> >>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
> >>> <liljegren.mats2@gmail.com> wrote:
> >>>> I have a follow-up on this:
> >>>>
> >>>> ixgbe version 3.13.10-k
> >>>> ixgbevf version 2.7.12-k
> >>>>
> >>>> (These are what was provided by Ubuntu 13.10)
> >>>>
> >>>> I tried the following sequence on the host, before starting the guest:
> >>>> 1) sudo rmmod ixgbe
> >>>> 2) sudo modprobe ixgbe max_vfs=2
> >>>> 3) sudo ifconfig em1 up  # This is the physical function
> >>>> 4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
> >>>> 5) ping 192.168.2.1
> >>>>
> >>>> I can see that the ping request reaches its target, and a reply is
> >>>> sent back. But this reply is not received by the ping shell command.
> >>>>
> >>>> Why?
> >>>>
> >>>> Regards,
> >>>> Mats
> >>>>
> >>>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
> >>>> <liljegren.mats2@gmail.com> wrote:
> >>>>> I'm trying to get a modified version of the l2fwd example running,
> >>>>> and have problems with packets being silently thrown away. I can
> >>>>> receive packets, and my printf's indicates that the packets are
> >>>>> being sent to the correct port, using correct MAC address as
> >>>>> source address. And still, the packets are lost.
> >>>>>
> >>>>> Since the port is a virtual function, it seems like I cannot use
> >>>>> tcpdump on it to see the network traffic. There is nothing coming
> >>>>> out of the cable (activity light not flashing, the receiving end
> >>>>> running tcpdump does not see any traffic).
> >>>>>
> >>>>> I'm using two X540 with two virtual functions each. The
> >>>>> application runs in a KVM/Qemu environmen.
> >>>>>
> >>>>> Any suggestions how to debug this?
> >>>>>
> >>>>> Regards,
> >>>>> Mats

Hi Mats

Basically, there is no way to change MAC address of a VF device from within the VM, you have to do it on the host. So, whatever the FAQ section describes (i.e. ip link ...) has to be done on the host, not inside the VM. That is, you have to load ixgbe kernel module on the host with max_vfs= set to whatever number of VF devices you need, and then go through each VF device and assign a MAC address via the "ip link" command, as described in the FAQ section of the DPDK Release Notes. You can then load either ixgbevf or DPDK VF drivers on the guest, and they will initialize correctly.

Best regards,
Anatoly Burakov
DPDK SW Engineer
--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

Hi Qinglai,

How did you configure the PCI passthrough in KVM? Did you use command
line parameter as described in the instructions, or do you use virsh
XML files, or maybe using virt-manager?

The steps that works best (i.e. can receive but not transmit) are:

1) sudo modprobe ixgbe max_vfs=2
2) sudo ip link set em1 vf 0 mac <phys mac>
3) sudo ifconfig em1 up
4) sudo modprobe pci-stub
5) echo "8086 1515" | sudo tee /sys/bus/pci/drivers/pci-stub/new_id
6) echo "0000:02:10.0" | sudo tee
/sys/bus/pci/devices/0000\:02\:10.0/driver/unbind
7) echo "0000:02:10.0" | sudo tee /sys/bus/pci/drivers/pci-stub/bind
8) virsh create vm vm.xml

The guest then does:

5) sudo modprobe uio
6) sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio 0000:02:10.0
7) sudo $RTE_SDK/tools/pci_unbind.py -b igb_uio
8) sudo <run DPDK app>

The virsh xml file has the following entry for the virtual function:
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x0' />
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
    </hostdev>

I actually have two virtual functions and two physical functions, but
I just repeat above for each.

Regards
Mats

On Tue, Feb 4, 2014 at 12:14 PM, jigsaw <jigsaw-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi Mats,
>
> I didn't have any deviation. What I did is just loading ixgbe (with
> extra params for vf, as you mentioned in your first email),
> and DPDK is up and running in guest. Of course I also followed section
> 8.10 of the DPDK release notes.
>
> I can switch between DPDK and ixgebvf in guest at runtime and
> everything works fine.
>
> Sorry I can't help to debug coz I have only 82599EB at hand.
>
> -Qinglai
>
> On Tue, Feb 4, 2014 at 12:45 PM, Mats Liljegren
> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Hi Qinglai,
>>
>> Thanks for the response!
>>
>> My previous attempt was with ixgbe loaded in the host. I also needed
>> to load ixgbevf, but this seems to be because of a short-coming in
>> libvirt. Maybe loading ixgbevf and then unbind it when running the
>> guest is what causes my problems. I could receive packets with this
>> setup, but not transmit. I used extra debug to syslog, and it showed
>> that packets was placed in the transmit queue, but these packets was
>> never sent.
>>
>> I'll see if I can get this working without loading ixgbevf in the host.
>>
>> What instructions did you follow to get this working? Did you do any
>> deviation from the instructions?
>>
>> Regards
>> Mats
>>
>> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw <jigsaw-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> Hi Mats,
>>>
>>> I've tried vf with 82599EB and it works fine. As long as the VF is
>>> visible in guest, DPDK's VF driver should work just as ixgbevf, which
>>> shares more or less the same code.
>>>
>>> I don't understand why you would expect DPDK@guest to work as VF,
>>> while the host has no ixgbe loaded.
>>> To make further debug I'd suggest compile ixgbe driver with your own
>>> syslogs. At least you will be able to see the signalling between vf
>>> and ixgbe drivers.
>>>
>>> -Qinglai
>>>
>>>
>>> On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren
>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> This is my fourth mail in my desperate attempt to get DPDK running in
>>>> KVM and no comments so far, not even any "it works for me". Am I the
>>>> only one crazy enough to believe that this can be done?
>>>>
>>>> Anyway, out of desperation I tried to get it running without having
>>>> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed
>>>> the instructions in the Programmer's Guide, chapter "Setting Up a KVM
>>>> Virtual Machine Monitor", using the PMD version of the instructions.
>>>>
>>>> I get as far as being able to see my four virtual functions in the
>>>> guest using "lspci". But starting the DPDK application gives me the
>>>> following error:
>>>>
>>>> PMD:    The MAC address is not valid.
>>>>         The most likely cause of this error is that the VM host
>>>>         has not assigned a valid MAC address to this VF device.
>>>>         Please consult the DPDK Release Notes (FAQ section) for
>>>>         a possible solution to this problem.
>>>>
>>>> This may be true, but without any kernel modules loaded, how am I
>>>> supposed to change any MAC addresses? Can this be done from within
>>>> DPDK?
>>>>
>>>> As a side-note, I did try to load ixgbevf in the guest, but it
>>>> produced no interfaces. There was no error messages in the syslog
>>>> though.
>>>>
>>>> Is it possible to get X540 working in a guest or should I switch hardware?
>>>>
>>>> Since the instructions assumes I know the command line to KVM to start
>>>> my guest (which I do not), I cannot followed them precisely. I use
>>>> virsh and XML file, and maybe I've misunderstood how to translate the
>>>> pci-assign parameter to XML code. I currently use a <hostdev> entry,
>>>> but I've also tried <interface type='hostdev'>. Neither has been
>>>> working for me so far, though the <interface> version got me as far as
>>>> being able to receive packets at least, but not transmitting.
>>>>
>>>> Regards
>>>> Mats
>>>>
>>>>
>>>> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren
>>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>> Never mind, I was hit by the infamous MAC spoofing... I got it working
>>>>> on both the host and the guest using ixgbevf driver, so apparently the
>>>>> cables are correctly attached.
>>>>>
>>>>> Using DPDK is still no-go. It can receive packets, but when sending
>>>>> the packets the function returns success, but the driver reports
>>>>> nothing (i.e. no errors, no sent packets, no nothing, except for
>>>>> received packets of course).
>>>>>
>>>>> What could cause this behavior?
>>>>>
>>>>> Regards
>>>>> Mats
>>>>>
>>>>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
>>>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>>> I have a follow-up on this:
>>>>>>
>>>>>> ixgbe version 3.13.10-k
>>>>>> ixgbevf version 2.7.12-k
>>>>>>
>>>>>> (These are what was provided by Ubuntu 13.10)
>>>>>>
>>>>>> I tried the following sequence on the host, before starting the guest:
>>>>>> 1) sudo rmmod ixgbe
>>>>>> 2) sudo modprobe ixgbe max_vfs=2
>>>>>> 3) sudo ifconfig em1 up  # This is the physical function
>>>>>> 4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
>>>>>> 5) ping 192.168.2.1
>>>>>>
>>>>>> I can see that the ping request reaches its target, and a reply is
>>>>>> sent back. But this reply is not received by the ping shell command.
>>>>>>
>>>>>> Why?
>>>>>>
>>>>>> Regards,
>>>>>> Mats
>>>>>>
>>>>>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
>>>>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>>>> I'm trying to get a modified version of the l2fwd example running, and
>>>>>>> have problems with packets being silently thrown away. I can receive
>>>>>>> packets, and my printf's indicates that the packets are being sent to
>>>>>>> the correct port, using correct MAC address as source address. And
>>>>>>> still, the packets are lost.
>>>>>>>
>>>>>>> Since the port is a virtual function, it seems like I cannot use
>>>>>>> tcpdump on it to see the network traffic. There is nothing coming out
>>>>>>> of the cable (activity light not flashing, the receiving end running
>>>>>>> tcpdump does not see any traffic).
>>>>>>>
>>>>>>> I'm using two X540 with two virtual functions each. The application
>>>>>>> runs in a KVM/Qemu environmen.
>>>>>>>
>>>>>>> Any suggestions how to debug this?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Mats

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

Hi Anatoly,

Thanks for the clarification. Then I can't use those instructions,
which is what I was starting to suspect. I have to find another way of
getting DPDK running in the guest then. Using ixgbevf in the guest
works fine, though.

Regards
Mats


On Tue, Feb 4, 2014 at 12:21 PM, Burakov, Anatoly
<anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> -----Original Message-----
>> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Mats Liljegren
>> Sent: Tuesday, February 04, 2014 10:45 AM
>> To: jigsaw
>> Cc: dev-VfR2kkLFssw@public.gmane.org
>> Subject: Re: [dpdk-dev] How to debug packet sends to virtual functions
>>
>> Hi Qinglai,
>>
>> Thanks for the response!
>>
>> My previous attempt was with ixgbe loaded in the host. I also needed to load
>> ixgbevf, but this seems to be because of a short-coming in libvirt. Maybe
>> loading ixgbevf and then unbind it when running the guest is what causes my
>> problems. I could receive packets with this setup, but not transmit. I used
>> extra debug to syslog, and it showed that packets was placed in the transmit
>> queue, but these packets was never sent.
>>
>> I'll see if I can get this working without loading ixgbevf in the host.
>>
>> What instructions did you follow to get this working? Did you do any
>> deviation from the instructions?
>>
>> Regards
>> Mats
>>
>> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw <jigsaw-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > Hi Mats,
>> >
>> > I've tried vf with 82599EB and it works fine. As long as the VF is
>> > visible in guest, DPDK's VF driver should work just as ixgbevf, which
>> > shares more or less the same code.
>> >
>> > I don't understand why you would expect DPDK@guest to work as VF,
>> > while the host has no ixgbe loaded.
>> > To make further debug I'd suggest compile ixgbe driver with your own
>> > syslogs. At least you will be able to see the signalling between vf
>> > and ixgbe drivers.
>> >
>> > -Qinglai
>> >
>> >
>> > On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren
>> > <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> >> This is my fourth mail in my desperate attempt to get DPDK running in
>> >> KVM and no comments so far, not even any "it works for me". Am I the
>> >> only one crazy enough to believe that this can be done?
>> >>
>> >> Anyway, out of desperation I tried to get it running without having
>> >> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed
>> >> the instructions in the Programmer's Guide, chapter "Setting Up a KVM
>> >> Virtual Machine Monitor", using the PMD version of the instructions.
>> >>
>> >> I get as far as being able to see my four virtual functions in the
>> >> guest using "lspci". But starting the DPDK application gives me the
>> >> following error:
>> >>
>> >> PMD:    The MAC address is not valid.
>> >>         The most likely cause of this error is that the VM host
>> >>         has not assigned a valid MAC address to this VF device.
>> >>         Please consult the DPDK Release Notes (FAQ section) for
>> >>         a possible solution to this problem.
>> >>
>> >> This may be true, but without any kernel modules loaded, how am I
>> >> supposed to change any MAC addresses? Can this be done from within
>> >> DPDK?
>> >>
>> >> As a side-note, I did try to load ixgbevf in the guest, but it
>> >> produced no interfaces. There was no error messages in the syslog
>> >> though.
>> >>
>> >> Is it possible to get X540 working in a guest or should I switch hardware?
>> >>
>> >> Since the instructions assumes I know the command line to KVM to
>> >> start my guest (which I do not), I cannot followed them precisely. I
>> >> use virsh and XML file, and maybe I've misunderstood how to translate
>> >> the pci-assign parameter to XML code. I currently use a <hostdev>
>> >> entry, but I've also tried <interface type='hostdev'>. Neither has
>> >> been working for me so far, though the <interface> version got me as
>> >> far as being able to receive packets at least, but not transmitting.
>> >>
>> >> Regards
>> >> Mats
>> >>
>> >>
>> >> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren
>> >> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> >>> Never mind, I was hit by the infamous MAC spoofing... I got it
>> >>> working on both the host and the guest using ixgbevf driver, so
>> >>> apparently the cables are correctly attached.
>> >>>
>> >>> Using DPDK is still no-go. It can receive packets, but when sending
>> >>> the packets the function returns success, but the driver reports
>> >>> nothing (i.e. no errors, no sent packets, no nothing, except for
>> >>> received packets of course).
>> >>>
>> >>> What could cause this behavior?
>> >>>
>> >>> Regards
>> >>> Mats
>> >>>
>> >>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
>> >>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> >>>> I have a follow-up on this:
>> >>>>
>> >>>> ixgbe version 3.13.10-k
>> >>>> ixgbevf version 2.7.12-k
>> >>>>
>> >>>> (These are what was provided by Ubuntu 13.10)
>> >>>>
>> >>>> I tried the following sequence on the host, before starting the guest:
>> >>>> 1) sudo rmmod ixgbe
>> >>>> 2) sudo modprobe ixgbe max_vfs=2
>> >>>> 3) sudo ifconfig em1 up  # This is the physical function
>> >>>> 4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
>> >>>> 5) ping 192.168.2.1
>> >>>>
>> >>>> I can see that the ping request reaches its target, and a reply is
>> >>>> sent back. But this reply is not received by the ping shell command.
>> >>>>
>> >>>> Why?
>> >>>>
>> >>>> Regards,
>> >>>> Mats
>> >>>>
>> >>>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
>> >>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> >>>>> I'm trying to get a modified version of the l2fwd example running,
>> >>>>> and have problems with packets being silently thrown away. I can
>> >>>>> receive packets, and my printf's indicates that the packets are
>> >>>>> being sent to the correct port, using correct MAC address as
>> >>>>> source address. And still, the packets are lost.
>> >>>>>
>> >>>>> Since the port is a virtual function, it seems like I cannot use
>> >>>>> tcpdump on it to see the network traffic. There is nothing coming
>> >>>>> out of the cable (activity light not flashing, the receiving end
>> >>>>> running tcpdump does not see any traffic).
>> >>>>>
>> >>>>> I'm using two X540 with two virtual functions each. The
>> >>>>> application runs in a KVM/Qemu environmen.
>> >>>>>
>> >>>>> Any suggestions how to debug this?
>> >>>>>
>> >>>>> Regards,
>> >>>>> Mats
>
> Hi Mats
>
> Basically, there is no way to change MAC address of a VF device from within the VM, you have to do it on the host. So, whatever the FAQ section describes (i.e. ip link ...) has to be done on the host, not inside the VM. That is, you have to load ixgbe kernel module on the host with max_vfs= set to whatever number of VF devices you need, and then go through each VF device and assign a MAC address via the "ip link" command, as described in the FAQ section of the DPDK Release Notes. You can then load either ixgbevf or DPDK VF drivers on the guest, and they will initialize correctly.
>
> Best regards,
> Anatoly Burakov
> DPDK SW Engineer
> --------------------------------------------------------------
> Intel Shannon Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
> Business address: Dromore House, East Park, Shannon, Co. Clare
>
> This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
>

Re: How to debug packet sends to virtual functions

[Burakov, Anatoly]

> -----Original Message-----
> From: Mats Liljegren [mailto:liljegren.mats2@gmail.com]
> Sent: Tuesday, February 04, 2014 11:48 AM
> To: Burakov, Anatoly
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] How to debug packet sends to virtual functions
> 
> Hi Anatoly,
> 
> Thanks for the clarification. Then I can't use those instructions, which is what I
> was starting to suspect. I have to find another way of getting DPDK running in
> the guest then. Using ixgbevf in the guest works fine, though.
> 
> Regards
> Mats
> 

Hi Mats

Let me clarify - do you run DPDK on both host and guest? E.g. you load igb_uio on the host as well? If not, I don't see why you can't use those instructions - you're creating VF devices on the host anyway, and that's where you should set their MAC addresses.

Best regards,
Anatoly Burakov
DPDK SW Engineer

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare

Re: How to debug packet sends to virtual functions

[jigsaw]

Hi Mats,

Yes the virsh xml is created in essentially the same way. And the
setup in host is also the same as yours.
But no PCI passthru. As I understand, in a PF-VF deployment it makes
no sense for PCI passthru, coz the PCI device must be visible and
under control of host. (Pls correct me if Im wrong).

-Qinglai

On Tue, Feb 4, 2014 at 1:31 PM, Mats Liljegren
<liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi Qinglai,
>
> How did you configure the PCI passthrough in KVM? Did you use command
> line parameter as described in the instructions, or do you use virsh
> XML files, or maybe using virt-manager?
>
> The steps that works best (i.e. can receive but not transmit) are:
>
> 1) sudo modprobe ixgbe max_vfs=2
> 2) sudo ip link set em1 vf 0 mac <phys mac>
> 3) sudo ifconfig em1 up
> 4) sudo modprobe pci-stub
> 5) echo "8086 1515" | sudo tee /sys/bus/pci/drivers/pci-stub/new_id
> 6) echo "0000:02:10.0" | sudo tee
> /sys/bus/pci/devices/0000\:02\:10.0/driver/unbind
> 7) echo "0000:02:10.0" | sudo tee /sys/bus/pci/drivers/pci-stub/bind
> 8) virsh create vm vm.xml
>
> The guest then does:
>
> 5) sudo modprobe uio
> 6) sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio 0000:02:10.0
> 7) sudo $RTE_SDK/tools/pci_unbind.py -b igb_uio
> 8) sudo <run DPDK app>
>
> The virsh xml file has the following entry for the virtual function:
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>     <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x0' />
>       </source>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>     </hostdev>
>
> I actually have two virtual functions and two physical functions, but
> I just repeat above for each.
>
> Regards
> Mats
>
> On Tue, Feb 4, 2014 at 12:14 PM, jigsaw <jigsaw-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Hi Mats,
>>
>> I didn't have any deviation. What I did is just loading ixgbe (with
>> extra params for vf, as you mentioned in your first email),
>> and DPDK is up and running in guest. Of course I also followed section
>> 8.10 of the DPDK release notes.
>>
>> I can switch between DPDK and ixgebvf in guest at runtime and
>> everything works fine.
>>
>> Sorry I can't help to debug coz I have only 82599EB at hand.
>>
>> -Qinglai
>>
>> On Tue, Feb 4, 2014 at 12:45 PM, Mats Liljegren
>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> Hi Qinglai,
>>>
>>> Thanks for the response!
>>>
>>> My previous attempt was with ixgbe loaded in the host. I also needed
>>> to load ixgbevf, but this seems to be because of a short-coming in
>>> libvirt. Maybe loading ixgbevf and then unbind it when running the
>>> guest is what causes my problems. I could receive packets with this
>>> setup, but not transmit. I used extra debug to syslog, and it showed
>>> that packets was placed in the transmit queue, but these packets was
>>> never sent.
>>>
>>> I'll see if I can get this working without loading ixgbevf in the host.
>>>
>>> What instructions did you follow to get this working? Did you do any
>>> deviation from the instructions?
>>>
>>> Regards
>>> Mats
>>>
>>> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw <jigsaw-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> Hi Mats,
>>>>
>>>> I've tried vf with 82599EB and it works fine. As long as the VF is
>>>> visible in guest, DPDK's VF driver should work just as ixgbevf, which
>>>> shares more or less the same code.
>>>>
>>>> I don't understand why you would expect DPDK@guest to work as VF,
>>>> while the host has no ixgbe loaded.
>>>> To make further debug I'd suggest compile ixgbe driver with your own
>>>> syslogs. At least you will be able to see the signalling between vf
>>>> and ixgbe drivers.
>>>>
>>>> -Qinglai
>>>>
>>>>
>>>> On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren
>>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>> This is my fourth mail in my desperate attempt to get DPDK running in
>>>>> KVM and no comments so far, not even any "it works for me". Am I the
>>>>> only one crazy enough to believe that this can be done?
>>>>>
>>>>> Anyway, out of desperation I tried to get it running without having
>>>>> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed
>>>>> the instructions in the Programmer's Guide, chapter "Setting Up a KVM
>>>>> Virtual Machine Monitor", using the PMD version of the instructions.
>>>>>
>>>>> I get as far as being able to see my four virtual functions in the
>>>>> guest using "lspci". But starting the DPDK application gives me the
>>>>> following error:
>>>>>
>>>>> PMD:    The MAC address is not valid.
>>>>>         The most likely cause of this error is that the VM host
>>>>>         has not assigned a valid MAC address to this VF device.
>>>>>         Please consult the DPDK Release Notes (FAQ section) for
>>>>>         a possible solution to this problem.
>>>>>
>>>>> This may be true, but without any kernel modules loaded, how am I
>>>>> supposed to change any MAC addresses? Can this be done from within
>>>>> DPDK?
>>>>>
>>>>> As a side-note, I did try to load ixgbevf in the guest, but it
>>>>> produced no interfaces. There was no error messages in the syslog
>>>>> though.
>>>>>
>>>>> Is it possible to get X540 working in a guest or should I switch hardware?
>>>>>
>>>>> Since the instructions assumes I know the command line to KVM to start
>>>>> my guest (which I do not), I cannot followed them precisely. I use
>>>>> virsh and XML file, and maybe I've misunderstood how to translate the
>>>>> pci-assign parameter to XML code. I currently use a <hostdev> entry,
>>>>> but I've also tried <interface type='hostdev'>. Neither has been
>>>>> working for me so far, though the <interface> version got me as far as
>>>>> being able to receive packets at least, but not transmitting.
>>>>>
>>>>> Regards
>>>>> Mats
>>>>>
>>>>>
>>>>> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren
>>>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>>> Never mind, I was hit by the infamous MAC spoofing... I got it working
>>>>>> on both the host and the guest using ixgbevf driver, so apparently the
>>>>>> cables are correctly attached.
>>>>>>
>>>>>> Using DPDK is still no-go. It can receive packets, but when sending
>>>>>> the packets the function returns success, but the driver reports
>>>>>> nothing (i.e. no errors, no sent packets, no nothing, except for
>>>>>> received packets of course).
>>>>>>
>>>>>> What could cause this behavior?
>>>>>>
>>>>>> Regards
>>>>>> Mats
>>>>>>
>>>>>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
>>>>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>>>> I have a follow-up on this:
>>>>>>>
>>>>>>> ixgbe version 3.13.10-k
>>>>>>> ixgbevf version 2.7.12-k
>>>>>>>
>>>>>>> (These are what was provided by Ubuntu 13.10)
>>>>>>>
>>>>>>> I tried the following sequence on the host, before starting the guest:
>>>>>>> 1) sudo rmmod ixgbe
>>>>>>> 2) sudo modprobe ixgbe max_vfs=2
>>>>>>> 3) sudo ifconfig em1 up  # This is the physical function
>>>>>>> 4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
>>>>>>> 5) ping 192.168.2.1
>>>>>>>
>>>>>>> I can see that the ping request reaches its target, and a reply is
>>>>>>> sent back. But this reply is not received by the ping shell command.
>>>>>>>
>>>>>>> Why?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Mats
>>>>>>>
>>>>>>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
>>>>>>> <liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>>>>> I'm trying to get a modified version of the l2fwd example running, and
>>>>>>>> have problems with packets being silently thrown away. I can receive
>>>>>>>> packets, and my printf's indicates that the packets are being sent to
>>>>>>>> the correct port, using correct MAC address as source address. And
>>>>>>>> still, the packets are lost.
>>>>>>>>
>>>>>>>> Since the port is a virtual function, it seems like I cannot use
>>>>>>>> tcpdump on it to see the network traffic. There is nothing coming out
>>>>>>>> of the cable (activity light not flashing, the receiving end running
>>>>>>>> tcpdump does not see any traffic).
>>>>>>>>
>>>>>>>> I'm using two X540 with two virtual functions each. The application
>>>>>>>> runs in a KVM/Qemu environmen.
>>>>>>>>
>>>>>>>> Any suggestions how to debug this?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Mats

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

Hi Anatoly,

Just realized that the instructions gave two alternatives for the
host: DPDK igb_uio or ixgbe. It was the former, DPDK igb_uio that I
was referring to as being the non-working alternative.

Loading ixgbe enables me to set the mac addresses. This is also what I
tried previously, when I both got ixgbevf working in the guest as well
as receive to work in DPDK in the guest. The problem is making
transmit work in DPDK in the guest.

Regards
Mats



On Tue, Feb 4, 2014 at 12:52 PM, Burakov, Anatoly
<anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> -----Original Message-----
>> From: Mats Liljegren [mailto:liljegren.mats2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]
>> Sent: Tuesday, February 04, 2014 11:48 AM
>> To: Burakov, Anatoly
>> Cc: dev-VfR2kkLFssw@public.gmane.org
>> Subject: Re: [dpdk-dev] How to debug packet sends to virtual functions
>>
>> Hi Anatoly,
>>
>> Thanks for the clarification. Then I can't use those instructions, which is what I
>> was starting to suspect. I have to find another way of getting DPDK running in
>> the guest then. Using ixgbevf in the guest works fine, though.
>>
>> Regards
>> Mats
>>
>
> Hi Mats
>
> Let me clarify - do you run DPDK on both host and guest? E.g. you load igb_uio on the host as well? If not, I don't see why you can't use those instructions - you're creating VF devices on the host anyway, and that's where you should set their MAC addresses.
>
> Best regards,
> Anatoly Burakov
> DPDK SW Engineer
>
> --------------------------------------------------------------
> Intel Shannon Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare
>
>

Re: How to debug packet sends to virtual functions

[Burakov, Anatoly]

> -----Original Message-----
> From: Mats Liljegren [mailto:liljegren.mats2@gmail.com]
> Sent: Tuesday, February 04, 2014 1:04 PM
> To: Burakov, Anatoly
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] How to debug packet sends to virtual functions
> 
> Hi Anatoly,
> 
> Just realized that the instructions gave two alternatives for the
> host: DPDK igb_uio or ixgbe. It was the former, DPDK igb_uio that I was
> referring to as being the non-working alternative.
> 
> Loading ixgbe enables me to set the mac addresses. This is also what I tried
> previously, when I both got ixgbevf working in the guest as well as receive to
> work in DPDK in the guest. The problem is making transmit work in DPDK in
> the guest.
> 
> Regards
> Mats
> 
> 
Hi Mats

Technically, you can use igb_uio on the host as well (DPDK PMD supports creating virtual devices since at least release 1.5.0), it's just that you'll have to set everything up yourself inside your host DPDK application (you can't use "ip net" to set up VF devices if you're using DPDK drivers). Unfortunately, I'm not familiar enough with that part of the code to comment on what exactly you should do to make it work with igb_uio, but I can certainly ask around if you want.

I also noticed that you are running KVM with virsh. We always run our VM's by passing QEMU command line directly, without using virsh, so unfortunately I cannot be of much help here as I'm not familiar with virsh. Your best bet would be to get whatever parameters virsh passes to QEMU and modify them to suit your needs and according to DPDK documentation.

Best regards,
Anatoly Burakov
DPDK SW Engineer

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

On Tue, Feb 4, 2014 at 2:40 PM, Burakov, Anatoly
<anatoly.burakov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> Hi Mats
>
> Technically, you can use igb_uio on the host as well (DPDK PMD supports creating virtual devices since at least release 1.5.0), it's just that you'll have to set everything up yourself inside your host DPDK application (you can't use "ip net" to set up VF devices if you're using DPDK drivers). Unfortunately, I'm not familiar enough with that part of the code to comment on what exactly you should do to make it work with igb_uio, but I can certainly ask around if you want.
>
> I also noticed that you are running KVM with virsh. We always run our VM's by passing QEMU command line directly, without using virsh, so unfortunately I cannot be of much help here as I'm not familiar with virsh. Your best bet would be to get whatever parameters virsh passes to QEMU and modify them to suit your needs and according to DPDK documentation.

Hi Anatoly,

I finally got things working. I apparently missed an "ifconfig <dev>
up" in the guest, before starting dpdk. I'm still confused why this
would be needed. Is dpdk unable to do a full initialization of the
virtual function from the guest?


I'm also curious whether this is how it is intended to work, or am I
experiencing a strange type of bug? If not, is it documented that
ifconfig is needed in the guest? I thought I've read all documentation
thoroughly, but sometimes it is easy to miss the obvious anyway...

While I got one instance of DPDK running, I got a problem when
starting two instances of DPDK running against different virtual
functions. These virtual functions stems from the same physical
interfaces.

Starting them one or the other works fine. I have to adapt my static
arp entries since they have different MAC addresses, but this is only
expected. When starting them both however, I receive no packets. If
one is running and currently processing packets, it will stop doing so
the instance the second instance starts.

Anyone knowing what could cause such behavior?

Regards
Mats Liljegren

Re: How to debug packet sends to virtual functions

[Burakov, Anatoly]

Hi Mats

> Hi Anatoly,
> 
> I finally got things working. I apparently missed an "ifconfig <dev> up" in the
> guest, before starting dpdk. I'm still confused why this would be needed. Is
> dpdk unable to do a full initialization of the virtual function from the guest?
> 
You wouldn't be able to call ifconfig on your VF device if you have bound your guest VF device to igb_uio driver. Have you bound your VF device to igb_uio? Or you have enabled automatic port unbinding (which is disabled by default in recent releases)?

> While I got one instance of DPDK running, I got a problem when starting two
> instances of DPDK running against different virtual functions. These virtual
> functions stems from the same physical interfaces.
> 
> Starting them one or the other works fine. I have to adapt my static arp
> entries since they have different MAC addresses, but this is only expected.
> When starting them both however, I receive no packets. If one is running
> and currently processing packets, it will stop doing so the instance the second
> instance starts.
> 
Are you referring to two different VM's each having a separate VF device, or are you trying to run two primary processes on one VM with different VF devices?

Best regards,
Anatoly Burakov
DPDK SW Engineer

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

>> I finally got things working. I apparently missed an "ifconfig <dev> up" in the
>> guest, before starting dpdk. I'm still confused why this would be needed. Is
>> dpdk unable to do a full initialization of the virtual function from the guest?
>>
> You wouldn't be able to call ifconfig on your VF device if you have bound your guest VF device to igb_uio driver. Have you bound your VF device to igb_uio? Or you have enabled automatic port unbinding (which is disabled by default in recent releases)?

The guest starts with loading the igbvf kernel driver, uses ifconfig
<dev> up, then loads and binds to igb_uio. After that, DPDK works. If
I skip any step here it doesn't work. Skipping "ifconfig" step
resulted in packets being received but I couldn't send them, they just
got queued up but was never sent.

>> While I got one instance of DPDK running, I got a problem when starting two
>> instances of DPDK running against different virtual functions. These virtual
>> functions stems from the same physical interfaces.
>>
>> Starting them one or the other works fine. I have to adapt my static arp
>> entries since they have different MAC addresses, but this is only expected.
>> When starting them both however, I receive no packets. If one is running
>> and currently processing packets, it will stop doing so the instance the second
>> instance starts.
>>
> Are you referring to two different VM's each having a separate VF device, or are you trying to run two primary processes on one VM with different VF devices?

It's the latter, i.e. two primary processes running DPDK on the same
VM but different VF devices. The VF devices belongs to the same
physical devices though.

Regards
Mats Liljegren

Re: How to debug packet sends to virtual functions

[Burakov, Anatoly]

Hi Mats

> The guest starts with loading the igbvf kernel driver, uses ifconfig <dev> up,
> then loads and binds to igb_uio. After that, DPDK works. If I skip any step
> here it doesn't work. Skipping "ifconfig" step resulted in packets being
> received but I couldn't send them, they just got queued up but was never
> sent.
> 

Hm, never seen this before. Apologies if you have already mentioned this, but what NIC are you using? I'll see if I can replicate your issues.

> It's the latter, i.e. two primary processes running DPDK on the same VM but
> different VF devices. The VF devices belongs to the same physical devices
> though.

I'll try that as well.

Best regards,
Anatoly Burakov
DPDK SW Engineer

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

Hi Anatoly,

>> The guest starts with loading the igbvf kernel driver, uses ifconfig <dev> up,
>> then loads and binds to igb_uio. After that, DPDK works. If I skip any step
>> here it doesn't work. Skipping "ifconfig" step resulted in packets being
>> received but I couldn't send them, they just got queued up but was never
>> sent.
>>
>
> Hm, never seen this before. Apologies if you have already mentioned this, but what NIC are you using? I'll see if I can replicate your issues.

Not sure if I've mentioned it, but it is a quad I350. Port 0 and 3 on
host only, and 1 and 2 are being used by DPDK in a guest. The
challenge was to make a ping go from port 0 to 3 and back, via a
somewhat modified DPDK l2fwd example. Port 1 and 2 had two virtual
functions each, and each such pair was given to an instance of this
l2fwd example.

>> It's the latter, i.e. two primary processes running DPDK on the same VM but
>> different VF devices. The VF devices belongs to the same physical devices
>> though.
>
> I'll try that as well.

If you need more help with how we set things up, I'll be happy to help you.

Regards
Mats Liljegren

Re: How to debug packet sends to virtual functions

[Mats Liljegren]

>>> It's the latter, i.e. two primary processes running DPDK on the same VM but
>>> different VF devices. The VF devices belongs to the same physical devices
>>> though.
>>
>> I'll try that as well.
>
> If you need more help with how we set things up, I'll be happy to help you.

I found the bug, it was an illegal arp entry that I had created
causing the problem. So this case has been solved.

I re-checked this ifconfig issue, and now I can get it to work without
ifconfig in the guest. Not sure why it didn't work without it
previously, but apparently, there is something else I did that made
things work.

Sorry about the fuzz, but thanks for your effort!

Regards
Mats Liljegren

Re: How to debug packet sends to virtual functions

[Burakov, Anatoly]

> I found the bug, it was an illegal arp entry that I had created causing the
> problem. So this case has been solved.
> 
> I re-checked this ifconfig issue, and now I can get it to work without ifconfig
> in the guest. Not sure why it didn't work without it previously, but
> apparently, there is something else I did that made things work.
> 
> Sorry about the fuzz, but thanks for your effort!

No problem, glad to hear it's not a DPDK bug :-)

Best regards,
Anatoly Burakov
DPDK SW Engineer

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare