From mboxrd@z Thu Jan 1 00:00:00 1970 From: jigsaw Subject: Re: How to debug packet sends to virtual functions Date: Tue, 4 Feb 2014 14:11:19 +0200 Message-ID: References:

Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Cc: "dev-VfR2kkLFssw@public.gmane.org" To: Mats Liljegren Return-path: In-Reply-To: List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces-VfR2kkLFssw@public.gmane.org Sender: "dev" Hi Mats, Yes the virsh xml is created in essentially the same way. And the setup in host is also the same as yours. But no PCI passthru. As I understand, in a PF-VF deployment it makes no sense for PCI passthru, coz the PCI device must be visible and under control of host. (Pls correct me if Im wrong). -Qinglai On Tue, Feb 4, 2014 at 1:31 PM, Mats Liljegren wrote: > Hi Qinglai, > > How did you configure the PCI passthrough in KVM? Did you use command > line parameter as described in the instructions, or do you use virsh > XML files, or maybe using virt-manager? > > The steps that works best (i.e. can receive but not transmit) are: > > 1) sudo modprobe ixgbe max_vfs=2 > 2) sudo ip link set em1 vf 0 mac > 3) sudo ifconfig em1 up > 4) sudo modprobe pci-stub > 5) echo "8086 1515" | sudo tee /sys/bus/pci/drivers/pci-stub/new_id > 6) echo "0000:02:10.0" | sudo tee > /sys/bus/pci/devices/0000\:02\:10.0/driver/unbind > 7) echo "0000:02:10.0" | sudo tee /sys/bus/pci/drivers/pci-stub/bind > 8) virsh create vm vm.xml > > The guest then does: > > 5) sudo modprobe uio > 6) sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio 0000:02:10.0 > 7) sudo $RTE_SDK/tools/pci_unbind.py -b igb_uio > 8) sudo > > The virsh xml file has the following entry for the virtual function: > > >

> >

function='0x0'/> > > > I actually have two virtual functions and two physical functions, but > I just repeat above for each. > > Regards > Mats > > On Tue, Feb 4, 2014 at 12:14 PM, jigsaw wrote: >> Hi Mats, >> >> I didn't have any deviation. What I did is just loading ixgbe (with >> extra params for vf, as you mentioned in your first email), >> and DPDK is up and running in guest. Of course I also followed section >> 8.10 of the DPDK release notes. >> >> I can switch between DPDK and ixgebvf in guest at runtime and >> everything works fine. >> >> Sorry I can't help to debug coz I have only 82599EB at hand. >> >> -Qinglai >> >> On Tue, Feb 4, 2014 at 12:45 PM, Mats Liljegren >> wrote: >>> Hi Qinglai, >>> >>> Thanks for the response! >>> >>> My previous attempt was with ixgbe loaded in the host. I also needed >>> to load ixgbevf, but this seems to be because of a short-coming in >>> libvirt. Maybe loading ixgbevf and then unbind it when running the >>> guest is what causes my problems. I could receive packets with this >>> setup, but not transmit. I used extra debug to syslog, and it showed >>> that packets was placed in the transmit queue, but these packets was >>> never sent. >>> >>> I'll see if I can get this working without loading ixgbevf in the host. >>> >>> What instructions did you follow to get this working? Did you do any >>> deviation from the instructions? >>> >>> Regards >>> Mats >>> >>> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw wrote: >>>> Hi Mats, >>>> >>>> I've tried vf with 82599EB and it works fine. As long as the VF is >>>> visible in guest, DPDK's VF driver should work just as ixgbevf, which >>>> shares more or less the same code. >>>> >>>> I don't understand why you would expect DPDK@guest to work as VF, >>>> while the host has no ixgbe loaded. >>>> To make further debug I'd suggest compile ixgbe driver with your own >>>> syslogs. At least you will be able to see the signalling between vf >>>> and ixgbe drivers. >>>> >>>> -Qinglai >>>> >>>> >>>> On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren >>>> wrote: >>>>> This is my fourth mail in my desperate attempt to get DPDK running in >>>>> KVM and no comments so far, not even any "it works for me". Am I the >>>>> only one crazy enough to believe that this can be done? >>>>> >>>>> Anyway, out of desperation I tried to get it running without having >>>>> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed >>>>> the instructions in the Programmer's Guide, chapter "Setting Up a KVM >>>>> Virtual Machine Monitor", using the PMD version of the instructions. >>>>> >>>>> I get as far as being able to see my four virtual functions in the >>>>> guest using "lspci". But starting the DPDK application gives me the >>>>> following error: >>>>> >>>>> PMD: The MAC address is not valid. >>>>> The most likely cause of this error is that the VM host >>>>> has not assigned a valid MAC address to this VF device. >>>>> Please consult the DPDK Release Notes (FAQ section) for >>>>> a possible solution to this problem. >>>>> >>>>> This may be true, but without any kernel modules loaded, how am I >>>>> supposed to change any MAC addresses? Can this be done from within >>>>> DPDK? >>>>> >>>>> As a side-note, I did try to load ixgbevf in the guest, but it >>>>> produced no interfaces. There was no error messages in the syslog >>>>> though. >>>>> >>>>> Is it possible to get X540 working in a guest or should I switch hardware? >>>>> >>>>> Since the instructions assumes I know the command line to KVM to start >>>>> my guest (which I do not), I cannot followed them precisely. I use >>>>> virsh and XML file, and maybe I've misunderstood how to translate the >>>>> pci-assign parameter to XML code. I currently use a entry, >>>>> but I've also tried . Neither has been >>>>> working for me so far, though the version got me as far as >>>>> being able to receive packets at least, but not transmitting. >>>>> >>>>> Regards >>>>> Mats >>>>> >>>>> >>>>> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren >>>>> wrote: >>>>>> Never mind, I was hit by the infamous MAC spoofing... I got it working >>>>>> on both the host and the guest using ixgbevf driver, so apparently the >>>>>> cables are correctly attached. >>>>>> >>>>>> Using DPDK is still no-go. It can receive packets, but when sending >>>>>> the packets the function returns success, but the driver reports >>>>>> nothing (i.e. no errors, no sent packets, no nothing, except for >>>>>> received packets of course). >>>>>> >>>>>> What could cause this behavior? >>>>>> >>>>>> Regards >>>>>> Mats >>>>>> >>>>>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren >>>>>> wrote: >>>>>>> I have a follow-up on this: >>>>>>> >>>>>>> ixgbe version 3.13.10-k >>>>>>> ixgbevf version 2.7.12-k >>>>>>> >>>>>>> (These are what was provided by Ubuntu 13.10) >>>>>>> >>>>>>> I tried the following sequence on the host, before starting the guest: >>>>>>> 1) sudo rmmod ixgbe >>>>>>> 2) sudo modprobe ixgbe max_vfs=2 >>>>>>> 3) sudo ifconfig em1 up # This is the physical function >>>>>>> 4) sudo ifconfig em1_0 192.168.2.2 # This is the virtual function >>>>>>> 5) ping 192.168.2.1 >>>>>>> >>>>>>> I can see that the ping request reaches its target, and a reply is >>>>>>> sent back. But this reply is not received by the ping shell command. >>>>>>> >>>>>>> Why? >>>>>>> >>>>>>> Regards, >>>>>>> Mats >>>>>>> >>>>>>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren >>>>>>> wrote: >>>>>>>> I'm trying to get a modified version of the l2fwd example running, and >>>>>>>> have problems with packets being silently thrown away. I can receive >>>>>>>> packets, and my printf's indicates that the packets are being sent to >>>>>>>> the correct port, using correct MAC address as source address. And >>>>>>>> still, the packets are lost. >>>>>>>> >>>>>>>> Since the port is a virtual function, it seems like I cannot use >>>>>>>> tcpdump on it to see the network traffic. There is nothing coming out >>>>>>>> of the cable (activity light not flashing, the receiving end running >>>>>>>> tcpdump does not see any traffic). >>>>>>>> >>>>>>>> I'm using two X540 with two virtual functions each. The application >>>>>>>> runs in a KVM/Qemu environmen. >>>>>>>> >>>>>>>> Any suggestions how to debug this? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mats