seg fault in InterVM communication using virtio

seg fault in InterVM communication using virtio

[Priyanka]

Hi All,

We have 2 VM running on qemu-kvm hypervisor. We have provided vNIC to 
the VMs using macvtap device and virtio as the device model in bridge 
mode. We are running DPDK applications (udp client and server) on the 
VMs. We observe that the inter-VM communication is not happening using 
drivers i.e igb_uio and uio_pci_generic. We were able to communicate 
between the VMs properly using SRIOV. But we were not getting 
performance as expected so we switched to virtio. But using virtio the 
DPDK communication hangs after sending 128 packets. We also observe a 
seg fault in the dmesg of the VM.

Please provide us some guideline to do the inter-VM communication using 
virtio. Are we correct if we are using rte_eth_tx/rx_burst to send and 
receive packets from the vNIC.

Thanks,

Priyanka

Re: seg fault in InterVM communication using virtio

[Yuanhan Liu]

On Mon, Feb 20, 2017 at 04:00:12PM +0530, Priyanka wrote:
> Hi All,
> 
> We have 2 VM running on qemu-kvm hypervisor. We have provided vNIC to the
> VMs using macvtap device and virtio as the device model in bridge mode. We
> are running DPDK applications (udp client and server) on the VMs. We observe
> that the inter-VM communication is not happening using drivers i.e igb_uio
> and uio_pci_generic. We were able to communicate between the VMs properly
> using SRIOV. But we were not getting performance as expected so we switched
> to virtio. But using virtio the DPDK communication hangs after sending 128
> packets. We also observe a seg fault in the dmesg of the VM.
> 
> Please provide us some guideline to do the inter-VM communication using
> virtio. Are we correct if we are using rte_eth_tx/rx_burst to send and
> receive packets from the vNIC.

Would you provide more info, say how do you start the vm, and how do you
run the dpdk apps? Providing all the commands that could reproduce your
issue will be better.

	--yliu

Re: seg fault in InterVM communication using virtio

[mitali]

[-- Attachment #1: Type: text/plain, Size: 1779 bytes --]

Hi Yuanhan,

We are using virt-manager to start the VMs. We are attaching the image 
which shows the vNIC setup for the VMs.

We have written our own dpdk application to send and receive UDP 
packets. The command we are using is

*./build/app/dpdkapi -c 1 -- -p 1* where dpdkapi is our application.

Inside the application we are sending and receiving packets from vNIC 
using rte_eth_tx_burst() and rte_eth_rx_burst().

At the sender side after 127 packets the application stops and we 
observe a seg fault in dmesg. Using the print statements we observed 
that it faults in rte_eth_tx_burst(). All the 127 were received at the 
receiving VM.

Thanks,

Mitali


On Wednesday 22 February 2017 02:13 PM, Yuanhan Liu wrote:
> On Mon, Feb 20, 2017 at 04:00:12PM +0530, Priyanka wrote:
>> Hi All,
>>
>> We have 2 VM running on qemu-kvm hypervisor. We have provided vNIC to the
>> VMs using macvtap device and virtio as the device model in bridge mode. We
>> are running DPDK applications (udp client and server) on the VMs. We observe
>> that the inter-VM communication is not happening using drivers i.e igb_uio
>> and uio_pci_generic. We were able to communicate between the VMs properly
>> using SRIOV. But we were not getting performance as expected so we switched
>> to virtio. But using virtio the DPDK communication hangs after sending 128
>> packets. We also observe a seg fault in the dmesg of the VM.
>>
>> Please provide us some guideline to do the inter-VM communication using
>> virtio. Are we correct if we are using rte_eth_tx/rx_burst to send and
>> receive packets from the vNIC.
> Would you provide more info, say how do you start the vm, and how do you
> run the dpdk apps? Providing all the commands that could reproduce your
> issue will be better.
>
> 	--yliu


[-- Attachment #2: VM_virtio.png --]
[-- Type: image/png, Size: 149226 bytes --]

Re: seg fault in InterVM communication using virtio

[Yuanhan Liu]

On Wed, Feb 22, 2017 at 02:25:29PM +0530, mitali wrote:
> Hi Yuanhan,
> 
> We are using virt-manager to start the VMs. We are attaching the image which
> shows the vNIC setup for the VMs.
> 
> We have written our own dpdk application to send and receive UDP packets. The
> command we are using is
> 
> ./build/app/dpdkapi -c 1 -- -p 1 where dpdkapi is our application.
> 
> Inside the application we are sending and receiving packets from vNIC using
> rte_eth_tx_burst() and rte_eth_rx_burst().
> 
> At the sender side after 127 packets the application stops and we observe a seg
> fault in dmesg. Using the print statements we observed that it faults in
> rte_eth_tx_burst(). All the 127 were received at the receiving VM.

Would you recompile dpdk with debug info enaled, which could be done by:

	make ... EXTRA_CFLAGS='-g3 -O0'

And run it with gdb then dump the call stack?

	--yliu

Re: seg fault in InterVM communication using virtio

[mitali]

Hi,

We have some doubt regarding virtio front end. As we are running dpdk 
application inside VM with virtio setup, do we need to use virtio 
specific functions like virtio_recv_pkts() and virtio_xmit_pkts() to 
receive and transmit packets, functions for port initialization like 
virtio_dev_queue_setup()? If yes, we are unable to find any application 
which uses these functions. Can you provide us some pointers on examples 
which uses these functions.

Thanks,
Mitali

On Thursday 23 February 2017 11:57 AM, Yuanhan Liu wrote:
> On Wed, Feb 22, 2017 at 02:25:29PM +0530, mitali wrote:
>> Hi Yuanhan,
>>
>> We are using virt-manager to start the VMs. We are attaching the image which
>> shows the vNIC setup for the VMs.
>>
>> We have written our own dpdk application to send and receive UDP packets. The
>> command we are using is
>>
>> ./build/app/dpdkapi -c 1 -- -p 1 where dpdkapi is our application.
>>
>> Inside the application we are sending and receiving packets from vNIC using
>> rte_eth_tx_burst() and rte_eth_rx_burst().
>>
>> At the sender side after 127 packets the application stops and we observe a seg
>> fault in dmesg. Using the print statements we observed that it faults in
>> rte_eth_tx_burst(). All the 127 were received at the receiving VM.
> Would you recompile dpdk with debug info enaled, which could be done by:
>
> 	make ... EXTRA_CFLAGS='-g3 -O0'
>
> And run it with gdb then dump the call stack?
>
> 	--yliu

Re: seg fault in InterVM communication using virtio

[ppnaik]

Hi,

We have some doubt regarding virtio front end. As we are running dpdk 
application inside VM with virtio setup, do we need to use virtio 
specific functions like virtio_recv_pkts() and virtio_xmit_pkts() to 
receive and transmit packets, functions for port initialization like 
virtio_dev_queue_setup()? If yes, we are unable to find any application 
which uses these functions. Can you provide us some pointers on examples 
which uses these functions.

Thanks,
Priyanka


On 2017-02-23 11:57, Yuanhan Liu wrote:
> On Wed, Feb 22, 2017 at 02:25:29PM +0530, mitali wrote:
>> Hi Yuanhan,
>>
>> We are using virt-manager to start the VMs. We are attaching the 
>> image which
>> shows the vNIC setup for the VMs.
>>
>> We have written our own dpdk application to send and receive UDP 
>> packets. The
>> command we are using is
>>
>> ./build/app/dpdkapi -c 1 -- -p 1 where dpdkapi is our application.
>>
>> Inside the application we are sending and receiving packets from 
>> vNIC using
>> rte_eth_tx_burst() and rte_eth_rx_burst().
>>
>> At the sender side after 127 packets the application stops and we 
>> observe a seg
>> fault in dmesg. Using the print statements we observed that it 
>> faults in
>> rte_eth_tx_burst(). All the 127 were received at the receiving VM.
>
> Would you recompile dpdk with debug info enaled, which could be done 
> by:
>
> 	make ... EXTRA_CFLAGS='-g3 -O0'
>
> And run it with gdb then dump the call stack?
>
> 	--yliu

Re: seg fault in InterVM communication using virtio

[Yuanhan Liu]

On Wed, Mar 01, 2017 at 07:20:37PM +0530, mitali wrote:
> Hi,
> 
> We have some doubt regarding virtio front end. As we are running dpdk
> application inside VM with virtio setup, do we need to use virtio specific
> functions like virtio_recv_pkts() and virtio_xmit_pkts() to receive and
> transmit packets, functions for port initialization like
> virtio_dev_queue_setup()? If yes, we are unable to find any application
> which uses these functions. Can you provide us some pointers on examples
> which uses these functions.

If you are running DPDK inside guest (with virtio), then the virtio is
yet another PMD, meaning you should program it with the normal DPDK
ethdev APIs, something like rte_eth_dev_configure, rte_eth_rx/tx_queue_setup,
etc.

	--yliu
> 
> Thanks,
> Mitali
> 
> On Thursday 23 February 2017 11:57 AM, Yuanhan Liu wrote:
> >On Wed, Feb 22, 2017 at 02:25:29PM +0530, mitali wrote:
> >>Hi Yuanhan,
> >>
> >>We are using virt-manager to start the VMs. We are attaching the image which
> >>shows the vNIC setup for the VMs.
> >>
> >>We have written our own dpdk application to send and receive UDP packets. The
> >>command we are using is
> >>
> >>./build/app/dpdkapi -c 1 -- -p 1 where dpdkapi is our application.
> >>
> >>Inside the application we are sending and receiving packets from vNIC using
> >>rte_eth_tx_burst() and rte_eth_rx_burst().
> >>
> >>At the sender side after 127 packets the application stops and we observe a seg
> >>fault in dmesg. Using the print statements we observed that it faults in
> >>rte_eth_tx_burst(). All the 127 were received at the receiving VM.
> >Would you recompile dpdk with debug info enaled, which could be done by:
> >
> >	make ... EXTRA_CFLAGS='-g3 -O0'
> >
> >And run it with gdb then dump the call stack?
> >
> >	--yliu

Re: seg fault in InterVM communication using virtio

[mitali]

Hi,

Ok. But with other PMDs like igb, when we are providing sriov vf to VM 
and running our client/server dpdk application then there is no issue in 
communication between VMs.
As soon as we use virtio setup for VMs and running same dpdk 
application, it stops after sending 127 packet and seg fault comes while 
executing rte_eth_tx_burst(). We are still not clear why with virtio its 
creating this issue but not with any other PMD.

Mitali

On Thursday 02 March 2017 05:53 PM, Yuanhan Liu wrote:
> On Wed, Mar 01, 2017 at 07:20:37PM +0530, mitali wrote:
>> Hi,
>>
>> We have some doubt regarding virtio front end. As we are running dpdk
>> application inside VM with virtio setup, do we need to use virtio specific
>> functions like virtio_recv_pkts() and virtio_xmit_pkts() to receive and
>> transmit packets, functions for port initialization like
>> virtio_dev_queue_setup()? If yes, we are unable to find any application
>> which uses these functions. Can you provide us some pointers on examples
>> which uses these functions.
> If you are running DPDK inside guest (with virtio), then the virtio is
> yet another PMD, meaning you should program it with the normal DPDK
> ethdev APIs, something like rte_eth_dev_configure, rte_eth_rx/tx_queue_setup,
> etc.
>
> 	--yliu
>> Thanks,
>> Mitali
>>
>> On Thursday 23 February 2017 11:57 AM, Yuanhan Liu wrote:
>>> On Wed, Feb 22, 2017 at 02:25:29PM +0530, mitali wrote:
>>>> Hi Yuanhan,
>>>>
>>>> We are using virt-manager to start the VMs. We are attaching the image which
>>>> shows the vNIC setup for the VMs.
>>>>
>>>> We have written our own dpdk application to send and receive UDP packets. The
>>>> command we are using is
>>>>
>>>> ./build/app/dpdkapi -c 1 -- -p 1 where dpdkapi is our application.
>>>>
>>>> Inside the application we are sending and receiving packets from vNIC using
>>>> rte_eth_tx_burst() and rte_eth_rx_burst().
>>>>
>>>> At the sender side after 127 packets the application stops and we observe a seg
>>>> fault in dmesg. Using the print statements we observed that it faults in
>>>> rte_eth_tx_burst(). All the 127 were received at the receiving VM.
>>> Would you recompile dpdk with debug info enaled, which could be done by:
>>>
>>> 	make ... EXTRA_CFLAGS='-g3 -O0'
>>>
>>> And run it with gdb then dump the call stack?
>>>
>>> 	--yliu

Re: seg fault in InterVM communication using virtio

[Yuanhan Liu]

On Thu, Mar 02, 2017 at 06:02:28PM +0530, mitali wrote:
> Hi,
> 
> Ok. But with other PMDs like igb, when we are providing sriov vf to VM and
> running our client/server dpdk application then there is no issue in
> communication between VMs.
> As soon as we use virtio setup for VMs and running same dpdk application, it
> stops after sending 127 packet and seg fault comes while executing
> rte_eth_tx_burst(). We are still not clear why with virtio its creating this
> issue but not with any other PMD.

No idea. You don't provide any further details. Please, would you just
enable the debug info, rebuild, run it with gdb and then dump the call
stack after crash?

	--yliu
> 
> Mitali
> 
> On Thursday 02 March 2017 05:53 PM, Yuanhan Liu wrote:
> >On Wed, Mar 01, 2017 at 07:20:37PM +0530, mitali wrote:
> >>Hi,
> >>
> >>We have some doubt regarding virtio front end. As we are running dpdk
> >>application inside VM with virtio setup, do we need to use virtio specific
> >>functions like virtio_recv_pkts() and virtio_xmit_pkts() to receive and
> >>transmit packets, functions for port initialization like
> >>virtio_dev_queue_setup()? If yes, we are unable to find any application
> >>which uses these functions. Can you provide us some pointers on examples
> >>which uses these functions.
> >If you are running DPDK inside guest (with virtio), then the virtio is
> >yet another PMD, meaning you should program it with the normal DPDK
> >ethdev APIs, something like rte_eth_dev_configure, rte_eth_rx/tx_queue_setup,
> >etc.
> >
> >	--yliu
> >>Thanks,
> >>Mitali
> >>
> >>On Thursday 23 February 2017 11:57 AM, Yuanhan Liu wrote:
> >>>On Wed, Feb 22, 2017 at 02:25:29PM +0530, mitali wrote:
> >>>>Hi Yuanhan,
> >>>>
> >>>>We are using virt-manager to start the VMs. We are attaching the image which
> >>>>shows the vNIC setup for the VMs.
> >>>>
> >>>>We have written our own dpdk application to send and receive UDP packets. The
> >>>>command we are using is
> >>>>
> >>>>./build/app/dpdkapi -c 1 -- -p 1 where dpdkapi is our application.
> >>>>
> >>>>Inside the application we are sending and receiving packets from vNIC using
> >>>>rte_eth_tx_burst() and rte_eth_rx_burst().
> >>>>
> >>>>At the sender side after 127 packets the application stops and we observe a seg
> >>>>fault in dmesg. Using the print statements we observed that it faults in
> >>>>rte_eth_tx_burst(). All the 127 were received at the receiving VM.
> >>>Would you recompile dpdk with debug info enaled, which could be done by:
> >>>
> >>>	make ... EXTRA_CFLAGS='-g3 -O0'
> >>>
> >>>And run it with gdb then dump the call stack?
> >>>
> >>>	--yliu

Re: seg fault in InterVM communication using virtio

[mitali]

[-- Attachment #1: Type: text/plain, Size: 2899 bytes --]

Hi,

I rebuild dpdk with debug enabled and running the same dpdk application.

Please go through the attached images showing the error as well as call 
stack.

Mitali

On Thursday 02 March 2017 06:29 PM, Yuanhan Liu wrote:
> On Thu, Mar 02, 2017 at 06:02:28PM +0530, mitali wrote:
>> Hi,
>>
>> Ok. But with other PMDs like igb, when we are providing sriov vf to VM and
>> running our client/server dpdk application then there is no issue in
>> communication between VMs.
>> As soon as we use virtio setup for VMs and running same dpdk application, it
>> stops after sending 127 packet and seg fault comes while executing
>> rte_eth_tx_burst(). We are still not clear why with virtio its creating this
>> issue but not with any other PMD.
> No idea. You don't provide any further details. Please, would you just
> enable the debug info, rebuild, run it with gdb and then dump the call
> stack after crash?
>
> 	--yliu
>> Mitali
>>
>> On Thursday 02 March 2017 05:53 PM, Yuanhan Liu wrote:
>>> On Wed, Mar 01, 2017 at 07:20:37PM +0530, mitali wrote:
>>>> Hi,
>>>>
>>>> We have some doubt regarding virtio front end. As we are running dpdk
>>>> application inside VM with virtio setup, do we need to use virtio specific
>>>> functions like virtio_recv_pkts() and virtio_xmit_pkts() to receive and
>>>> transmit packets, functions for port initialization like
>>>> virtio_dev_queue_setup()? If yes, we are unable to find any application
>>>> which uses these functions. Can you provide us some pointers on examples
>>>> which uses these functions.
>>> If you are running DPDK inside guest (with virtio), then the virtio is
>>> yet another PMD, meaning you should program it with the normal DPDK
>>> ethdev APIs, something like rte_eth_dev_configure, rte_eth_rx/tx_queue_setup,
>>> etc.
>>>
>>> 	--yliu
>>>> Thanks,
>>>> Mitali
>>>>
>>>> On Thursday 23 February 2017 11:57 AM, Yuanhan Liu wrote:
>>>>> On Wed, Feb 22, 2017 at 02:25:29PM +0530, mitali wrote:
>>>>>> Hi Yuanhan,
>>>>>>
>>>>>> We are using virt-manager to start the VMs. We are attaching the image which
>>>>>> shows the vNIC setup for the VMs.
>>>>>>
>>>>>> We have written our own dpdk application to send and receive UDP packets. The
>>>>>> command we are using is
>>>>>>
>>>>>> ./build/app/dpdkapi -c 1 -- -p 1 where dpdkapi is our application.
>>>>>>
>>>>>> Inside the application we are sending and receiving packets from vNIC using
>>>>>> rte_eth_tx_burst() and rte_eth_rx_burst().
>>>>>>
>>>>>> At the sender side after 127 packets the application stops and we observe a seg
>>>>>> fault in dmesg. Using the print statements we observed that it faults in
>>>>>> rte_eth_tx_burst(). All the 127 were received at the receiving VM.
>>>>> Would you recompile dpdk with debug info enaled, which could be done by:
>>>>>
>>>>> 	make ... EXTRA_CFLAGS='-g3 -O0'
>>>>>
>>>>> And run it with gdb then dump the call stack?
>>>>>
>>>>> 	--yliu


[-- Attachment #2: gdb_error_call_stack.png --]
[-- Type: image/png, Size: 213966 bytes --]

[-- Attachment #3: vm_gdb_error.png --]
[-- Type: image/png, Size: 304846 bytes --]