IVSHMEM device performance

IVSHMEM device performance

[Eli Britstein]

Hi

In a VM, I add a IVSHMEM device, on which the MBUFS mempool resides, and also rings I create (I run a DPDK application in the VM).
I saw there is a performance penalty if I use such device, instead of hugepages (the VM's hugepages). My VM's memory is *NOT* backed with host's hugepages.
The memory behind the IVSHMEM device is a host hugepage (I use a patched version of QEMU, as provided by Intel).
I thought maybe the reason is that this memory is seen by the VM as a mapped PCI memory region, so it is not cached, but I am not sure.
So, my direction was to change the kernel (in the VM) so it will consider this memory as a regular memory (and thus cached), instead of a PCI memory region.
However, I am not sure my direction is correct, and even if so, I am not sure how/where to change the kernel (my starting point was  mm/mmap.c, but I'm not sure it's the correct place to start).

Any suggestion is welcomed.
Thanks,
Eli.


-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

Re: IVSHMEM device performance

[Markus Armbruster]

Cc: qemu-devel

Eli Britstein <eli.britstein@toganetworks.com> writes:

> Hi
>
> In a VM, I add a IVSHMEM device, on which the MBUFS mempool resides, and also rings I create (I run a DPDK application in the VM).
> I saw there is a performance penalty if I use such device, instead of hugepages (the VM's hugepages). My VM's memory is *NOT* backed with host's hugepages.
> The memory behind the IVSHMEM device is a host hugepage (I use a patched version of QEMU, as provided by Intel).
> I thought maybe the reason is that this memory is seen by the VM as a mapped PCI memory region, so it is not cached, but I am not sure.
> So, my direction was to change the kernel (in the VM) so it will consider this memory as a regular memory (and thus cached), instead of a PCI memory region.
> However, I am not sure my direction is correct, and even if so, I am not sure how/where to change the kernel (my starting point was  mm/mmap.c, but I'm not sure it's the correct place to start).
>
> Any suggestion is welcomed.
> Thanks,
> Eli.

Re: [Qemu-devel] IVSHMEM device performance

[Markus Armbruster]

Cc: qemu-devel

Eli Britstein <eli.britstein@toganetworks.com> writes:

> Hi
>
> In a VM, I add a IVSHMEM device, on which the MBUFS mempool resides, and also rings I create (I run a DPDK application in the VM).
> I saw there is a performance penalty if I use such device, instead of hugepages (the VM's hugepages). My VM's memory is *NOT* backed with host's hugepages.
> The memory behind the IVSHMEM device is a host hugepage (I use a patched version of QEMU, as provided by Intel).
> I thought maybe the reason is that this memory is seen by the VM as a mapped PCI memory region, so it is not cached, but I am not sure.
> So, my direction was to change the kernel (in the VM) so it will consider this memory as a regular memory (and thus cached), instead of a PCI memory region.
> However, I am not sure my direction is correct, and even if so, I am not sure how/where to change the kernel (my starting point was  mm/mmap.c, but I'm not sure it's the correct place to start).
>
> Any suggestion is welcomed.
> Thanks,
> Eli.

Re: IVSHMEM device performance

[Michael S. Tsirkin]

On Mon, Apr 11, 2016 at 10:56:54AM +0200, Markus Armbruster wrote:
> Cc: qemu-devel
> 
> Eli Britstein <eli.britstein@toganetworks.com> writes:
> 
> > Hi
> >
> > In a VM, I add a IVSHMEM device, on which the MBUFS mempool resides, and also rings I create (I run a DPDK application in the VM).
> > I saw there is a performance penalty if I use such device, instead of hugepages (the VM's hugepages). My VM's memory is *NOT* backed with host's hugepages.
> > The memory behind the IVSHMEM device is a host hugepage (I use a patched version of QEMU, as provided by Intel).
> > I thought maybe the reason is that this memory is seen by the VM as a mapped PCI memory region, so it is not cached, but I am not sure.
> > So, my direction was to change the kernel (in the VM) so it will consider this memory as a regular memory (and thus cached), instead of a PCI memory region.
> > However, I am not sure my direction is correct, and even if so, I am not sure how/where to change the kernel (my starting point was  mm/mmap.c, but I'm not sure it's the correct place to start).
> >
> > Any suggestion is welcomed.
> > Thanks,
> > Eli.

A cleaner way is just to use virtio, keeping everything in VM's
memory, with access either by data copies in hypervisor, or
directly using vhost-user.
For example, with vhost-pci: https://wiki.opnfv.org/vm2vm_mst
there has been recent work on this, see slides 12-14 in
http://schd.ws/hosted_files/ons2016/36/Nakajima_and_Ergin_PreSwitch_final.pdf

This is very much work in progress, but if you are interested
you should probably get in touch with Nakajima et al.

-- 
MST

Re: [Qemu-devel] IVSHMEM device performance

[Michael S. Tsirkin]

On Mon, Apr 11, 2016 at 10:56:54AM +0200, Markus Armbruster wrote:
> Cc: qemu-devel
> 
> Eli Britstein <eli.britstein@toganetworks.com> writes:
> 
> > Hi
> >
> > In a VM, I add a IVSHMEM device, on which the MBUFS mempool resides, and also rings I create (I run a DPDK application in the VM).
> > I saw there is a performance penalty if I use such device, instead of hugepages (the VM's hugepages). My VM's memory is *NOT* backed with host's hugepages.
> > The memory behind the IVSHMEM device is a host hugepage (I use a patched version of QEMU, as provided by Intel).
> > I thought maybe the reason is that this memory is seen by the VM as a mapped PCI memory region, so it is not cached, but I am not sure.
> > So, my direction was to change the kernel (in the VM) so it will consider this memory as a regular memory (and thus cached), instead of a PCI memory region.
> > However, I am not sure my direction is correct, and even if so, I am not sure how/where to change the kernel (my starting point was  mm/mmap.c, but I'm not sure it's the correct place to start).
> >
> > Any suggestion is welcomed.
> > Thanks,
> > Eli.

A cleaner way is just to use virtio, keeping everything in VM's
memory, with access either by data copies in hypervisor, or
directly using vhost-user.
For example, with vhost-pci: https://wiki.opnfv.org/vm2vm_mst
there has been recent work on this, see slides 12-14 in
http://schd.ws/hosted_files/ons2016/36/Nakajima_and_Ergin_PreSwitch_final.pdf

This is very much work in progress, but if you are interested
you should probably get in touch with Nakajima et al.

-- 
MST

RE: IVSHMEM device performance

[Eli Britstein]

> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst@redhat.com]
> Sent: Monday, 11 April, 2016 3:28 PM
> To: Markus Armbruster
> Cc: Eli Britstein; qemu-devel@nongnu.org; kvm@vger.kernel.org
> Subject: Re: IVSHMEM device performance
>
> On Mon, Apr 11, 2016 at 10:56:54AM +0200, Markus Armbruster wrote:
> > Cc: qemu-devel
> >
> > Eli Britstein <eli.britstein@toganetworks.com> writes:
> >
> > > Hi
> > >
> > > In a VM, I add a IVSHMEM device, on which the MBUFS mempool
> resides, and also rings I create (I run a DPDK application in the VM).
> > > I saw there is a performance penalty if I use such device, instead of
> hugepages (the VM's hugepages). My VM's memory is *NOT* backed with
> host's hugepages.
> > > The memory behind the IVSHMEM device is a host hugepage (I use a
> patched version of QEMU, as provided by Intel).
> > > I thought maybe the reason is that this memory is seen by the VM as a
> mapped PCI memory region, so it is not cached, but I am not sure.
> > > So, my direction was to change the kernel (in the VM) so it will consider
> this memory as a regular memory (and thus cached), instead of a PCI
> memory region.
> > > However, I am not sure my direction is correct, and even if so, I am not
> sure how/where to change the kernel (my starting point was  mm/mmap.c,
> but I'm not sure it's the correct place to start).
> > >
> > > Any suggestion is welcomed.
> > > Thanks,
> > > Eli.
>
> A cleaner way is just to use virtio, keeping everything in VM's memory, with
> access either by data copies in hypervisor, or directly using vhost-user.
> For example, with vhost-pci: https://wiki.opnfv.org/vm2vm_mst there has
> been recent work on this, see slides 12-14 in
> http://schd.ws/hosted_files/ons2016/36/Nakajima_and_Ergin_PreSwitch_fi
> nal.pdf
>
> This is very much work in progress, but if you are interested you should
> probably get in touch with Nakajima et al.
[Eli Britstein] This is indeed very interesting and I will further look into it.
However, if I'm not wrong, this requires some support from the host which I would like to avoid.
My requirement from the host is only to provide an IVSHMEM device for several VMs, and my applications are running on VMs only. So, I think that vhost-pci is not applicable in my case. Am I wrong?
Can you think of a reason why accessing that PCI memory mapped memory (which is really the host's hugepage) is more expensive than accessing the VM's hugepages (even though they are not really the host's hugepage)?
Do you think my suspicion that as a PCI mapped memory it doesn't use cache is correct? If so, do you think I can change it (either in some configuration or changing the VM's kernel)?
Any other direction?

Thanks, Eli
>
> --
> MST
-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

Re: [Qemu-devel] IVSHMEM device performance

[Eli Britstein]

> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst@redhat.com]
> Sent: Monday, 11 April, 2016 3:28 PM
> To: Markus Armbruster
> Cc: Eli Britstein; qemu-devel@nongnu.org; kvm@vger.kernel.org
> Subject: Re: IVSHMEM device performance
>
> On Mon, Apr 11, 2016 at 10:56:54AM +0200, Markus Armbruster wrote:
> > Cc: qemu-devel
> >
> > Eli Britstein <eli.britstein@toganetworks.com> writes:
> >
> > > Hi
> > >
> > > In a VM, I add a IVSHMEM device, on which the MBUFS mempool
> resides, and also rings I create (I run a DPDK application in the VM).
> > > I saw there is a performance penalty if I use such device, instead of
> hugepages (the VM's hugepages). My VM's memory is *NOT* backed with
> host's hugepages.
> > > The memory behind the IVSHMEM device is a host hugepage (I use a
> patched version of QEMU, as provided by Intel).
> > > I thought maybe the reason is that this memory is seen by the VM as a
> mapped PCI memory region, so it is not cached, but I am not sure.
> > > So, my direction was to change the kernel (in the VM) so it will consider
> this memory as a regular memory (and thus cached), instead of a PCI
> memory region.
> > > However, I am not sure my direction is correct, and even if so, I am not
> sure how/where to change the kernel (my starting point was  mm/mmap.c,
> but I'm not sure it's the correct place to start).
> > >
> > > Any suggestion is welcomed.
> > > Thanks,
> > > Eli.
>
> A cleaner way is just to use virtio, keeping everything in VM's memory, with
> access either by data copies in hypervisor, or directly using vhost-user.
> For example, with vhost-pci: https://wiki.opnfv.org/vm2vm_mst there has
> been recent work on this, see slides 12-14 in
> http://schd.ws/hosted_files/ons2016/36/Nakajima_and_Ergin_PreSwitch_fi
> nal.pdf
>
> This is very much work in progress, but if you are interested you should
> probably get in touch with Nakajima et al.
[Eli Britstein] This is indeed very interesting and I will further look into it.
However, if I'm not wrong, this requires some support from the host which I would like to avoid.
My requirement from the host is only to provide an IVSHMEM device for several VMs, and my applications are running on VMs only. So, I think that vhost-pci is not applicable in my case. Am I wrong?
Can you think of a reason why accessing that PCI memory mapped memory (which is really the host's hugepage) is more expensive than accessing the VM's hugepages (even though they are not really the host's hugepage)?
Do you think my suspicion that as a PCI mapped memory it doesn't use cache is correct? If so, do you think I can change it (either in some configuration or changing the VM's kernel)?
Any other direction?

Thanks, Eli
>
> --
> MST
-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

Re: [Qemu-devel] IVSHMEM device performance

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

On 04/11/2016 07:18 AM, Eli Britstein wrote:
> 

[meta-comment]


> Thanks, Eli
>>
>> --
>> MST
> -------------------------------------------------------------------------------------------------------------------------------------------------
> This email and any files transmitted and/or attachments with it are confidential and proprietary information of

Disclaimers like this are inappropriate on publicly-archived lists, and
unenforceable.  If your employer insists on spamming us with legalese,
it might be better for you to send mail from a personal account.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

RE: IVSHMEM device performance

[Wang, Wei W]

> -----Original Message-----
> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On
> Behalf Of Eli Britstein
> Sent: Monday, April 11, 2016 2:21 PM
> To: kvm@vger.kernel.org
> Subject: IVSHMEM device performance
> 
> Hi
> 
> In a VM, I add a IVSHMEM device, on which the MBUFS mempool resides, and
> also rings I create (I run a DPDK application in the VM).
> I saw there is a performance penalty if I use such device, instead of hugepages
> (the VM's hugepages). My VM's memory is *NOT* backed with host's hugepages.
> The memory behind the IVSHMEM device is a host hugepage (I use a patched
> version of QEMU, as provided by Intel).
> I thought maybe the reason is that this memory is seen by the VM as a mapped
> PCI memory region, so it is not cached, but I am not sure.
> So, my direction was to change the kernel (in the VM) so it will consider this
> memory as a regular memory (and thus cached), instead of a PCI memory region.
> However, I am not sure my direction is correct, and even if so, I am not sure
> how/where to change the kernel (my starting point was  mm/mmap.c, but I'm
> not sure it's the correct place to start).
> 
> Any suggestion is welcomed.
> Thanks,
> Eli.

Hi Eli,

I think you can try the following two things:
1. use ioremap_cache() to map the PCI bar, instead of ioremap();
2. touch the entire bar memory before using it.

Best,
Wei

Re: IVSHMEM device performance

[Paolo Bonzini]

On 11/04/2016 08:21, Eli Britstein wrote:
> I thought maybe the reason is that this memory is seen by the VM as a
> mapped PCI memory region, so it is not cached,

KVM actually overrides the type of the memory region and always makes it
cached.

Can you try sending the "lspci -vv" output?

Paolo

RE: IVSHMEM device performance

[Eli Britstein]

[-- Attachment #1: Type: text/plain, Size: 1831 bytes --]

Attached.
Also, if need to change ioremap to ioremap_cache, please advise where. I assume in mmap it is already too late.

> -----Original Message-----
> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo
> Bonzini
> Sent: Thursday, 14 April, 2016 3:45 PM
> To: Eli Britstein; kvm@vger.kernel.org
> Subject: Re: IVSHMEM device performance
>
>
>
> On 11/04/2016 08:21, Eli Britstein wrote:
> > I thought maybe the reason is that this memory is seen by the VM as a
> > mapped PCI memory region, so it is not cached,
>
> KVM actually overrides the type of the memory region and always makes it
> cached.
>
> Can you try sending the "lspci -vv" output?
>
> Paolo
-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------


[-- Attachment #2: lspci_vv_ivshmem.txt --]
[-- Type: text/plain, Size: 578 bytes --]

00:04.0 RAM memory: Red Hat, Inc Inter-VM shared memory
        Subsystem: Red Hat, Inc QEMU Virtual Machine
        Physical Slot: 4
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 11
        Region 0: Memory at febf2000 (32-bit, non-prefetchable) [size=256]
        Region 2: Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Kernel modules: virtio_pci

Re: IVSHMEM device performance

[Paolo Bonzini]

On 17/04/2016 09:18, Eli Britstein wrote:
> Attached. Also, if need to change ioremap to ioremap_cache, please
> advise where. I assume in mmap it is already too late.

No, you need not do that.  If memory was uncached, the performance
penalty would be 100x or worse.

Paolo

RE: IVSHMEM device performance

[Eli Britstein]

OK, but if so, my suspicious about the performance penalty is wrong, and I don't have any other thought.
Do you?

> -----Original Message-----
> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> Sent: Sunday, 17 April, 2016 3:04 PM
> To: Eli Britstein; kvm@vger.kernel.org
> Subject: Re: IVSHMEM device performance
>
>
>
> On 17/04/2016 09:18, Eli Britstein wrote:
> > Attached. Also, if need to change ioremap to ioremap_cache, please
> > advise where. I assume in mmap it is already too late.
>
> No, you need not do that.  If memory was uncached, the performance
> penalty would be 100x or worse.
>
> Paolo
-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

RE: IVSHMEM device performance

[Eli Britstein]

[-- Attachment #1: Type: text/plain, Size: 3539 bytes --]

Does anyone have a direction?

Thanks, Eli

> -----Original Message-----
> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org]
> On Behalf Of Eli Britstein
> Sent: Sunday, 17 April, 2016 6:58 PM
> To: Paolo Bonzini; kvm@vger.kernel.org
> Subject: RE: IVSHMEM device performance
>
> OK, but if so, my suspicious about the performance penalty is wrong, and I
> don't have any other thought.
> Do you?
>
> > -----Original Message-----
> > From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> > Sent: Sunday, 17 April, 2016 3:04 PM
> > To: Eli Britstein; kvm@vger.kernel.org
> > Subject: Re: IVSHMEM device performance
> >
> >
> >
> > On 17/04/2016 09:18, Eli Britstein wrote:
> > > Attached. Also, if need to change ioremap to ioremap_cache, please
> > > advise where. I assume in mmap it is already too late.
> >
> > No, you need not do that.  If memory was uncached, the performance
> > penalty would be 100x or worse.
> >
> > Paolo
[Eli Britstein]

> -----Original Message-----
> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On
> Behalf Of Eli Britstein
> Sent: Monday, April 11, 2016 2:21 PM
> To: kvm@vger.kernel.org
> Subject: IVSHMEM device performance
>
> Hi
>
> In a VM, I add a IVSHMEM device, on which the MBUFS mempool resides,
> and also rings I create (I run a DPDK application in the VM).
> I saw there is a performance penalty if I use such device, instead of
> hugepages (the VM's hugepages). My VM's memory is *NOT* backed with host's hugepages.
> The memory behind the IVSHMEM device is a host hugepage (I use a
> patched version of QEMU, as provided by Intel).
> I thought maybe the reason is that this memory is seen by the VM as a
> mapped PCI memory region, so it is not cached, but I am not sure.
> So, my direction was to change the kernel (in the VM) so it will
> consider this memory as a regular memory (and thus cached), instead of a PCI memory region.
> However, I am not sure my direction is correct, and even if so, I am
> not sure how/where to change the kernel (my starting point was
> mm/mmap.c, but I'm not sure it's the correct place to start).
>
> Any suggestion is welcomed.
> Thanks,
> Eli.

Hi Eli,

I think you can try the following two things:
1. use ioremap_cache() to map the PCI bar, instead of ioremap(); 2. touch the entire bar memory before using it.

Best,
Wei
-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------


[-- Attachment #2: lspci_vv_ivshmem.txt --]
[-- Type: text/plain, Size: 578 bytes --]

00:04.0 RAM memory: Red Hat, Inc Inter-VM shared memory
        Subsystem: Red Hat, Inc QEMU Virtual Machine
        Physical Slot: 4
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 11
        Region 0: Memory at febf2000 (32-bit, non-prefetchable) [size=256]
        Region 2: Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Kernel modules: virtio_pci

RE: IVSHMEM device performance

[Wang, Wei W]

On Thu 5/5/2016 4:57 PM, Eli Britstein wrote:
> Does anyone have a direction?

Have you tried to touch all the BAR area before using it? 
I doubt that your performance penalty comes from page faults.

Best,
Wei 

> 
> > -----Original Message-----
> > From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org]
> > On Behalf Of Eli Britstein
> > Sent: Sunday, 17 April, 2016 6:58 PM
> > To: Paolo Bonzini; kvm@vger.kernel.org
> > Subject: RE: IVSHMEM device performance
> >
> > OK, but if so, my suspicious about the performance penalty is wrong,
> > and I don't have any other thought.
> > Do you?
> >
> > > -----Original Message-----
> > > From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> > > Sent: Sunday, 17 April, 2016 3:04 PM
> > > To: Eli Britstein; kvm@vger.kernel.org
> > > Subject: Re: IVSHMEM device performance
> > >
> > >
> > >
> > > On 17/04/2016 09:18, Eli Britstein wrote:
> > > > Attached. Also, if need to change ioremap to ioremap_cache, please
> > > > advise where. I assume in mmap it is already too late.
> > >
> > > No, you need not do that.  If memory was uncached, the performance
> > > penalty would be 100x or worse.
> > >
> > > Paolo
> [Eli Britstein]
> 
> > -----Original Message-----
> > From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On
> > Behalf Of Eli Britstein
> > Sent: Monday, April 11, 2016 2:21 PM
> > To: kvm@vger.kernel.org
> > Subject: IVSHMEM device performance
> >
> > Hi
> >
> > In a VM, I add a IVSHMEM device, on which the MBUFS mempool resides,
> > and also rings I create (I run a DPDK application in the VM).
> > I saw there is a performance penalty if I use such device, instead of
> > hugepages (the VM's hugepages). My VM's memory is *NOT* backed with
> host's hugepages.
> > The memory behind the IVSHMEM device is a host hugepage (I use a
> > patched version of QEMU, as provided by Intel).
> > I thought maybe the reason is that this memory is seen by the VM as a
> > mapped PCI memory region, so it is not cached, but I am not sure.
> > So, my direction was to change the kernel (in the VM) so it will
> > consider this memory as a regular memory (and thus cached), instead of a PCI
> memory region.
> > However, I am not sure my direction is correct, and even if so, I am
> > not sure how/where to change the kernel (my starting point was
> > mm/mmap.c, but I'm not sure it's the correct place to start).
> >
> > Any suggestion is welcomed.
> > Thanks,
> > Eli.
> 
> Hi Eli,
> 
> I think you can try the following two things:
> 1. use ioremap_cache() to map the PCI bar, instead of ioremap(); 2. touch the
> entire bar memory before using it.
> 
> Best,
> Wei
> --------------------------------------------------------------------------------------------------
> -----------------------------------------------
> This email and any files transmitted and/or attachments with it are confidential
> and proprietary information of Toga Networks Ltd., and intended solely for the
> use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the system manager. This
> message contains confidential information of Toga Networks Ltd., and is
> intended only for the individual named. If you are not the named addressee you
> should not disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and delete this
> e-mail from your system. If you are not the intended recipient you are notified
> that disclosing, copying, distributing or taking any action in reliance on the
> contents of this information is strictly prohibited.
> --------------------------------------------------------------------------------------------------
> ----------------------------------------------

RE: IVSHMEM device performance

[Eli Britstein]

The penalty I see is not transitional, but consistent, so it cannot be something of this sort.
My suspicious is that the fact that this region cannot be mmapped with MAP_HUGETLB flag makes some difference, but I really don't know.

Thanks, Eli

> -----Original Message-----
> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org]
> On Behalf Of Wang, Wei W
> Sent: Friday, 06 May, 2016 3:50 AM
> To: Eli Britstein; kvm@vger.kernel.org
> Subject: RE: IVSHMEM device performance
>
> On Thu 5/5/2016 4:57 PM, Eli Britstein wrote:
> > Does anyone have a direction?
>
> Have you tried to touch all the BAR area before using it?
> I doubt that your performance penalty comes from page faults.
>
> Best,
> Wei
>
> >
> > > -----Original Message-----
> > > From: kvm-owner@vger.kernel.org [mailto:kvm-
> owner@vger.kernel.org]
> > > On Behalf Of Eli Britstein
> > > Sent: Sunday, 17 April, 2016 6:58 PM
> > > To: Paolo Bonzini; kvm@vger.kernel.org
> > > Subject: RE: IVSHMEM device performance
> > >
> > > OK, but if so, my suspicious about the performance penalty is wrong,
> > > and I don't have any other thought.
> > > Do you?
> > >
> > > > -----Original Message-----
> > > > From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> > > > Sent: Sunday, 17 April, 2016 3:04 PM
> > > > To: Eli Britstein; kvm@vger.kernel.org
> > > > Subject: Re: IVSHMEM device performance
> > > >
> > > >
> > > >
> > > > On 17/04/2016 09:18, Eli Britstein wrote:
> > > > > Attached. Also, if need to change ioremap to ioremap_cache,
> > > > > please advise where. I assume in mmap it is already too late.
> > > >
> > > > No, you need not do that.  If memory was uncached, the performance
> > > > penalty would be 100x or worse.
> > > >
> > > > Paolo
> > [Eli Britstein]
> >
> > > -----Original Message-----
> > > From: kvm-owner@vger.kernel.org [mailto:kvm-
> owner@vger.kernel.org]
> > > On Behalf Of Eli Britstein
> > > Sent: Monday, April 11, 2016 2:21 PM
> > > To: kvm@vger.kernel.org
> > > Subject: IVSHMEM device performance
> > >
> > > Hi
> > >
> > > In a VM, I add a IVSHMEM device, on which the MBUFS mempool
> resides,
> > > and also rings I create (I run a DPDK application in the VM).
> > > I saw there is a performance penalty if I use such device, instead
> > > of hugepages (the VM's hugepages). My VM's memory is *NOT* backed
> > > with
> > host's hugepages.
> > > The memory behind the IVSHMEM device is a host hugepage (I use a
> > > patched version of QEMU, as provided by Intel).
> > > I thought maybe the reason is that this memory is seen by the VM as
> > > a mapped PCI memory region, so it is not cached, but I am not sure.
> > > So, my direction was to change the kernel (in the VM) so it will
> > > consider this memory as a regular memory (and thus cached), instead
> > > of a PCI
> > memory region.
> > > However, I am not sure my direction is correct, and even if so, I am
> > > not sure how/where to change the kernel (my starting point was
> > > mm/mmap.c, but I'm not sure it's the correct place to start).
> > >
> > > Any suggestion is welcomed.
> > > Thanks,
> > > Eli.
> >
> > Hi Eli,
> >
> > I think you can try the following two things:
> > 1. use ioremap_cache() to map the PCI bar, instead of ioremap(); 2.
> > touch the entire bar memory before using it.
> >
> > Best,
> > Wei
-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------