KVM ARM Archive on lore.kernel.org
 help / color / Atom feed
From: Zenghui Yu <yuzenghui@huawei.com>
To: Marc Zyngier <marc.zyngier@arm.com>,
	<linux-arm-kernel@lists.infradead.org>,
	<kvmarm@lists.cs.columbia.edu>, <kvm@vger.kernel.org>
Cc: "Raslan, KarimAllah" <karahmed@amazon.de>,
	"Saidi, Ali" <alisaidi@amazon.com>
Subject: Re: [PATCH v2 7/9] KVM: arm/arm64: vgic-its: Cache successful MSI->LPI translation
Date: Wed, 26 Jun 2019 00:00:54 +0800
Message-ID: <7af32ebf-91a8-ef63-6108-4ca506fd364e@huawei.com> (raw)
In-Reply-To: <169cc847-ebfa-44b6-00e7-c69dccdbbd62@arm.com>

Hi Marc,

On 2019/6/25 20:31, Marc Zyngier wrote:
> Hi Zenghui,
> 
> On 25/06/2019 12:50, Zenghui Yu wrote:
>> Hi Marc,
>>
>> On 2019/6/12 1:03, Marc Zyngier wrote:
>>> On a successful translation, preserve the parameters in the LPI
>>> translation cache. Each translation is reusing the last slot
>>> in the list, naturally evincting the least recently used entry.
>>>
>>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>>> ---
>>>    virt/kvm/arm/vgic/vgic-its.c | 86 ++++++++++++++++++++++++++++++++++++
>>>    1 file changed, 86 insertions(+)
>>>
>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>> index 0aa0cbbc3af6..62932458476a 100644
>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>> @@ -546,6 +546,90 @@ static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,
>>>    	return 0;
>>>    }
>>>    
>>> +static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,
>>> +					       phys_addr_t db,
>>> +					       u32 devid, u32 eventid)
>>> +{
>>> +	struct vgic_translation_cache_entry *cte;
>>> +	struct vgic_irq *irq = NULL;
>>> +
>>> +	list_for_each_entry(cte, &dist->lpi_translation_cache, entry) {
>>> +		/*
>>> +		 * If we hit a NULL entry, there is nothing after this
>>> +		 * point.
>>> +		 */
>>> +		if (!cte->irq)
>>> +			break;
>>> +
>>> +		if (cte->db == db &&
>>> +		    cte->devid == devid &&
>>> +		    cte->eventid == eventid) {
>>> +			/*
>>> +			 * Move this entry to the head, as it is the
>>> +			 * most recently used.
>>> +			 */
>>> +			list_move(&cte->entry, &dist->lpi_translation_cache);
>>
>> Only for performance reasons: if we hit at the "head" of the list, we
>> don't need to do a list_move().
>> In our tests, we found that a single list_move() takes nearly (sometimes
>> even more than) one microsecond, for some unknown reason...
> 
> Huh... That's odd.
> 
> Can you narrow down under which conditions this happens? I'm not sure if
> checking for the list head would be more efficient, as you end-up
> fetching the head anyway. Can you try replacing this line with:
> 
> 	if (!list_is_first(&cte->entry, &dist->lpi_translation_cache))
> 		list_move(&cte->entry, &dist->lpi_translation_cache);
> 
> and let me know whether it helps?

It helps. With this change, the overhead of list_move() is gone.

We run 16 4-vcpu VMs on the host, each with a vhost-user nic, and run
"iperf" in pairs between them.  It's likely to hit at the head of the
cache list in our tests.
With this change, the sys% utilization of vhostdpfwd threads will
decrease by about 10%.  But I don't know the reason exactly (I haven't
found any clues in code yet, in implementation of list_move...).


Thanks,
zenghui



_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

  reply index

Thread overview: 40+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-11 17:03 [PATCH v2 0/9] KVM: arm/arm64: vgic: ITS translation cache Marc Zyngier
2019-06-11 17:03 ` [PATCH v2 1/9] KVM: arm/arm64: vgic: Add LPI translation cache definition Marc Zyngier
2019-06-12  8:16   ` Julien Thierry
2019-06-12  8:49     ` Julien Thierry
2019-06-12  9:52     ` Marc Zyngier
2019-06-12 10:58       ` Julien Thierry
2019-06-12 12:28         ` Julien Thierry
2019-07-23 12:43   ` Auger Eric
2019-06-11 17:03 ` [PATCH v2 2/9] KVM: arm/arm64: vgic: Add __vgic_put_lpi_locked primitive Marc Zyngier
2019-06-11 17:03 ` [PATCH v2 3/9] KVM: arm/arm64: vgic-its: Add MSI-LPI translation cache invalidation Marc Zyngier
2019-07-23 12:39   ` Auger Eric
2019-06-11 17:03 ` [PATCH v2 4/9] KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on specific commands Marc Zyngier
2019-07-01 12:38   ` Auger Eric
2019-07-22 10:54     ` Marc Zyngier
2019-07-23 12:25       ` Auger Eric
2019-07-23 12:43         ` Marc Zyngier
2019-07-23 12:47           ` Auger Eric
2019-07-23 12:50             ` Marc Zyngier
2019-06-11 17:03 ` [PATCH v2 5/9] KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on disabling LPIs Marc Zyngier
2019-07-23 15:09   ` Auger Eric
2019-06-11 17:03 ` [PATCH v2 6/9] KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on vgic teardown Marc Zyngier
2019-07-23 15:10   ` Auger Eric
2019-06-11 17:03 ` [PATCH v2 7/9] KVM: arm/arm64: vgic-its: Cache successful MSI->LPI translation Marc Zyngier
2019-06-25 11:50   ` Zenghui Yu
2019-06-25 12:31     ` Marc Zyngier
2019-06-25 16:00       ` Zenghui Yu [this message]
2019-06-26  3:54         ` Zenghui Yu
2019-06-26  7:55         ` Marc Zyngier
2019-07-23 15:21   ` Auger Eric
2019-06-11 17:03 ` [PATCH v2 8/9] KVM: arm/arm64: vgic-its: Check the LPI translation cache on MSI injection Marc Zyngier
2019-07-23 15:10   ` Auger Eric
2019-07-23 15:45     ` Marc Zyngier
2019-07-24  7:41       ` Auger Eric
2019-06-11 17:03 ` [PATCH v2 9/9] KVM: arm/arm64: vgic-irqfd: Implement kvm_arch_set_irq_inatomic Marc Zyngier
2019-07-23 15:14   ` Auger Eric
2019-07-25  8:24     ` Marc Zyngier
2019-07-23 11:14 ` [PATCH v2 0/9] KVM: arm/arm64: vgic: ITS translation cache Andre Przywara
2019-07-25  8:50   ` Marc Zyngier
2019-07-25 10:01     ` Andre Przywara
2019-07-25 15:37       ` Marc Zyngier

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=7af32ebf-91a8-ef63-6108-4ca506fd364e@huawei.com \
    --to=yuzenghui@huawei.com \
    --cc=alisaidi@amazon.com \
    --cc=karahmed@amazon.de \
    --cc=kvm@vger.kernel.org \
    --cc=kvmarm@lists.cs.columbia.edu \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=marc.zyngier@arm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

KVM ARM Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/kvmarm/0 kvmarm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 kvmarm kvmarm/ https://lore.kernel.org/kvmarm \
		kvmarm@lists.cs.columbia.edu kvmarm@archiver.kernel.org
	public-inbox-index kvmarm


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/edu.columbia.cs.lists.kvmarm


AGPL code for this site: git clone https://public-inbox.org/ public-inbox