From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1760FC48BD4 for ; Tue, 25 Jun 2019 16:02:02 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id A0457208CB for ; Tue, 25 Jun 2019 16:02:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A0457208CB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0F9064A4FB; Tue, 25 Jun 2019 12:02:01 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bv9OOMHOIqF5; Tue, 25 Jun 2019 12:02:00 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 01E034A4A3; Tue, 25 Jun 2019 12:02:00 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id BD3B84A49F for ; Tue, 25 Jun 2019 12:01:58 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4YKXLaheb6d2 for ; Tue, 25 Jun 2019 12:01:57 -0400 (EDT) Received: from huawei.com (szxga07-in.huawei.com [45.249.212.35]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 066124A32E for ; Tue, 25 Jun 2019 12:01:57 -0400 (EDT) Received: from DGGEMS409-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 5CF3AEA899F81CF85AC9; Wed, 26 Jun 2019 00:01:53 +0800 (CST) Received: from [127.0.0.1] (10.184.12.158) by DGGEMS409-HUB.china.huawei.com (10.3.19.209) with Microsoft SMTP Server id 14.3.439.0; Wed, 26 Jun 2019 00:01:45 +0800 Subject: Re: [PATCH v2 7/9] KVM: arm/arm64: vgic-its: Cache successful MSI->LPI translation To: Marc Zyngier , , , References: <20190611170336.121706-1-marc.zyngier@arm.com> <20190611170336.121706-8-marc.zyngier@arm.com> <53de88e9-3550-bd7f-8266-35c5e75fae4e@huawei.com> <169cc847-ebfa-44b6-00e7-c69dccdbbd62@arm.com> From: Zenghui Yu Message-ID: <7af32ebf-91a8-ef63-6108-4ca506fd364e@huawei.com> Date: Wed, 26 Jun 2019 00:00:54 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:64.0) Gecko/20100101 Thunderbird/64.0 MIME-Version: 1.0 In-Reply-To: <169cc847-ebfa-44b6-00e7-c69dccdbbd62@arm.com> Content-Language: en-US X-Originating-IP: [10.184.12.158] X-CFilter-Loop: Reflected Cc: "Raslan, KarimAllah" , "Saidi, Ali" X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Hi Marc, On 2019/6/25 20:31, Marc Zyngier wrote: > Hi Zenghui, > > On 25/06/2019 12:50, Zenghui Yu wrote: >> Hi Marc, >> >> On 2019/6/12 1:03, Marc Zyngier wrote: >>> On a successful translation, preserve the parameters in the LPI >>> translation cache. Each translation is reusing the last slot >>> in the list, naturally evincting the least recently used entry. >>> >>> Signed-off-by: Marc Zyngier >>> --- >>> virt/kvm/arm/vgic/vgic-its.c | 86 ++++++++++++++++++++++++++++++++++++ >>> 1 file changed, 86 insertions(+) >>> >>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c >>> index 0aa0cbbc3af6..62932458476a 100644 >>> --- a/virt/kvm/arm/vgic/vgic-its.c >>> +++ b/virt/kvm/arm/vgic/vgic-its.c >>> @@ -546,6 +546,90 @@ static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm, >>> return 0; >>> } >>> >>> +static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist, >>> + phys_addr_t db, >>> + u32 devid, u32 eventid) >>> +{ >>> + struct vgic_translation_cache_entry *cte; >>> + struct vgic_irq *irq = NULL; >>> + >>> + list_for_each_entry(cte, &dist->lpi_translation_cache, entry) { >>> + /* >>> + * If we hit a NULL entry, there is nothing after this >>> + * point. >>> + */ >>> + if (!cte->irq) >>> + break; >>> + >>> + if (cte->db == db && >>> + cte->devid == devid && >>> + cte->eventid == eventid) { >>> + /* >>> + * Move this entry to the head, as it is the >>> + * most recently used. >>> + */ >>> + list_move(&cte->entry, &dist->lpi_translation_cache); >> >> Only for performance reasons: if we hit at the "head" of the list, we >> don't need to do a list_move(). >> In our tests, we found that a single list_move() takes nearly (sometimes >> even more than) one microsecond, for some unknown reason... > > Huh... That's odd. > > Can you narrow down under which conditions this happens? I'm not sure if > checking for the list head would be more efficient, as you end-up > fetching the head anyway. Can you try replacing this line with: > > if (!list_is_first(&cte->entry, &dist->lpi_translation_cache)) > list_move(&cte->entry, &dist->lpi_translation_cache); > > and let me know whether it helps? It helps. With this change, the overhead of list_move() is gone. We run 16 4-vcpu VMs on the host, each with a vhost-user nic, and run "iperf" in pairs between them. It's likely to hit at the head of the cache list in our tests. With this change, the sys% utilization of vhostdpfwd threads will decrease by about 10%. But I don't know the reason exactly (I haven't found any clues in code yet, in implementation of list_move...). Thanks, zenghui _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm