From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70FE7C4320A for ; Wed, 18 Aug 2021 23:56:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4ADDD60F11 for ; Wed, 18 Aug 2021 23:56:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235103AbhHRX46 (ORCPT ); Wed, 18 Aug 2021 19:56:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234949AbhHRX44 (ORCPT ); Wed, 18 Aug 2021 19:56:56 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62FDFC061764 for ; Wed, 18 Aug 2021 16:56:21 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id z23-20020a17090abd97b0290176898bbb9cso2147488pjr.8 for ; Wed, 18 Aug 2021 16:56:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:message-id:mime-version:subject:from:to:cc; bh=Scvb2gfJzrYfNJq90EL74fT7SS7nVGstZhceM9SRTyY=; b=CPqsJcRdPyzWVw2UkNu0Wn9hUr9Qxq5bXEX1GOFSOmGUzn4ZXDSKec71mwcN+FHo9/ J7Hl7xy+UKc/xgVNUN88urCR/U61zjMcY6asIwsJ1NN2YS9qGf+aKvKCDIrLpWvwMxY7 2BngKUPmIzEBdYGZPYHGwKJ9jNq1gVsHj9z0RLCoU1r/NmEOi3G9vIVtbVYCcORZAjlI UndfdDGE3KQaTr65pLvCdgbH6llmSIJz96OFnUlXCMFadPkBkt7tK63U5VkNx63TbLCo ERRq3MYzS7Mvlz1HG8GN5rM0km+KLmdMUSZjv6uehqHwlltzlCfbOOiB7zBMlniQMnlZ qqtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:message-id:mime-version:subject :from:to:cc; bh=Scvb2gfJzrYfNJq90EL74fT7SS7nVGstZhceM9SRTyY=; b=PKc/jwTK10VI9xfGG6T8z8lTmBDyAqrxSaWm1XRj1JeSAhxxSEukRWz8iheHEZGn6I XE7qZGGytpZ/31DNWF/gStHnm3NTegxo08Gcu9I+SHQyI/FHPsDO13RvJpHiEx4DV1/Z zHBEUnXauZuaiSkO5PcghlcZkB+YVl1X32841VCzaEqjof2bLsVVAhm92RJ5jsVa8Xi4 ZsvYFzYEyAJbNhnp15QRkAKwTlfACjf9zTuxKRJ9QznyRrsrc4SOvEH9UfB0gNPyk1vM SBNm5fsmIOYutfzpLZ/duqumIsCP1kkf48aU1cZ8kQAUFACcx8kBJqiz1OBcGXwS7jJ/ Jsww== X-Gm-Message-State: AOAM5334hnHVKQJZuOEHx8Hyr/oln7Pqmkd//OVTc3GuYE9VbcAol8tm Y19KGygi5gBAZxL8Zi0ur/SKWVfxFjE= X-Google-Smtp-Source: ABdhPJyqChTXz2F80g0qJy/+xySrxm2DGETkwfuo5JqfDwHZAekTYYLYUCbmnHGiPShsiTlZr2Nz/k/ji+0= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:e9c6:b029:12d:4cb3:3985 with SMTP id 6-20020a170902e9c6b029012d4cb33985mr9312260plk.56.1629330980601; Wed, 18 Aug 2021 16:56:20 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 18 Aug 2021 23:56:15 +0000 Message-Id: <20210818235615.2047588-1-seanjc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.rc1.237.g0d66db33f3-goog Subject: [PATCH] KVM: x86/mmu: Complete prefetch for trailing SPTEs for direct, legacy MMU From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Ben Gardon Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Make a final call to direct_pte_prefetch_many() if there are "trailing" SPTEs to prefetch, i.e. SPTEs for GFNs following the faulting GFN. The call to direct_pte_prefetch_many() in the loop only handles the case where there are !PRESENT SPTEs preceding a PRESENT SPTE. E.g. if the faulting GFN is a multiple of 8 (the prefetch size) and all SPTEs for the following GFNs are !PRESENT, the loop will terminate with "start = sptep+1" and not prefetch any SPTEs. Prefetching trailing SPTEs as intended can drastically reduce the number of guest page faults, e.g. accessing the first byte of every 4kb page in a 6gb chunk of virtual memory, in a VM with 8gb of preallocated memory, the number of pf_fixed events observed in L0 drops from ~1.75M to <0.27M. Note, this only affects memory that is backed by 4kb pages as KVM doesn't prefetch when installing hugepages. Shadow paging prefetching is not affected as it does not batch the prefetches due to the need to process the corresponding guest PTE. The TDP MMU is not affected because it doesn't have prefetching, yet... Fixes: 957ed9effd80 ("KVM: MMU: prefetch ptes when intercepted guest #PF") Cc: Sergey Senozhatsky Cc: Ben Gardon Signed-off-by: Sean Christopherson --- Cc'd Ben as this highlights a potential gap with the TDP MMU, which lacks prefetching of any sort. For large VMs, which are likely backed by hugepages anyways, this is a non-issue as the benefits of holding mmu_lock for read likely masks the cost of taking more VM-Exits. But VMs with a small number of vCPUs won't benefit as much from parallel page faults, e.g. there's no benefit at all if there's a single vCPU. arch/x86/kvm/mmu/mmu.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a272ccbddfa1..daf7df35f788 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2818,11 +2818,13 @@ static void __direct_pte_prefetch(struct kvm_vcpu *vcpu, if (!start) continue; if (direct_pte_prefetch_many(vcpu, sp, start, spte) < 0) - break; + return; start = NULL; } else if (!start) start = spte; } + if (start) + direct_pte_prefetch_many(vcpu, sp, start, spte); } static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep) -- 2.33.0.rc1.237.g0d66db33f3-goog