From: Alexander Graf <graf@amazon.com>
To: Anup Patel <anup@brainfault.org>
Cc: Anup Patel <Anup.Patel@wdc.com>,
Palmer Dabbelt <palmer@sifive.com>,
"Paul Walmsley" <paul.walmsley@sifive.com>,
Paolo Bonzini <pbonzini@redhat.com>, Radim K <rkrcmar@redhat.com>,
Daniel Lezcano <daniel.lezcano@linaro.org>,
Thomas Gleixner <tglx@linutronix.de>,
Atish Patra <Atish.Patra@wdc.com>,
Alistair Francis <Alistair.Francis@wdc.com>,
Damien Le Moal <Damien.LeMoal@wdc.com>,
Christoph Hellwig <hch@infradead.org>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
"linux-riscv@lists.infradead.org"
<linux-riscv@lists.infradead.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v5 13/20] RISC-V: KVM: Implement stage2 page table programming
Date: Thu, 22 Aug 2019 15:27:12 +0200 [thread overview]
Message-ID: <a44f86ac-8902-0aa3-1eee-013ac97d667b@amazon.com> (raw)
In-Reply-To: <CAAhSdy1h+m0gA2pro-XAb4qhe0Q+8knjW+8+6jaz3efOdKWskA@mail.gmail.com>
On 22.08.19 14:38, Anup Patel wrote:
> On Thu, Aug 22, 2019 at 5:58 PM Alexander Graf <graf@amazon.com> wrote:
>>
>> On 22.08.19 10:45, Anup Patel wrote:
>>> This patch implements all required functions for programming
>>> the stage2 page table for each Guest/VM.
>>>
>>> At high-level, the flow of stage2 related functions is similar
>>> from KVM ARM/ARM64 implementation but the stage2 page table
>>> format is quite different for KVM RISC-V.
>>>
>>> Signed-off-by: Anup Patel <anup.patel@wdc.com>
>>> Acked-by: Paolo Bonzini <pbonzini@redhat.com>
>>> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
>>> ---
>>> arch/riscv/include/asm/kvm_host.h | 10 +
>>> arch/riscv/include/asm/pgtable-bits.h | 1 +
>>> arch/riscv/kvm/mmu.c | 637 +++++++++++++++++++++++++-
>>> 3 files changed, 638 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
>>> index 3b09158f80f2..a37775c92586 100644
>>> --- a/arch/riscv/include/asm/kvm_host.h
>>> +++ b/arch/riscv/include/asm/kvm_host.h
>>> @@ -72,6 +72,13 @@ struct kvm_mmio_decode {
>>> int shift;
>>> };
>>>
>>> +#define KVM_MMU_PAGE_CACHE_NR_OBJS 32
>>> +
>>> +struct kvm_mmu_page_cache {
>>> + int nobjs;
>>> + void *objects[KVM_MMU_PAGE_CACHE_NR_OBJS];
>>> +};
>>> +
>>> struct kvm_cpu_context {
>>> unsigned long zero;
>>> unsigned long ra;
>>> @@ -163,6 +170,9 @@ struct kvm_vcpu_arch {
>>> /* MMIO instruction details */
>>> struct kvm_mmio_decode mmio_decode;
>>>
>>> + /* Cache pages needed to program page tables with spinlock held */
>>> + struct kvm_mmu_page_cache mmu_page_cache;
>>> +
>>> /* VCPU power-off state */
>>> bool power_off;
>>>
>>> diff --git a/arch/riscv/include/asm/pgtable-bits.h b/arch/riscv/include/asm/pgtable-bits.h
>>> index bbaeb5d35842..be49d62fcc2b 100644
>>> --- a/arch/riscv/include/asm/pgtable-bits.h
>>> +++ b/arch/riscv/include/asm/pgtable-bits.h
>>> @@ -26,6 +26,7 @@
>>>
>>> #define _PAGE_SPECIAL _PAGE_SOFT
>>> #define _PAGE_TABLE _PAGE_PRESENT
>>> +#define _PAGE_LEAF (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)
>>>
>>> /*
>>> * _PAGE_PROT_NONE is set on not-present pages (and ignored by the hardware) to
>>> diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
>>> index 2b965f9aac07..9e95ab6769f6 100644
>>> --- a/arch/riscv/kvm/mmu.c
>>> +++ b/arch/riscv/kvm/mmu.c
>>> @@ -18,6 +18,432 @@
>>> #include <asm/page.h>
>>> #include <asm/pgtable.h>
>>>
>>> +#ifdef CONFIG_64BIT
>>> +#define stage2_have_pmd true
>>> +#define stage2_gpa_size ((phys_addr_t)(1ULL << 39))
>>> +#define stage2_cache_min_pages 2
>>> +#else
>>> +#define pmd_index(x) 0
>>> +#define pfn_pmd(x, y) ({ pmd_t __x = { 0 }; __x; })
>>> +#define stage2_have_pmd false
>>> +#define stage2_gpa_size ((phys_addr_t)(1ULL << 32))
>>> +#define stage2_cache_min_pages 1
>>> +#endif
>>> +
>>> +static int stage2_cache_topup(struct kvm_mmu_page_cache *pcache,
>>> + int min, int max)
>>> +{
>>> + void *page;
>>> +
>>> + BUG_ON(max > KVM_MMU_PAGE_CACHE_NR_OBJS);
>>> + if (pcache->nobjs >= min)
>>> + return 0;
>>> + while (pcache->nobjs < max) {
>>> + page = (void *)__get_free_page(GFP_KERNEL | __GFP_ZERO);
>>> + if (!page)
>>> + return -ENOMEM;
>>> + pcache->objects[pcache->nobjs++] = page;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static void stage2_cache_flush(struct kvm_mmu_page_cache *pcache)
>>> +{
>>> + while (pcache && pcache->nobjs)
>>> + free_page((unsigned long)pcache->objects[--pcache->nobjs]);
>>> +}
>>> +
>>> +static void *stage2_cache_alloc(struct kvm_mmu_page_cache *pcache)
>>> +{
>>> + void *p;
>>> +
>>> + if (!pcache)
>>> + return NULL;
>>> +
>>> + BUG_ON(!pcache->nobjs);
>>> + p = pcache->objects[--pcache->nobjs];
>>> +
>>> + return p;
>>> +}
>>> +
>>> +struct local_guest_tlb_info {
>>> + struct kvm_vmid *vmid;
>>> + gpa_t addr;
>>> +};
>>> +
>>> +static void local_guest_tlb_flush_vmid_gpa(void *info)
>>> +{
>>> + struct local_guest_tlb_info *infop = info;
>>> +
>>> + __kvm_riscv_hfence_gvma_vmid_gpa(READ_ONCE(infop->vmid->vmid_version),
>>> + infop->addr);
>>> +}
>>> +
>>> +static void stage2_remote_tlb_flush(struct kvm *kvm, gpa_t addr)
>>> +{
>>> + struct local_guest_tlb_info info;
>>> + struct kvm_vmid *vmid = &kvm->arch.vmid;
>>> +
>>> + /* TODO: This should be SBI call */
>>> + info.vmid = vmid;
>>> + info.addr = addr;
>>> + preempt_disable();
>>> + smp_call_function_many(cpu_all_mask, local_guest_tlb_flush_vmid_gpa,
>>> + &info, true);
>>
>> This is all nice and dandy on the toy 4 core systems we have today, but
>> it will become a bottleneck further down the road.
>>
>> How many VMIDs do you have? Could you just allocate a new one every time
>> you switch host CPUs? Then you know exactly which CPUs to flush by
>> looking at all your vcpu structs and a local field that tells you which
>> pCPU they're on at this moment.
>>
>> Either way, it's nothing that should block inclusion. For today, we're fine.
>
> We are not happy about this either.
>
> Other two options, we have are:
> 1. Have SBI calls for remote HFENCEs
> 2. Propose RISC-V ISA extension for remote FENCEs
>
> Option1 is mostly extending SBI spec and implementing it in runtime
> firmware.
>
> Option2 is ideal solution but requires consensus among wider audience
> in RISC-V foundation.
>
> At this point, we are fine with a simple solution.
It's fine to explicitly IPI other CPUs to flush their TLBs. What is not
fine is to IPI *all* CPUs to flush their TLBs.
Alex
next prev parent reply other threads:[~2019-08-22 13:27 UTC|newest]
Thread overview: 61+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-22 8:42 [PATCH v5 00/20] KVM RISC-V Support Anup Patel
2019-08-22 8:42 ` [PATCH v5 01/20] KVM: RISC-V: Add KVM_REG_RISCV for ONE_REG interface Anup Patel
2019-08-22 8:42 ` [PATCH v5 02/20] RISC-V: Add bitmap reprensenting ISA features common across CPUs Anup Patel
2019-08-22 8:43 ` [PATCH v5 03/20] RISC-V: Export few kernel symbols Anup Patel
2019-08-22 8:43 ` [PATCH v5 04/20] RISC-V: Add hypervisor extension related CSR defines Anup Patel
2019-08-22 8:43 ` [PATCH v5 05/20] RISC-V: Add initial skeletal KVM support Anup Patel
2019-08-22 8:43 ` [PATCH v5 06/20] RISC-V: KVM: Implement VCPU create, init and destroy functions Anup Patel
2019-08-22 8:44 ` [PATCH v5 07/20] RISC-V: KVM: Implement VCPU interrupts and requests handling Anup Patel
2019-08-22 8:44 ` [PATCH v5 08/20] RISC-V: KVM: Implement KVM_GET_ONE_REG/KVM_SET_ONE_REG ioctls Anup Patel
2019-08-22 12:01 ` Alexander Graf
2019-08-22 14:00 ` Anup Patel
2019-08-22 14:12 ` Alexander Graf
2019-08-23 11:20 ` Anup Patel
2019-08-23 11:42 ` Graf (AWS), Alexander
2019-08-22 14:05 ` Anup Patel
2019-08-22 8:44 ` [PATCH v5 09/20] RISC-V: KVM: Implement VCPU world-switch Anup Patel
2019-08-22 8:44 ` [PATCH v5 10/20] RISC-V: KVM: Handle MMIO exits for VCPU Anup Patel
2019-08-22 12:10 ` Alexander Graf
2019-08-22 12:21 ` Andrew Jones
2019-08-22 12:27 ` Anup Patel
2019-08-22 12:14 ` Alexander Graf
2019-08-22 12:33 ` Anup Patel
2019-08-22 13:25 ` Alexander Graf
2019-08-22 13:55 ` Anup Patel
2019-08-22 8:45 ` [PATCH v5 11/20] RISC-V: KVM: Handle WFI " Anup Patel
2019-08-22 12:19 ` Alexander Graf
2019-08-22 12:50 ` Anup Patel
2019-08-22 8:45 ` [PATCH v5 12/20] RISC-V: KVM: Implement VMID allocator Anup Patel
2019-08-22 8:45 ` [PATCH v5 13/20] RISC-V: KVM: Implement stage2 page table programming Anup Patel
2019-08-22 12:28 ` Alexander Graf
2019-08-22 12:38 ` Anup Patel
2019-08-22 13:27 ` Alexander Graf [this message]
2019-08-22 13:58 ` Anup Patel
2019-08-22 14:09 ` Alexander Graf
2019-08-23 11:21 ` Anup Patel
2019-08-22 8:45 ` [PATCH v5 14/20] RISC-V: KVM: Implement MMU notifiers Anup Patel
2019-08-22 8:46 ` [PATCH v5 15/20] RISC-V: KVM: Add timer functionality Anup Patel
2019-08-23 7:52 ` Alexander Graf
2019-08-23 11:04 ` Anup Patel
2019-08-23 11:33 ` Graf (AWS), Alexander
2019-08-23 11:46 ` Anup Patel
2019-08-23 11:49 ` Alexander Graf
2019-08-23 12:11 ` Anup Patel
2019-08-23 12:25 ` Alexander Graf
2019-08-22 8:46 ` [PATCH v5 16/20] RISC-V: KVM: FP lazy save/restore Anup Patel
2019-08-22 8:46 ` [PATCH v5 17/20] RISC-V: KVM: Implement ONE REG interface for FP registers Anup Patel
2019-08-22 8:46 ` [PATCH v5 18/20] RISC-V: KVM: Add SBI v0.1 support Anup Patel
2019-08-23 8:04 ` Alexander Graf
2019-08-23 11:17 ` Anup Patel
2019-08-23 11:38 ` Graf (AWS), Alexander
2019-08-23 12:00 ` Anup Patel
2019-08-23 12:19 ` Alexander Graf
2019-08-23 12:28 ` Anup Patel
2019-08-22 8:47 ` [PATCH v5 19/20] RISC-V: Enable VIRTIO drivers in RV64 and RV32 defconfig Anup Patel
2019-08-22 8:47 ` [PATCH v5 20/20] RISC-V: KVM: Add MAINTAINERS entry Anup Patel
2019-08-23 8:08 ` [PATCH v5 00/20] KVM RISC-V Support Alexander Graf
2019-08-23 11:25 ` Anup Patel
2019-08-23 11:44 ` Graf (AWS), Alexander
2019-08-23 12:10 ` Paolo Bonzini
2019-08-23 12:19 ` Anup Patel
2019-08-23 12:28 ` Alexander Graf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a44f86ac-8902-0aa3-1eee-013ac97d667b@amazon.com \
--to=graf@amazon.com \
--cc=Alistair.Francis@wdc.com \
--cc=Anup.Patel@wdc.com \
--cc=Atish.Patra@wdc.com \
--cc=Damien.LeMoal@wdc.com \
--cc=anup@brainfault.org \
--cc=daniel.lezcano@linaro.org \
--cc=hch@infradead.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=palmer@sifive.com \
--cc=paul.walmsley@sifive.com \
--cc=pbonzini@redhat.com \
--cc=rkrcmar@redhat.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).