Re: [PATCH 00/13] kvm: selftests: add aarch64 framework and dirty

From: Andrew Jones <drjones@redhat.com>
To: Christoffer Dall <christoffer.dall@arm.com>
Cc: kvm@vger.kernel.org, marc.zyngier@arm.com, pbonzini@redhat.com,
	kvmarm@lists.cs.columbia.edu
Subject: Re: [PATCH 00/13] kvm: selftests: add aarch64 framework and dirty
Date: Tue, 30 Oct 2018 18:38:20 +0100	[thread overview]
Message-ID: <20181030173820.3v2czxqkeaicfqtn@kamzik.brq.redhat.com> (raw)
In-Reply-To: <20181029174002.GB12057@e113682-lin.lund.arm.com>

Hi Christoffer,

Thanks for your interest in these tests. There isn't any documentation
that I know of, but it's a good idea to have some. I'll write something
up soon. I'll also try to answer your questions now.

On Mon, Oct 29, 2018 at 06:40:02PM +0100, Christoffer Dall wrote:
> Hi Drew,
> 
> On Tue, Sep 18, 2018 at 07:54:23PM +0200, Andrew Jones wrote:
> > This series provides KVM selftests that test dirty log tracking on
> > AArch64 for both 4K and 64K guest page sizes. Additionally the
> > framework provides an easy way to test dirty log tracking with the
> > recently posted dynamic IPA and 52bit IPA series[1].
> 
> I was trying to parse the commit text of patch 2, and I realized that I
> don't understand the 'hypercall to userspace' thing at all, which
> probably means I have no idea how the selftests work overall.

There are three parts to a kvm selftest: 1) the test code which runs in
host userspace and _is_ the kvm userspace used with kvm, 2) the vcpu
thread code which executes KVM_RUN for the guest code and possibly also
some host userspace test code, and 3) the guest code, which is naturally
run in the vcpu thread, but in guest mode.

The need for a "ucall" arises for 2's "possibly also some host userspace
test code". In that case the guest code needs to invoke an exit from guest
mode, not just to kvm, but all the way to kvm userspace. For AArch64, as
you know, this can be done with an MMIO access. The reason patch 2
generalizes the concept is because for x86 this can and is done with a
PIO access.

> 
> I then spent a while reading various bits of documentation in the kernel
> tree, LWN, etc., only to realize that I don't understand how this test
> framework actually works.
> 
> Are the selftests modules, userspace programs, or code that is compiled
> with the kernel, and (somehow?) run from userspace.  I thought the
> latter, partially based on your explanation at ELC, but then I don't
> understand how the "compile and run" make target works.

The tests are standalone userspace programs which are compiled separately,
but have dependencies on kernel headers. As stated above, for kvm, each
selftest is a kvm userspace (including its vcpu thread code) and guest
code combined. While there's a lot of complexity in the framework,
particularly for memory management, and a bit for vcpu setup, most of that
can be shared among tests using the kvm_util.h and test_util.h APIs,
allowing a given test to only have a relatively simple main(), vcpu thread
"vcpu_worker()" function, and "guest_code()" function. Guest mode code can
easily share code with the kvm userspace test code (assuming the guest
page tables are set up in the default way) and even data can be shared as
long the accesses are done with the appropriate mappings (gva vs. hva).
There's a small API to help with that as well.

> 
> Can you help me paint the overall picture, or point me to the piece of
> documentation/presentation that explains the high-level picture, which I
> must have obviously missed somehow?

We definitely need the documentation and, in hindsight, it looks like it
would have been a good BoF topic last week too.

I think this framework has a lot of potential for KVM API testing and
even for quick & dirty guest code instruction sequence tests (although
instruction sequences would also fit kvm-unit-tests). I hope I can help
get you and anyone else interested started.

Thanks,
drew

> 
> 
> Thanks!
> 
>     Christoffer
> 
> > 
> > The series breaks down into parts as follows:
> > 
> >  01-02: generalize guest code to host userspace exit support by
> >         introducing "ucalls" - hypercalls to userspace
> >  03-05: prepare common code for a new architecture
> >  06-07: add virtual memory setup support for AArch64
> >     08: add vcpu setup support for AArch64
> >     09: port the dirty log test to AArch64
> >  10-11: add 64K guest page size support for the dirty log test
> >  12-13: prepare the dirty log test to also test > 40-bit guest
> >         physical address setups by allowing the test memory
> >         region to be placed at the top of physical memory
> > 
> > [1] https://www.spinics.net/lists/arm-kernel/msg676819.html
> > 
> > Thanks,
> > drew
> > 
> > 
> > Andrew Jones (13):
> >   kvm: selftests: vcpu_setup: set cr4.osfxsr
> >   kvm: selftests: introduce ucall
> >   kvm: selftests: move arch-specific files to arch-specific locations
> >   kvm: selftests: add cscope make target
> >   kvm: selftests: tidy up kvm_util
> >   kvm: selftests: add vm_phy_pages_alloc
> >   kvm: selftests: add virt mem support for aarch64
> >   kvm: selftests: add vcpu support for aarch64
> >   kvm: selftests: port dirty_log_test to aarch64
> >   kvm: selftests: introduce new VM mode for 64K pages
> >   kvm: selftests: dirty_log_test: also test 64K pages on aarch64
> >   kvm: selftests: stop lying to aarch64 tests about PA-bits
> >   kvm: selftests: support high GPAs in dirty_log_test
> > 
> >  tools/testing/selftests/kvm/.gitignore        |  11 +-
> >  tools/testing/selftests/kvm/Makefile          |  36 +-
> >  tools/testing/selftests/kvm/dirty_log_test.c  | 374 +++++++++----
> >  .../selftests/kvm/include/aarch64/processor.h |  55 ++
> >  .../testing/selftests/kvm/include/kvm_util.h  | 166 +++---
> >  .../testing/selftests/kvm/include/sparsebit.h |   6 +-
> >  .../testing/selftests/kvm/include/test_util.h |   6 +-
> >  .../kvm/include/{x86.h => x86_64/processor.h} |  24 +-
> >  .../selftests/kvm/include/{ => x86_64}/vmx.h  |   6 +-
> >  .../selftests/kvm/lib/aarch64/processor.c     | 311 +++++++++++
> >  tools/testing/selftests/kvm/lib/assert.c      |   2 +-
> >  tools/testing/selftests/kvm/lib/kvm_util.c    | 499 +++++++-----------
> >  .../selftests/kvm/lib/kvm_util_internal.h     |  33 +-
> >  tools/testing/selftests/kvm/lib/ucall.c       | 144 +++++
> >  .../kvm/lib/{x86.c => x86_64/processor.c}     | 197 ++++++-
> >  .../selftests/kvm/lib/{ => x86_64}/vmx.c      |   4 +-
> >  .../kvm/{ => x86_64}/cr4_cpuid_sync_test.c    |  14 +-
> >  .../kvm/{ => x86_64}/set_sregs_test.c         |   2 +-
> >  .../selftests/kvm/{ => x86_64}/state_test.c   |  25 +-
> >  .../kvm/{ => x86_64}/sync_regs_test.c         |   2 +-
> >  .../kvm/{ => x86_64}/vmx_tsc_adjust_test.c    |  23 +-
> >  21 files changed, 1329 insertions(+), 611 deletions(-)
> >  create mode 100644 tools/testing/selftests/kvm/include/aarch64/processor.h
> >  rename tools/testing/selftests/kvm/include/{x86.h => x86_64/processor.h} (98%)
> >  rename tools/testing/selftests/kvm/include/{ => x86_64}/vmx.h (99%)
> >  create mode 100644 tools/testing/selftests/kvm/lib/aarch64/processor.c
> >  create mode 100644 tools/testing/selftests/kvm/lib/ucall.c
> >  rename tools/testing/selftests/kvm/lib/{x86.c => x86_64/processor.c} (85%)
> >  rename tools/testing/selftests/kvm/lib/{ => x86_64}/vmx.c (99%)
> >  rename tools/testing/selftests/kvm/{ => x86_64}/cr4_cpuid_sync_test.c (91%)
> >  rename tools/testing/selftests/kvm/{ => x86_64}/set_sregs_test.c (98%)
> >  rename tools/testing/selftests/kvm/{ => x86_64}/state_test.c (90%)
> >  rename tools/testing/selftests/kvm/{ => x86_64}/sync_regs_test.c (99%)
> >  rename tools/testing/selftests/kvm/{ => x86_64}/vmx_tsc_adjust_test.c (91%)
> > 
> > -- 
> > 2.17.1
> > 