Re: Deadlock due to EPT_VIOLATION

[Sean Christopherson <seanjc@google.com>]

On Mon, Jul 24, 2023, Amaan Cheval wrote:
> > > I've also run a `function_graph` trace on some of the affected hosts, if you
> > > think it might be helpful...
> >
> > It wouldn't hurt to see it.
> >
> 
> Here you go:
> https://transfer.sh/SfXSCHp5xI/ept-function-graph.log

Yeesh.  There is a ridiculous amount of potentially problematic activity.  KSM is
active in that trace, it looks like NUMA balancing might be in play, there might
be hugepage shattering, etc.

> > > Another interesting observation we made was that when we migrate a guest to a
> > > different host, the guest _stays_ locked up and throws EPT violations on the new
> > > host as well
> >
> > Ooh, that's *very* interesting.  That pretty much rules out memslot and mmu_notifier
> > issues.
> 
> Good to know, thanks!

Let me rephrase that statement: it rules out a certain class of memslot and
mmu_notifier bugs, namely bugs where KVM would incorrect leave an invalidation
refcount (for lack of a better term) elevated.  It doesn't mean memslot changes
and/or mmu_notifier events aren't at fault.

Can you migrate a hung guest to a host that is completely unloaded?  And ideally,
disable KSM and NUMA autobalancing on the target host.  And then get a
function_graph trace on that host, assuming the vCPU remains stuck.  There is *so*
much going on in the above graph that it's impossible to determine if there's a
kernel bug, e.g. it's possible the vCPU is stuck purely because it's being trashed
to the point where it can't make forward progress.

> > To mostly confirm this is likely what's happening, can you enable all of the async
> > #PF tracepoints in KVM?  The exact tracepoints might vary dependending on which kernel
> > version you're running, just enable everything with "async" in the name, e.g.
> >
> >   # ls -1 /sys/kernel/debug/tracing/events/kvm | grep async
> >   kvm_async_pf_completed/
> >   kvm_async_pf_not_present/
> >   kvm_async_pf_ready/
> >   kvm_async_pf_repeated_fault/
> >   kvm_try_async_get_page/
> >
> > If kvm_try_async_get_page() is more or less keeping pace with the "pf_taken" stat,
> > then this is likely what's happening.
> 
> I did this and unfortunately, don't see any of these functions being
> called at all despite
> EPT_VIOLATIONs still being thrown and pf_taken still climbing. (Tried both with
> `trace-cmd -e ...` and using `bpftrace` and none of those functions
> are being called
> during the deadlock/guest being stuck.)

Well fudge.

> > And then to really confirm, this small bpf program will yell if get_user_pages_remote()
> > fails when attempting get a single page (which is always the case for KVM's async
> > #PF usage).
> >
> > $ tail gup_remote.bt
> > kretfunc:get_user_pages_remote
> > {
> >         if ( args->nr_pages == 1 && retval != 1 ) {
> >                 printf("Failed remote gup() on address %lx, ret = %d\n", args->start, retval);
> >         }
> > }
> >
> 
> Our hosts don't have kfunc/kretfunc support (`bpftrace --info` reports
> `kret: no`),
> but I tried just a kprobe to verify that get_user_pages_remote is
> being called at all -
> does not seem like it is, unfortunately:
> 
> ```
> # bpftrace -e 'kprobe:get_user_pages_remote { @[comm] = count(); }'
> Attaching 1 probe...
> ^C
> #
> ```
> 
> So I guess that disproves the async #PF theory?

Yeah.  Definitely not related async page fault.