Re: Deadlock due to EPT_VIOLATION

[Sean Christopherson <seanjc@google.com>]

On Mon, Aug 14, 2023, Eric Wheeler wrote:
> On Tue, 8 Aug 2023, Sean Christopherson wrote:
> > > If you have any suggestions on how modifying the host kernel (and then migrating
> > > a locked up guest to it) or eBPF programs that might help illuminate the issue
> > > further, let me know!
> > > 
> > > Thanks for all your help so far!
> > 
> > Since it sounds like you can test with a custom kernel, try running with this
> > patch and then enable the kvm_page_fault tracepoint when a vCPU gets stuck.  The
> > below expands said tracepoint to capture information about mmu_notifiers and
> > memslots generation.  With luck, it will reveal a smoking gun.
> 
> Getting this patch into production systems is challenging, perhaps live
> patching is an option:

Ah, I take when you gathered information after a live migration you were migrating
VMs into a sidecar environment.

> Questions:
> 
> 1. Do you know if this would be safe to insert as a live kernel patch?

Hmm, probably not safe.

> For example, does adding to TRACE_EVENT modify a struct (which is not
> live-patch-safe) or is it something that should plug in with simple
> function redirection?

Yes, the tracepoint defines a struct, e.g. in this case trace_event_raw_kvm_page_fault.

Looking back, I think I misinterpreted an earlier response regarding bpftrace and
unnecessarily abandoned that tactic. *sigh*

If your environment provides btf info, then this bpftrace program should provide
the mmu_notifier half of the tracepoint hack-a-patch.  If this yields nothing
interesting then we can try diving into whether or not the mmu_root is stale, but
let's cross that bridge when we have to.

I recommend loading this only when you have a stuck vCPU, it'll be quite noisy.

kprobe:handle_ept_violation
{
	printf("vcpu = %lx pid = %u MMU seq = %lx, in-prog = %lx, start = %lx, end = %lx\n",
	       arg0, ((struct kvm_vcpu *)arg0)->pid->numbers[0].nr,
	       ((struct kvm_vcpu *)arg0)->kvm->mmu_invalidate_seq,
	       ((struct kvm_vcpu *)arg0)->kvm->mmu_invalidate_in_progress,
	       ((struct kvm_vcpu *)arg0)->kvm->mmu_invalidate_range_start,
	       ((struct kvm_vcpu *)arg0)->kvm->mmu_invalidate_range_end);
}

If you don't have BTF info, we can still use a bpf program, but to get at the
fields of interested, I think we'd have to resort to pointer arithmetic with struct
offsets grab from your build.