Re: cr3 OOS optimisation breaks 32-bit GNU/kFreeBSD guest

From: Aurelien Jarno <aurelien@aurel32.net>
To: Marcelo Tosatti <mtosatti@redhat.com>
Cc: kvm@vger.kernel.org
Subject: Re: cr3 OOS optimisation breaks 32-bit GNU/kFreeBSD guest
Date: Mon, 23 Feb 2009 15:01:15 +0100	[thread overview]
Message-ID: <20090223140115.GB5946@hall.aurel32.net> (raw)
In-Reply-To: <20090223014713.GA11438@amt.cnet>

On Sun, Feb 22, 2009 at 10:47:13PM -0300, Marcelo Tosatti wrote:
> On Mon, Feb 23, 2009 at 01:33:05AM +0100, Aurelien Jarno wrote:
> > Hi,
> > 
> > Since kvm-81, I have noticed that GNU/kFreeBSD 32-bit guest are crashing
> > under high load (during a compilation for example) with the following 
> > error message:
> > 
> > | Fatal trap 12: page fault while in kernel mode
> > | fault virtual address   = 0x4
> > | fault code              = supervisor read, page not present
> > | instruction pointer     = 0x20:0xc0a4fc00
> > | stack pointer           = 0x28:0xe66d7a70
> > | frame pointer           = 0x28:0xe66d7a80
> > | code segment            = base 0x0, limit 0xfffff, type 0x1b
> > |                         = DPL 0, pres 1, def32 1, gran 1
> > | processor eflags        = interrupt enabled, resume, IOPL = 0
> > | current process         = 24037 (bash)
> > | trap number             = 12
> > | panic: page fault
> > | Uptime: 4m7s
> > | Cannot dump. No dump device defined.
> > | Automatic reboot in 15 seconds - press a key on the console to abort
> >  
> > I haven't tried yet with a plain FreeBSD guest, but I also expect it to
> > crash given the kernel (version 7.1) is almost the same. A closer 
> > investigation has shown that the following commit is causing the 
> > problem:
> > 
> > | commit 6364a3918cb5c28376849e7fca3e09bd66b859f3
> > | Author: Marcelo Tosatti <mtosatti@redhat.com>
> > | Date:   Mon Dec 1 22:32:04 2008 -0200
> > | 
> > |     KVM: MMU: skip global pgtables on sync due to cr3 switch
> > | 
> > |     Skip syncing global pages on cr3 switch (but not on cr4/cr0). This is
> > |     important for Linux 32-bit guests with PAE, where the kmap page is
> > |     marked as global.
> > | 
> > |     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> > |     Signed-off-by: Avi Kivity <avi@redhat.com>
> > 
> > As expected, loading the KVM module with oos_shadow=0 workaround the
> > problem. Please note that the guest is running in 32-bit mode, does not
> > use PAE, and uses global pages. My host has an Intel Q9450 CPU, and the
> > problem appears with both a 2.6.26 and a 2.6.28 64-bit kernel.
> > 
> > Does anybody see any problem in this patch? How can I further
> > debug the problem?
> 
> Aurelien,
> 
> Maybe there is a bug in the syncing code (eg: not all global pages are
> sync'ed when the OS requests a global sync), or FreeBSD is "relying" on
> invlpg/cr3 write to sync global pages (remember TLB entries can be
> invalidated internally by CPU).

As far as I understand the Intel IA32 manual, using invlpg to sync
global pages is correct. OTOH, cr3 is not. I think that if FreeBSD is
using cr3 to sync global pages that should also cause a problem on real
hardware sooner or later, right?

I'll try to look at the kernel code.

> If you want to debug it, would suggest looping over all MMU pages in
> mmu_sync_global, after the kvm_sync_page loop, and
> 
>       WARN_ON(sp->unsync && sp->global);
> 
> If that fails, check if the unsync and global flags mean what they are
> supposed to.

This doesn't detect any problem, which means that all pages marked as
global are synced correctly.

I have also tried to call kvm_mmu_sync_global() in kvm_set_cr3(), and
as expected the guest works correctly in that case.

If I understand correctly, and if the problem is not on the FreeBSD
side, the only remaining possible problem is if normal pages are wrongly
marked as global, and thus should be synced with cr3, while they are 
not. Am I right here?

> Sorry for the trouble and thanks for the detailed report, will take a
> close look at it this week.
> 

Thanks for your fast answer and for your help for debugging.

-- 
Aurelien Jarno	                        GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net