multicalls.c warning in xen_mc_flush

multicalls.c warning in xen_mc_flush

[William Dauchy]

Hello,

I'm getting a call trace when booting a dom0 v3.3.7 (x86_32) under xen 4.1.2.
It is about multicalls.c:129. I tried with MC_DEBUG = 1 but it appears
that b->mcidx = 1 when getting this call trace.
Does anybody have any clue?


WARNING: at /arch/x86/xen/multicalls.c:129 xen_mc_flush+0x12a/0x140()
Hardware name: C6100
Modules linked in:
Pid: 0, comm: swapper Not tainted 3.3.7-i386 #125
Call Trace:
 [<c102f34d>] warn_slowpath_common+0x6d/0xa0
 [<c100384a>] ? xen_mc_flush+0x12a/0x140
 [<c100384a>] ? xen_mc_flush+0x12a/0x140
 [<c102f39d>] warn_slowpath_null+0x1d/0x20
 [<c100384a>] xen_mc_flush+0x12a/0x140
 [<c1006035>] xen_set_pmd_hyper+0x75/0x80
 [<c102c4ee>] set_pmd_pfn+0x9e/0xf0
 [<c14e68c7>] init_alloc_remap+0x1c3/0x237
 [<c14e6315>] x86_numa_init+0x378/0x691
 [<c1004af6>] ? xen_set_fixmap+0xa6/0x160
 [<c14e6639>] initmem_init+0xb/0xd6
 [<c14dea61>] ? early_acpi_boot_init+0x6e/0xbe
 [<c14dede0>] ? acpi_boot_table_init+0x36/0x7d
 [<c14d7796>] setup_arch+0xb30/0xbe7
 [<c14d1566>] start_kernel+0x76/0x2ef
 [<c14d10ba>] i386_start_kernel+0xa9/0xb0
 [<c14d4a66>] xen_start_kernel+0x546/0x54e


Regards,
-- 
William

Re: multicalls.c warning in xen_mc_flush

[Konrad Rzeszutek Wilk]

On Tue, May 22, 2012 at 03:04:17PM +0200, William Dauchy wrote:
> Hello,
> 
> I'm getting a call trace when booting a dom0 v3.3.7 (x86_32) under xen 4.1.2.
> It is about multicalls.c:129. I tried with MC_DEBUG = 1 but it appears
> that b->mcidx = 1 when getting this call trace.
> Does anybody have any clue?

Yes. Do you have CONFIG_NUMA enabled? If you disable it, do the messages
disappear?

> 
> 
> WARNING: at /arch/x86/xen/multicalls.c:129 xen_mc_flush+0x12a/0x140()
> Hardware name: C6100
> Modules linked in:
> Pid: 0, comm: swapper Not tainted 3.3.7-i386 #125
> Call Trace:
>  [<c102f34d>] warn_slowpath_common+0x6d/0xa0
>  [<c100384a>] ? xen_mc_flush+0x12a/0x140
>  [<c100384a>] ? xen_mc_flush+0x12a/0x140
>  [<c102f39d>] warn_slowpath_null+0x1d/0x20
>  [<c100384a>] xen_mc_flush+0x12a/0x140
>  [<c1006035>] xen_set_pmd_hyper+0x75/0x80
>  [<c102c4ee>] set_pmd_pfn+0x9e/0xf0
>  [<c14e68c7>] init_alloc_remap+0x1c3/0x237
>  [<c14e6315>] x86_numa_init+0x378/0x691
>  [<c1004af6>] ? xen_set_fixmap+0xa6/0x160
>  [<c14e6639>] initmem_init+0xb/0xd6
>  [<c14dea61>] ? early_acpi_boot_init+0x6e/0xbe
>  [<c14dede0>] ? acpi_boot_table_init+0x36/0x7d
>  [<c14d7796>] setup_arch+0xb30/0xbe7
>  [<c14d1566>] start_kernel+0x76/0x2ef
>  [<c14d10ba>] i386_start_kernel+0xa9/0xb0
>  [<c14d4a66>] xen_start_kernel+0x546/0x54e
> 
> 
> Regards,
> -- 
> William
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: multicalls.c warning in xen_mc_flush

[William Dauchy]

On Tue, May 22, 2012 at 8:36 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> Yes. Do you have CONFIG_NUMA enabled? If you disable it, do the messages
> disappear?

yes.
hmm now I remember that you already talked about it
http://lists.xen.org/archives/html/xen-devel/2012-05/msg00082.html
if you have any patch to test, I'll be happy to help.

Regards,
-- 
William

Re: multicalls.c warning in xen_mc_flush

[Konrad Rzeszutek Wilk]

On Wed, May 23, 2012 at 12:46:04AM +0200, William Dauchy wrote:
> On Tue, May 22, 2012 at 8:36 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > Yes. Do you have CONFIG_NUMA enabled? If you disable it, do the messages
> > disappear?
> 
> yes.
> hmm now I remember that you already talked about it
> http://lists.xen.org/archives/html/xen-devel/2012-05/msg00082.html
> if you have any patch to test, I'll be happy to help.

Not yet. Could you ping me in  week say please?


> 
> Regards,
> -- 
> William
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: multicalls.c warning in xen_mc_flush

[William Dauchy]

Hi Konrad,

On Fri, May 25, 2012 at 11:01 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> Not yet. Could you ping me in  week say please?

sure.

-- 
William

Re: multicalls.c warning in xen_mc_flush

[William Dauchy]

On Fri, May 25, 2012 at 11:01 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> Not yet. Could you ping me in  week say please?

ping.
-- 
William

Re: multicalls.c warning in xen_mc_flush

[Konrad Rzeszutek Wilk]

On Tue, May 29, 2012 at 01:39:39PM +0200, William Dauchy wrote:
> On Fri, May 25, 2012 at 11:01 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > Not yet. Could you ping me in  week say please?
> 
> ping.

Pls try the attached patch.

>From e4c315c0c3d842712ae64ec95c099fd44e65291a Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Tue, 29 May 2012 21:38:22 -0400
Subject: [PATCH] x86/i386: Check PSE bit before using PAGE_KERNEL_LARGE.

During bootup we would unconditionally do this on non-NUMA machines:

setup_arch
  \-initmem_init
      \-x86_numa_init (with dummy_init as callback)
          \- init_alloc_remap
               \- set_pmd_pfn (with PAGE_PSE)

without checking to see if the CPU supports PSE. This
patch adds that and also allows the init_alloc_remap function
to properly work by falling back on PTEs.

This bug has been observed when running an i386 PV Xen
guest with CONFIG_NUMA built in - but it should be also
easily observed on other CPUs which do not expose the PSE support.

We would get this in the guest:

memblock_reserve: [0x0000002ac00000-0x0000002be00000] init_alloc_remap+0x195/0x251
------------[ cut here ]------------
WARNING: at /home/konrad/ssd/linux/arch/x86/xen/multicalls.c:129 xen_mc_flush+0x160/0x1e0()
Modules linked in:
Pid: 0, comm: swapper Not tainted 3.4.0-08268-gc0b1dd2 #1
Call Trace:
 [<c107b62d>] warn_slowpath_common+0x6d/0xa0
 [<c10380a0>] ? xen_mc_flush+0x160/0x1e0
 [<c10380a0>] ? xen_mc_flush+0x160/0x1e0
 [<c107b67d>] warn_slowpath_null+0x1d/0x20
 [<c10380a0>] xen_mc_flush+0x160/0x1e0
 [<c103a46d>] xen_set_pmd_hyper+0xad/0x170
 [<c103896d>] ? pte_pfn_to_mfn+0xad/0xc0
 [<c1074b2e>] set_pmd_pfn+0x9e/0xf0
 [<c172290d>] init_alloc_remap+0x1e3/0x251
 [<c1722325>] x86_numa_init+0x340/0x65e
 [<c103c7fe>] ? __raw_callee_save_xen_restore_fl+0x6/0x8
 [<c172265f>] initmem_init+0xb/0xd6
 [<c1719428>] ? acpi_boot_table_init+0x10/0x7d
 [<c1712dd1>] setup_arch+0xb9c/0xc8a
 [<c103c7fe>] ? __raw_callee_save_xen_restore_fl+0x6/0x8
 [<c170c8fb>] start_kernel+0xbe/0x395
 [<c170c306>] i386_start_kernel+0xa9/0xb0
 [<c170f86c>] xen_start_kernel+0x632/0x63a
 [<c1409078>] ? tmem_objnode_alloc+0x28/0xa0
---[ end trace a7919e7f17c0a725 ]---
------------[ cut here ]------------

with the hypervisor telling us:
(XEN) mm.c:943:d0 Attempt to map superpage without allowsuperpage flag in hypervisor

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/mm/pgtable_32.c |   29 ++++++++++++++++++++++++++++-
 1 files changed, 28 insertions(+), 1 deletions(-)

diff --git a/arch/x86/mm/pgtable_32.c b/arch/x86/mm/pgtable_32.c
index a69bcb8..32085ec 100644
--- a/arch/x86/mm/pgtable_32.c
+++ b/arch/x86/mm/pgtable_32.c
@@ -86,7 +86,34 @@ void set_pmd_pfn(unsigned long vaddr, unsigned long pfn, pgprot_t flags)
 	}
 	pud = pud_offset(pgd, vaddr);
 	pmd = pmd_offset(pud, vaddr);
-	set_pmd(pmd, pfn_pmd(pfn, flags));
+
+	if (cpu_has_pse)
+		set_pmd(pmd, pfn_pmd(pfn, flags));
+	else {
+		pgprot_t new_flag = PAGE_KERNEL;
+		pte_t *pte;
+		int i;
+
+		/*
+		 * This is run _after_ initial memory mapped so the
+		 * PTE page are allocated - but we check it just in case.
+		 */
+		if (pmd_none(*pmd)) {
+			printk(KERN_WARNING "set_pmd_pfn: pmd_none\n");
+			return;
+		}
+
+		pte = (pte_t *)pmd_page_vaddr(*pmd);
+		for (i = 0; i < PTRS_PER_PTE; i++) {
+			if (pte_none(*pte)) {
+				printk(KERN_WARNING "set_pmd_pfn: pte_none\n");
+				return;
+			}
+			set_pte(pte, pfn_pte(pfn + i, new_flag));
+			pte++;
+		}
+	}
+
 	/*
 	 * It's enough to flush this one mapping.
 	 * (PGE mappings get flushed as well)
-- 
1.7.7.6

Re: multicalls.c warning in xen_mc_flush

[William Dauchy]

Hello Konrad,

On Wed, May 30, 2012 at 4:18 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> Pls try the attached patch.

Thanks for your quick reply.
I tested the attached patch and it fixes the issue.

> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

Reported-by: William Dauchy <wdauchy@gmail.com>
Tested-by: William Dauchy <wdauchy@gmail.com>

Maybe we should think about proposing it in stable.
Cc: <stable@vger.kernel.org>

Regards,
-- 
William

Re: multicalls.c warning in xen_mc_flush

[Konrad Rzeszutek Wilk]

On Wed, May 30, 2012 at 11:29:43PM +0200, William Dauchy wrote:
> Hello Konrad,
> 
> On Wed, May 30, 2012 at 4:18 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > Pls try the attached patch.
> 
> Thanks for your quick reply.
> I tested the attached patch and it fixes the issue.

Cool. Thanks for testing it out.

> 
> > Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> 
> Reported-by: William Dauchy <wdauchy@gmail.com>
> Tested-by: William Dauchy <wdauchy@gmail.com>
> 
> Maybe we should think about proposing it in stable.
> Cc: <stable@vger.kernel.org>
> 
> Regards,
> -- 
> William
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel