x86-64 bad pmds in 2.6.11.6

x86-64 bad pmds in 2.6.11.6

[Dave Jones]

[apologies to Andi for getting this twice, I goofed the l-k address
 the first time]

 
 I arrived at the office today to find my workstation had this spew
 in its dmesg buffer..
 
 mm/memory.c:97: bad pmd ffff81004b017438(00000038a5500a88).
 mm/memory.c:97: bad pmd ffff81004b017440(0000000000000003).
 mm/memory.c:97: bad pmd ffff81004b017448(00007ffffffff73b).
 mm/memory.c:97: bad pmd ffff81004b017450(00007ffffffff73c).
 mm/memory.c:97: bad pmd ffff81004b017458(00007ffffffff73d).
 mm/memory.c:97: bad pmd ffff81004b017468(00007ffffffff73e).
 mm/memory.c:97: bad pmd ffff81004b017470(00007ffffffff73f).
 mm/memory.c:97: bad pmd ffff81004b017478(00007ffffffff740).
 mm/memory.c:97: bad pmd ffff81004b017480(00007ffffffff741).
 mm/memory.c:97: bad pmd ffff81004b017488(00007ffffffff742).
 mm/memory.c:97: bad pmd ffff81004b017490(00007ffffffff743).
 mm/memory.c:97: bad pmd ffff81004b017498(00007ffffffff744).
 mm/memory.c:97: bad pmd ffff81004b0174a0(00007ffffffff745).
 mm/memory.c:97: bad pmd ffff81004b0174a8(00007ffffffff746).
 mm/memory.c:97: bad pmd ffff81004b0174b0(00007ffffffff747).
 mm/memory.c:97: bad pmd ffff81004b0174b8(00007ffffffff748).
 mm/memory.c:97: bad pmd ffff81004b0174c0(00007ffffffff749).
 mm/memory.c:97: bad pmd ffff81004b0174c8(00007ffffffff74a).
 mm/memory.c:97: bad pmd ffff81004b0174d0(00007ffffffff74b).
 mm/memory.c:97: bad pmd ffff81004b0174d8(00007ffffffff74c).
 mm/memory.c:97: bad pmd ffff81004b0174e0(00007ffffffff74d).
 mm/memory.c:97: bad pmd ffff81004b0174e8(00007ffffffff74e).
 mm/memory.c:97: bad pmd ffff81004b0174f0(00007ffffffff74f).
 mm/memory.c:97: bad pmd ffff81004b0174f8(00007ffffffff750).
 mm/memory.c:97: bad pmd ffff81004b017500(00007ffffffff751).
 mm/memory.c:97: bad pmd ffff81004b017508(00007ffffffff752).
 mm/memory.c:97: bad pmd ffff81004b017510(00007ffffffff753).
 mm/memory.c:97: bad pmd ffff81004b017518(00007ffffffff754).
 mm/memory.c:97: bad pmd ffff81004b017520(00007ffffffff755).
 mm/memory.c:97: bad pmd ffff81004b017528(00007ffffffff756).
 mm/memory.c:97: bad pmd ffff81004b017530(00007ffffffff757).
 mm/memory.c:97: bad pmd ffff81004b017538(00007ffffffff758).
 mm/memory.c:97: bad pmd ffff81004b017540(00007ffffffff759).
 mm/memory.c:97: bad pmd ffff81004b017548(00007ffffffff75a).
 mm/memory.c:97: bad pmd ffff81004b017550(00007ffffffff75b).
 mm/memory.c:97: bad pmd ffff81004b017558(00007ffffffff75c).
 mm/memory.c:97: bad pmd ffff81004b017560(00007ffffffff75d).
 mm/memory.c:97: bad pmd ffff81004b017568(00007ffffffff75e).
 mm/memory.c:97: bad pmd ffff81004b017570(00007ffffffff75f).
 mm/memory.c:97: bad pmd ffff81004b017578(00007ffffffff760).
 mm/memory.c:97: bad pmd ffff81004b017580(00007ffffffff761).
 mm/memory.c:97: bad pmd ffff81004b017588(00007ffffffff762).
 mm/memory.c:97: bad pmd ffff81004b017590(00007ffffffff763).
 mm/memory.c:97: bad pmd ffff81004b017598(00007ffffffff764).
 mm/memory.c:97: bad pmd ffff81004b0175a0(00007ffffffff765).
 mm/memory.c:97: bad pmd ffff81004b0175a8(00007ffffffff766).
 mm/memory.c:97: bad pmd ffff81004b0175b0(00007ffffffff767).
 mm/memory.c:97: bad pmd ffff81004b0175b8(00007ffffffff768).
 mm/memory.c:97: bad pmd ffff81004b0175c0(00007ffffffff769).
 mm/memory.c:97: bad pmd ffff81004b0175c8(00007ffffffff76a).
 mm/memory.c:97: bad pmd ffff81004b0175d0(00007ffffffff76b).
 mm/memory.c:97: bad pmd ffff81004b0175d8(00007ffffffff76c).
 mm/memory.c:97: bad pmd ffff81004b0175e0(00007ffffffff76d).
 mm/memory.c:97: bad pmd ffff81004b0175e8(00007ffffffff76e).
 mm/memory.c:97: bad pmd ffff81004b0175f0(00007ffffffff76f).
 mm/memory.c:97: bad pmd ffff81004b0175f8(00007ffffffff770).
 mm/memory.c:97: bad pmd ffff81004b017600(00007ffffffff771).
 mm/memory.c:97: bad pmd ffff81004b017608(00007ffffffff772).
 mm/memory.c:97: bad pmd ffff81004b017610(00007ffffffff773).
 mm/memory.c:97: bad pmd ffff81004b017618(00007ffffffff774).
 mm/memory.c:97: bad pmd ffff81004b017628(0000000000000010).
 mm/memory.c:97: bad pmd ffff81004b017630(00000000078bfbff).
 mm/memory.c:97: bad pmd ffff81004b017638(0000000000000006).
 mm/memory.c:97: bad pmd ffff81004b017640(0000000000001000).
 mm/memory.c:97: bad pmd ffff81004b017648(0000000000000011).
 mm/memory.c:97: bad pmd ffff81004b017650(0000000000000064).
 mm/memory.c:97: bad pmd ffff81004b017658(0000000000000003).
 mm/memory.c:97: bad pmd ffff81004b017660(0000000000400040).
 mm/memory.c:97: bad pmd ffff81004b017668(0000000000000004).
 mm/memory.c:97: bad pmd ffff81004b017670(0000000000000038).
 mm/memory.c:97: bad pmd ffff81004b017678(0000000000000005).
 mm/memory.c:97: bad pmd ffff81004b017680(0000000000000008).
 mm/memory.c:97: bad pmd ffff81004b017688(0000000000000007).
 mm/memory.c:97: bad pmd ffff81004b017698(0000000000000008).
 mm/memory.c:97: bad pmd ffff81004b0176a8(0000000000000009).
 mm/memory.c:97: bad pmd ffff81004b0176b0(0000000000403840).
 mm/memory.c:97: bad pmd ffff81004b0176b8(000000000000000b).
 mm/memory.c:97: bad pmd ffff81004b0176c0(00000000000001f4).
 mm/memory.c:97: bad pmd ffff81004b0176c8(000000000000000c).
 mm/memory.c:97: bad pmd ffff81004b0176d0(00000000000001f4).
 mm/memory.c:97: bad pmd ffff81004b0176d8(000000000000000d).
 mm/memory.c:97: bad pmd ffff81004b0176e0(00000000000001f4).
 mm/memory.c:97: bad pmd ffff81004b0176e8(000000000000000e).
 mm/memory.c:97: bad pmd ffff81004b0176f0(00000000000001f4).
 mm/memory.c:97: bad pmd ffff81004b0176f8(0000000000000017).
 mm/memory.c:97: bad pmd ffff81004b017708(000000000000000f).
 mm/memory.c:97: bad pmd ffff81004b017710(00007ffffffff734).
 mm/memory.c:97: bad pmd ffff81004b017730(5f36387800000000).
 mm/memory.c:97: bad pmd ffff81004b017738(0000000000003436).
 
 
I've not done a memtest86 run on this (yet), but I'll be very
surprised if this is bad RAM, especially considering other
folks also seem to have hit the same thing when they moved
to 2.6.11.  (My workstation ran 2.6.9/2.6.10 without incident
previously).

http://lkml.org/lkml/2005/3/11/42 for example lists a similar
dump (though obviously differing addresses).
Googling around reveals a bunch of other similar dumps.
 
 		Dave

Re: x86-64 bad pmds in 2.6.11.6

[Andi Kleen]

On Wed, Mar 30, 2005 at 04:44:55PM -0500, Dave Jones wrote:
> [apologies to Andi for getting this twice, I goofed the l-k address
>  the first time]
> 
>  
>  I arrived at the office today to find my workstation had this spew
>  in its dmesg buffer..

Looks like random memory corruption to me.

Can you enable slab debugging etc.?

>  mm/memory.c:97: bad pmd ffff81004b017438(00000038a5500a88).
>  mm/memory.c:97: bad pmd ffff81004b017440(0000000000000003).
>  mm/memory.c:97: bad pmd ffff81004b017448(00007ffffffff73b).
>  mm/memory.c:97: bad pmd ffff81004b017450(00007ffffffff73c).
>  mm/memory.c:97: bad pmd ffff81004b017458(00007ffffffff73d).
>  mm/memory.c:97: bad pmd ffff81004b017468(00007ffffffff73e).
>  mm/memory.c:97: bad pmd ffff81004b017470(00007ffffffff73f).
>  mm/memory.c:97: bad pmd ffff81004b017478(00007ffffffff740).
>  mm/memory.c:97: bad pmd ffff81004b017480(00007ffffffff741).
>  mm/memory.c:97: bad pmd ffff81004b017488(00007ffffffff742).
>  mm/memory.c:97: bad pmd ffff81004b017490(00007ffffffff743).
>  mm/memory.c:97: bad pmd ffff81004b017498(00007ffffffff744).
>  mm/memory.c:97: bad pmd ffff81004b0174a0(00007ffffffff745).
>  mm/memory.c:97: bad pmd ffff81004b0174a8(00007ffffffff746).
>  mm/memory.c:97: bad pmd ffff81004b0174b0(00007ffffffff747).
>  mm/memory.c:97: bad pmd ffff81004b0174b8(00007ffffffff748).
>  mm/memory.c:97: bad pmd ffff81004b0174c0(00007ffffffff749).
>  mm/memory.c:97: bad pmd ffff81004b0174c8(00007ffffffff74a).
>  mm/memory.c:97: bad pmd ffff81004b0174d0(00007ffffffff74b).
>  mm/memory.c:97: bad pmd ffff81004b0174d8(00007ffffffff74c).
>  mm/memory.c:97: bad pmd ffff81004b0174e0(00007ffffffff74d).
>  mm/memory.c:97: bad pmd ffff81004b0174e8(00007ffffffff74e).
>  mm/memory.c:97: bad pmd ffff81004b0174f0(00007ffffffff74f).
>  mm/memory.c:97: bad pmd ffff81004b0174f8(00007ffffffff750).
>  mm/memory.c:97: bad pmd ffff81004b017500(00007ffffffff751).
>  mm/memory.c:97: bad pmd ffff81004b017508(00007ffffffff752).
>  mm/memory.c:97: bad pmd ffff81004b017510(00007ffffffff753).
>  mm/memory.c:97: bad pmd ffff81004b017518(00007ffffffff754).
>  mm/memory.c:97: bad pmd ffff81004b017520(00007ffffffff755).
>  mm/memory.c:97: bad pmd ffff81004b017528(00007ffffffff756).
>  mm/memory.c:97: bad pmd ffff81004b017530(00007ffffffff757).
>  mm/memory.c:97: bad pmd ffff81004b017538(00007ffffffff758).
>  mm/memory.c:97: bad pmd ffff81004b017540(00007ffffffff759).
>  mm/memory.c:97: bad pmd ffff81004b017548(00007ffffffff75a).
>  mm/memory.c:97: bad pmd ffff81004b017550(00007ffffffff75b).
>  mm/memory.c:97: bad pmd ffff81004b017558(00007ffffffff75c).
>  mm/memory.c:97: bad pmd ffff81004b017560(00007ffffffff75d).
>  mm/memory.c:97: bad pmd ffff81004b017568(00007ffffffff75e).
>  mm/memory.c:97: bad pmd ffff81004b017570(00007ffffffff75f).
>  mm/memory.c:97: bad pmd ffff81004b017578(00007ffffffff760).
>  mm/memory.c:97: bad pmd ffff81004b017580(00007ffffffff761).
>  mm/memory.c:97: bad pmd ffff81004b017588(00007ffffffff762).
>  mm/memory.c:97: bad pmd ffff81004b017590(00007ffffffff763).
>  mm/memory.c:97: bad pmd ffff81004b017598(00007ffffffff764).
>  mm/memory.c:97: bad pmd ffff81004b0175a0(00007ffffffff765).
>  mm/memory.c:97: bad pmd ffff81004b0175a8(00007ffffffff766).
>  mm/memory.c:97: bad pmd ffff81004b0175b0(00007ffffffff767).
>  mm/memory.c:97: bad pmd ffff81004b0175b8(00007ffffffff768).
>  mm/memory.c:97: bad pmd ffff81004b0175c0(00007ffffffff769).
>  mm/memory.c:97: bad pmd ffff81004b0175c8(00007ffffffff76a).
>  mm/memory.c:97: bad pmd ffff81004b0175d0(00007ffffffff76b).
>  mm/memory.c:97: bad pmd ffff81004b0175d8(00007ffffffff76c).
>  mm/memory.c:97: bad pmd ffff81004b0175e0(00007ffffffff76d).
>  mm/memory.c:97: bad pmd ffff81004b0175e8(00007ffffffff76e).
>  mm/memory.c:97: bad pmd ffff81004b0175f0(00007ffffffff76f).
>  mm/memory.c:97: bad pmd ffff81004b0175f8(00007ffffffff770).
>  mm/memory.c:97: bad pmd ffff81004b017600(00007ffffffff771).
>  mm/memory.c:97: bad pmd ffff81004b017608(00007ffffffff772).
>  mm/memory.c:97: bad pmd ffff81004b017610(00007ffffffff773).
>  mm/memory.c:97: bad pmd ffff81004b017618(00007ffffffff774).
>  mm/memory.c:97: bad pmd ffff81004b017628(0000000000000010).
>  mm/memory.c:97: bad pmd ffff81004b017630(00000000078bfbff).
>  mm/memory.c:97: bad pmd ffff81004b017638(0000000000000006).
>  mm/memory.c:97: bad pmd ffff81004b017640(0000000000001000).
>  mm/memory.c:97: bad pmd ffff81004b017648(0000000000000011).
>  mm/memory.c:97: bad pmd ffff81004b017650(0000000000000064).
>  mm/memory.c:97: bad pmd ffff81004b017658(0000000000000003).
>  mm/memory.c:97: bad pmd ffff81004b017660(0000000000400040).
>  mm/memory.c:97: bad pmd ffff81004b017668(0000000000000004).
>  mm/memory.c:97: bad pmd ffff81004b017670(0000000000000038).
>  mm/memory.c:97: bad pmd ffff81004b017678(0000000000000005).
>  mm/memory.c:97: bad pmd ffff81004b017680(0000000000000008).
>  mm/memory.c:97: bad pmd ffff81004b017688(0000000000000007).
>  mm/memory.c:97: bad pmd ffff81004b017698(0000000000000008).
>  mm/memory.c:97: bad pmd ffff81004b0176a8(0000000000000009).
>  mm/memory.c:97: bad pmd ffff81004b0176b0(0000000000403840).
>  mm/memory.c:97: bad pmd ffff81004b0176b8(000000000000000b).
>  mm/memory.c:97: bad pmd ffff81004b0176c0(00000000000001f4).
>  mm/memory.c:97: bad pmd ffff81004b0176c8(000000000000000c).
>  mm/memory.c:97: bad pmd ffff81004b0176d0(00000000000001f4).
>  mm/memory.c:97: bad pmd ffff81004b0176d8(000000000000000d).
>  mm/memory.c:97: bad pmd ffff81004b0176e0(00000000000001f4).
>  mm/memory.c:97: bad pmd ffff81004b0176e8(000000000000000e).
>  mm/memory.c:97: bad pmd ffff81004b0176f0(00000000000001f4).
>  mm/memory.c:97: bad pmd ffff81004b0176f8(0000000000000017).
>  mm/memory.c:97: bad pmd ffff81004b017708(000000000000000f).
>  mm/memory.c:97: bad pmd ffff81004b017710(00007ffffffff734).
>  mm/memory.c:97: bad pmd ffff81004b017730(5f36387800000000).
>  mm/memory.c:97: bad pmd ffff81004b017738(0000000000003436).
>  
>  
> I've not done a memtest86 run on this (yet), but I'll be very
> surprised if this is bad RAM, especially considering other
> folks also seem to have hit the same thing when they moved
> to 2.6.11.  (My workstation ran 2.6.9/2.6.10 without incident
> previously).
> 
> http://lkml.org/lkml/2005/3/11/42 for example lists a similar
> dump (though obviously differing addresses).
> Googling around reveals a bunch of other similar dumps.

Yes I saw them, but I supposed it is some driver going bad.
If you want you can collect hardware data and see if there is
a common driver.

-Andi

Re: x86-64 bad pmds in 2.6.11.6

[Dave Jones]

On Thu, Mar 31, 2005 at 12:41:17PM +0200, Andi Kleen wrote:
 > On Wed, Mar 30, 2005 at 04:44:55PM -0500, Dave Jones wrote:
 > > [apologies to Andi for getting this twice, I goofed the l-k address
 > >  the first time]
 > > 
 > >  
 > >  I arrived at the office today to find my workstation had this spew
 > >  in its dmesg buffer..
 > 
 > Looks like random memory corruption to me.
 > 
 > Can you enable slab debugging etc.?

SLAB_DEBUG=y.  Nothing in the logs.

 > Yes I saw them, but I supposed it is some driver going bad.
 > If you want you can collect hardware data and see if there is
 > a common driver.

There's quite a bit in this box 

00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02)
00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
00:07.5 Multimedia audio controller: Advanced Micro Devices [AMD] AMD-8111 AC97 Audio (rev 03)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
02:07.0 USB Controller: NEC Corporation USB (rev 41)
02:07.1 USB Controller: NEC Corporation USB (rev 41)
02:07.2 USB Controller: NEC Corporation USB 2.0 (rev 02)
02:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter (rev 01)
02:08.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter (rev 01)
02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703X Gigabit Ethernet (rev 02)
03:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
03:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
03:0a.0 Unknown mass storage controller: Triones Technologies, Inc. HPT366/368/370/370A/372 (rev 03)
03:0b.0 Unknown mass storage controller: Silicon Image, Inc. (formerly CMD Technology Inc) SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
03:0c.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
04:00.0 Host bridge: Advanced Micro Devices [AMD] AMD-8151 System Controller (rev 13)
04:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8151 AGP Bridge (rev 13)
05:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G550 AGP (rev 01)

The SATA & SCSI controllers have no disks attached.  Firewire can be ignored (theres
no actual connector even for it on the board). The various USB controllers
are mostly unused. Only one of them is USB2.0, so that sees occasional
usb-storage use. Not noticed anything going bad there though.

		Dave

Re: x86-64 bad pmds in 2.6.11.6

[Sergey S. Kostyliov]

On Friday 01 April 2005 01:52, Dave Jones wrote:
> On Thu, Mar 31, 2005 at 12:41:17PM +0200, Andi Kleen wrote:
>  > On Wed, Mar 30, 2005 at 04:44:55PM -0500, Dave Jones wrote:
>  > > [apologies to Andi for getting this twice, I goofed the l-k address
>  > >  the first time]
>  > > 
>  > >  
>  > >  I arrived at the office today to find my workstation had this spew
>  > >  in its dmesg buffer..
>  > 
>  > Looks like random memory corruption to me.
>  > 
>  > Can you enable slab debugging etc.?
> 
> SLAB_DEBUG=y.  Nothing in the logs.
> 
>  > Yes I saw them, but I supposed it is some driver going bad.
>  > If you want you can collect hardware data and see if there is
>  > a common driver.
> 
> There's quite a bit in this box 
> 
> 00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02)
> 00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
> 00:07.5 Multimedia audio controller: Advanced Micro Devices [AMD] AMD-8111 AC97 Audio (rev 03)
> 00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
> 00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
> 00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
> 00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
> 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
> 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
> 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
> 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
> 00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
> 00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
> 00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
> 00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
> 02:07.0 USB Controller: NEC Corporation USB (rev 41)
> 02:07.1 USB Controller: NEC Corporation USB (rev 41)
> 02:07.2 USB Controller: NEC Corporation USB 2.0 (rev 02)
> 02:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter (rev 01)
> 02:08.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter (rev 01)
> 02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703X Gigabit Ethernet (rev 02)
> 03:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
> 03:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
> 03:0a.0 Unknown mass storage controller: Triones Technologies, Inc. HPT366/368/370/370A/372 (rev 03)
> 03:0b.0 Unknown mass storage controller: Silicon Image, Inc. (formerly CMD Technology Inc) SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
> 03:0c.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
> 04:00.0 Host bridge: Advanced Micro Devices [AMD] AMD-8151 System Controller (rev 13)
> 04:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8151 AGP Bridge (rev 13)
> 05:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G550 AGP (rev 01)
> 
> The SATA & SCSI controllers have no disks attached.  Firewire can be ignored (theres
> no actual connector even for it on the board). The various USB controllers
> are mostly unused. Only one of them is USB2.0, so that sees occasional
> usb-storage use. Not noticed anything going bad there though.
> 
> 		Dave

And here is my box (looks like there is no many hardware drivers
in common).

rathamahata@lights rathamahata $ /sbin/lspci
0000:00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07)
0000:00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05)
0000:00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03)
0000:00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
0000:00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
0000:00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
0000:00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
0000:01:01.0 PCI bridge: IBM PCI-X to PCI-X Bridge (rev 02)
0000:02:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID (rev 02)
0000:03:03.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller
0000:03:04.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller
0000:04:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
0000:04:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
0000:04:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
rathamahata@lights rathamahata $

e1000 is handled by by intel's e1000 driver


usb is not compiled in 
rathamahata@lights linux-2.6.11 $ grep CONFIG_USB .config
# CONFIG_USB is not set
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
# CONFIG_USB_GADGET is not set
rathamahata@lights linux-2.6.11 $

-- 
Sergey S. Kostyliov <rathamahata@ehouse.ru>
Jabber ID: rathamahata@jabber.org

Re: x86-64 bad pmds in 2.6.11.6

[Dave Jones]

On Thu, Mar 31, 2005 at 12:41:17PM +0200, Andi Kleen wrote:
 > On Wed, Mar 30, 2005 at 04:44:55PM -0500, Dave Jones wrote:
 > > [apologies to Andi for getting this twice, I goofed the l-k address
 > >  the first time]
 > > 
 > >  
 > >  I arrived at the office today to find my workstation had this spew
 > >  in its dmesg buffer..
 > 
 > Looks like random memory corruption to me.
 > 
 > Can you enable slab debugging etc.?
 > 
 > >  mm/memory.c:97: bad pmd ffff81004b017438(00000038a5500a88).
 > >  mm/memory.c:97: bad pmd ffff81004b017440(0000000000000003).
 > >  mm/memory.c:97: bad pmd ffff81004b017448(00007ffffffff73b).
 > >  mm/memory.c:97: bad pmd ffff81004b017450(00007ffffffff73c).
 > > etc..

I realised today that this happens every time X starts up for
the first time.   I did some experiments, and found that with 2.6.12rc1
it's gone. Either it got fixed accidentally, or its hidden now
by one of the many changes in 4-level patches.

I'll try and narrow this down a little more tomorrow, to see if I
can pinpoint the exact -bk snapshot (may be tricky given they were
broken for a while), as it'd be good to get this fixed in 2.6.11.x
if .12 isn't going to show up any time soon.

		Dave

Re: x86-64 bad pmds in 2.6.11.6

[Andi Kleen]

> I realised today that this happens every time X starts up for
> the first time.   I did some experiments, and found that with 2.6.12rc1
> it's gone. Either it got fixed accidentally, or its hidden now
> by one of the many changes in 4-level patches.
> 
> I'll try and narrow this down a little more tomorrow, to see if I
> can pinpoint the exact -bk snapshot (may be tricky given they were
> broken for a while), as it'd be good to get this fixed in 2.6.11.x
> if .12 isn't going to show up any time soon.

Can you supply a strace of the /dev/mem, /dev/kmem accesses of 
your X server? (including the mmaps or read/writes if available)

My X server doesn't seem to cause that.

-Andi

Re: x86-64 bad pmds in 2.6.11.6

[Hugh Dickins]

On Thu, 7 Apr 2005, Andi Kleen wrote:
> Dave Jones wrote:
> > I realised today that this happens every time X starts up for
> > the first time.   I did some experiments, and found that with 2.6.12rc1
> > it's gone. Either it got fixed accidentally, or its hidden now
> > by one of the many changes in 4-level patches.
> > 
> > I'll try and narrow this down a little more tomorrow, to see if I
> > can pinpoint the exact -bk snapshot (may be tricky given they were
> > broken for a while), as it'd be good to get this fixed in 2.6.11.x
> > if .12 isn't going to show up any time soon.
> 
> Can you supply a strace of the /dev/mem, /dev/kmem accesses of 
> your X server? (including the mmaps or read/writes if available)
> 
> My X server doesn't seem to cause that.

I can't explain why it should appear fixed in 2.6.12-rc1 (probably
other complicating factors at work), but I do believe you've fixed
this in 2.6.12-rc2, and the patch which should go into -stable is
your load_cr3 patch below, which Linus took from Andrew on 28 March.

I say this because I was intrigued by the resemblance between Sergey's
and Dave's corruptions, and spent a while trying to work out where they
come from.  The giveaway is the little ASCII string they share at the
end (seen also in Clem's extract)

 mm/memory.c:97: bad pmd ffff81004b017730(5f36387800000000).
 mm/memory.c:97: bad pmd ffff81004b017738(0000000000003436).

That says "x86_64", and a grep for that as a string shows ELF_PLATFORM,
and a grep for that shows create_elf_tables in fs/binfmt_elf.c.  _All_
this pmd corruption (except for the first line, presumably pushing a
user address on stack) originates from create_elf_tables (the neatly
ascending stack addresses being the argv and envp pointers, incrementing
by 1 because only a NUL-string is found for each, the real strings being
off elsewhere in the intended new stack page, not in this pmd page).

It looks very much as if the mm being created has for pmd a page
which was used for user stack in the outgoing mm; but somehow exec's
exit_mmap TLB flushing hasn't taken effect.  I only now noticed this
patch where you fix just such an issue.

Hugh

From: "Andi Kleen" <ak@suse.de>

Always reload CR3 completely when a lazy MM thread drops a MM.  This avoids
keeping stale mappings around in the TLB that could be run into by the CPU by
itself (e.g.  during prefetches).

Signed-off-by: Andi Kleen <ak@suse.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
---

 25-akpm/arch/x86_64/kernel/smp.c         |    3 ++-
 25-akpm/include/asm-x86_64/mmu_context.h |   10 ++++++++--
 2 files changed, 10 insertions(+), 3 deletions(-)

diff -puN arch/x86_64/kernel/smp.c~x86_64-always-reload-cr3-completely-when-a-lazy-mm arch/x86_64/kernel/smp.c
--- 25/arch/x86_64/kernel/smp.c~x86_64-always-reload-cr3-completely-when-a-lazy-mm	Wed Mar 23 15:38:58 2005
+++ 25-akpm/arch/x86_64/kernel/smp.c	Wed Mar 23 15:38:58 2005
@@ -25,6 +25,7 @@
 #include <asm/pgalloc.h>
 #include <asm/tlbflush.h>
 #include <asm/mach_apic.h>
+#include <asm/mmu_context.h>
 #include <asm/proto.h>
 
 /*
@@ -52,7 +53,7 @@ static inline void leave_mm (unsigned lo
 	if (read_pda(mmu_state) == TLBSTATE_OK)
 		BUG();
 	clear_bit(cpu, &read_pda(active_mm)->cpu_vm_mask);
-	__flush_tlb();
+	load_cr3(swapper_pg_dir);
 }
 
 /*
diff -puN include/asm-x86_64/mmu_context.h~x86_64-always-reload-cr3-completely-when-a-lazy-mm include/asm-x86_64/mmu_context.h
--- 25/include/asm-x86_64/mmu_context.h~x86_64-always-reload-cr3-completely-when-a-lazy-mm	Wed Mar 23 15:38:58 2005
+++ 25-akpm/include/asm-x86_64/mmu_context.h	Wed Mar 23 15:38:58 2005
@@ -28,6 +28,11 @@ static inline void enter_lazy_tlb(struct
 }
 #endif
 
+static inline void load_cr3(pgd_t *pgd)
+{
+	asm volatile("movq %0,%%cr3" :: "r" (__pa(pgd)) : "memory");
+}
+
 static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, 
 			     struct task_struct *tsk)
 {
@@ -40,7 +45,8 @@ static inline void switch_mm(struct mm_s
 		write_pda(active_mm, next);
 #endif
 		set_bit(cpu, &next->cpu_vm_mask);
-		asm volatile("movq %0,%%cr3" :: "r" (__pa(next->pgd)) : "memory");
+		load_cr3(next->pgd);
+
 		if (unlikely(next->context.ldt != prev->context.ldt)) 
 			load_LDT_nolock(&next->context, cpu);
 	}
@@ -54,7 +60,7 @@ static inline void switch_mm(struct mm_s
 			 * tlb flush IPI delivery. We must reload CR3
 			 * to make sure to use no freed page tables.
 			 */
-			asm volatile("movq %0,%%cr3" :: "r" (__pa(next->pgd)) : "memory");
+			load_cr3(next->pgd);
 			load_LDT_nolock(&next->context, cpu);
 		}
 	}

Re: x86-64 bad pmds in 2.6.11.6

[Andi Kleen]

> It looks very much as if the mm being created has for pmd a page
> which was used for user stack in the outgoing mm; but somehow exec's
> exit_mmap TLB flushing hasn't taken effect.  I only now noticed this
> patch where you fix just such an issue.

Thanks for the analysis. However I doubt the load_cr3 patch can fix
it. All it does is to stop the CPU from prefetching mappings (which
can cause different problem). But the Linux code who does bad pmd checks
never looks at CR3 anyways, it always uses the current->mm. If
bad pmd sees a bad page it must be still in the page tables of the MM,
not a stable TLB entry.

It must be something else. Somehow we get a freed page into
the page table hierarchy. After the initial 4level implementation
I did not do many changes there, my suspection would be rather
on the recent memory.c changes.

-Andi

Re: x86-64 bad pmds in 2.6.11.6

[Hugh Dickins]

On Thu, 14 Apr 2005, Andi Kleen wrote:
> 
> Thanks for the analysis. However I doubt the load_cr3 patch can fix
> it. All it does is to stop the CPU from prefetching mappings (which
> can cause different problem).

I thought that the leave_mm code (before your patch) flushes the TLB, but
restores cr3 to the mm, while removing that cpu from the mm's cpu_vm_mask.

So any speculation, not just prefetching, on that cpu is in danger of
bringing address translations according to that mm back into the TLB.

But when the mm is torn down in exit_mmap, there's no longer any record
that the TLB on that cpu needs flushing, so stale translations remain.

As a rule, we always flush TLB _after_ invalidating, not just before,
for this kind of reason.

My paranoia of speculation may be excessive: I _think_ what I outline
above is a real possibility on Intel, but you and others know AMD much
better than I (and the reports I've seen are on AMD64, not EM64T).

> But the Linux code who does bad pmd checks
> never looks at CR3 anyways, it always uses the current->mm. If
> bad pmd sees a bad page it must be still in the page tables of the MM,
> not a stable TLB entry.

Sure, the "mm/memory.c:97: bad pmd" messages are coming from
clear_pmd_range, when the corrupted task exits later (but probably
not much later, since its user stack is oddly distributed across
two different pages: some mentioned SIGSEGVs I think).

The pmd really is bad, but it got to be bad because it had stack data
written into it by create_elf_tables, when the TLB mistakenly thought
it already knew what physical page 0x00007ffffffff000 was mapped to
(prior kernel accesses to that user stack are not by user address).

Hugh

Re: x86-64 bad pmds in 2.6.11.6

[Andi Kleen]

On Thu, Apr 14, 2005 at 06:34:58PM +0100, Hugh Dickins wrote:
> On Thu, 14 Apr 2005, Andi Kleen wrote:
> > 
> > Thanks for the analysis. However I doubt the load_cr3 patch can fix
> > it. All it does is to stop the CPU from prefetching mappings (which
> > can cause different problem).
> 
> I thought that the leave_mm code (before your patch) flushes the TLB, but
> restores cr3 to the mm, while removing that cpu from the mm's cpu_vm_mask.
> 
> So any speculation, not just prefetching, on that cpu is in danger of
> bringing address translations according to that mm back into the TLB.
> 
> But when the mm is torn down in exit_mmap, there's no longer any record
> that the TLB on that cpu needs flushing, so stale translations remain.
> 
> As a rule, we always flush TLB _after_ invalidating, not just before,
> for this kind of reason.

Yes this is all true. In fact I have several bug fixes for problems
in this area.

But this all cannot explain corruptions comming from the kernel, 
you tend to only see problems with the CPU prefetching something.

Note that with the cr3 reload you end up with init_mm, which
is not any useful mm. So even if there was a store from the kernel
into a stale mapping it would cause -EFAULT now.  But that is
not happening.

> 
> My paranoia of speculation may be excessive: I _think_ what I outline
> above is a real possibility on Intel, but you and others know AMD much
> better than I (and the reports I've seen are on AMD64, not EM64T).

It is not both on Intel and AMD :) These CPUs do a lot of prefetching
behind your back, any stale mappings at any time in the TLB eventually
cause problems. But other ones than this.


> Sure, the "mm/memory.c:97: bad pmd" messages are coming from
> clear_pmd_range, when the corrupted task exits later (but probably
> not much later, since its user stack is oddly distributed across
> two different pages: some mentioned SIGSEGVs I think).
> 
> The pmd really is bad, but it got to be bad because it had stack data
> written into it by create_elf_tables, when the TLB mistakenly thought
> it already knew what physical page 0x00007ffffffff000 was mapped to
> (prior kernel accesses to that user stack are not by user address).

What I meant is that the overwriting must be from Linux code
acting in the direct mapping, not due stale TLBs for addresses < __PAGE_OFFSET.

I will take a closer look at the rc1/rc2 patches later this evening
and see if I can spot something. Can only report back tomorrow though.

-Andi

Re: x86-64 bad pmds in 2.6.11.6 II

[Andi Kleen]

> I will take a closer look at the rc1/rc2 patches later this evening
> and see if I can spot something. Can only report back tomorrow though.

Actually itt started in .11 already - sigh - on rereading the thread.
That will make the code audit harder :/

-Andi

Re: x86-64 bad pmds in 2.6.11.6 II

[Chris Wright]

* Andi Kleen (ak@suse.de) wrote:
> > I will take a closer look at the rc1/rc2 patches later this evening
> > and see if I can spot something. Can only report back tomorrow though.
> 
> Actually itt started in .11 already - sigh - on rereading the thread.
> That will make the code audit harder :/

Yes, I've seen it in .11 and earlier kernels.  Happen to have same
"x86_64" string on my bad pmd dumps, but can't reproduce it at all.
So, for now, I can hold off on adding the reload cr3 patch to -stable
unless you think it should be there anyway.

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: x86-64 bad pmds in 2.6.11.6 II

[Andi Kleen]

On Thu, Apr 14, 2005 at 11:27:12AM -0700, Chris Wright wrote:
> * Andi Kleen (ak@suse.de) wrote:
> > > I will take a closer look at the rc1/rc2 patches later this evening
> > > and see if I can spot something. Can only report back tomorrow though.
> > 
> > Actually itt started in .11 already - sigh - on rereading the thread.
> > That will make the code audit harder :/
> 
> Yes, I've seen it in .11 and earlier kernels.  Happen to have same
> "x86_64" string on my bad pmd dumps, but can't reproduce it at all.
> So, for now, I can hold off on adding the reload cr3 patch to -stable
> unless you think it should be there anyway.

It is a bug fix (actually there is another related patch that fixes
a similar bug), but we lived with the problems for years so I guess
they can wait for .12. 

If there was a fix for the bad pmd problem it might be a candidate
for stable, but so far we dont know what causes it yet.

-Andi

Re: x86-64 bad pmds in 2.6.11.6 II

[Chris Wright]

* Andi Kleen (ak@suse.de) wrote:
> On Thu, Apr 14, 2005 at 11:27:12AM -0700, Chris Wright wrote:
> > Yes, I've seen it in .11 and earlier kernels.  Happen to have same
> > "x86_64" string on my bad pmd dumps, but can't reproduce it at all.
> > So, for now, I can hold off on adding the reload cr3 patch to -stable
> > unless you think it should be there anyway.
> 
> It is a bug fix (actually there is another related patch that fixes
> a similar bug), but we lived with the problems for years so I guess
> they can wait for .12. 

Sounds good.

> If there was a fix for the bad pmd problem it might be a candidate
> for stable, but so far we dont know what causes it yet.

If I figure a way to trigger here, I'll report back.

thanks,
-chris

Re: x86-64 bad pmds in 2.6.11.6 II

[Hugh Dickins]

On Fri, 15 Apr 2005, Chris Wright wrote:
> * Andi Kleen (ak@suse.de) wrote:
> > On Thu, Apr 14, 2005 at 11:27:12AM -0700, Chris Wright wrote:
> > > Yes, I've seen it in .11 and earlier kernels.  Happen to have same
> > > "x86_64" string on my bad pmd dumps, but can't reproduce it at all.
> > > So, for now, I can hold off on adding the reload cr3 patch to -stable
> > > unless you think it should be there anyway.
> > 
> > It is a bug fix (actually there is another related patch that fixes
> > a similar bug), but we lived with the problems for years so I guess
> > they can wait for .12. 
> 
> Sounds good.

I must confess, with all due respect to Andi, that I don't understand his
dismissal of the possibility that load_cr3 in leave_mm might be the fix
(to create_elf_tables writing user stack data into the pmd).

My belief is that leaving any opening for unpredictable speculations to
pull stale translations into the TLB, is a recipe for strange trouble
down the line when those translations may get used in actuality.

I'd been hoping Andi would come to see it my way overnight!
since I'm clearly not up to arguing the case persuasively.

But I certainly don't expect Chris to add an unjustified patch to -stable.

> > If there was a fix for the bad pmd problem it might be a candidate
> > for stable, but so far we dont know what causes it yet.
> 
> If I figure a way to trigger here, I'll report back.

Dave, earlier on you were quite able to reproduce the problem on 2.6.11,
finding it happened the first time you ran X.  Do you have any time to
reverify that, then try to reproduce with the load_cr3 in leave_mm patch?

But please don't waste your time on this unless you think it's plausible.

Thanks,
Hugh

Re: x86-64 bad pmds in 2.6.11.6 II

[Dave Jones]

On Fri, Apr 15, 2005 at 06:58:20PM +0100, Hugh Dickins wrote:

 > > > If there was a fix for the bad pmd problem it might be a candidate
 > > > for stable, but so far we dont know what causes it yet.
 > > If I figure a way to trigger here, I'll report back.
 > 
 > Dave, earlier on you were quite able to reproduce the problem on 2.6.11,
 > finding it happened the first time you ran X.  Do you have any time to
 > reverify that, then try to reproduce with the load_cr3 in leave_mm patch?
 > 
 > But please don't waste your time on this unless you think it's plausible.

I used to be able to reproduce it 100% by doing this on an vanilla
upstream kernel. Then it changed behaviour so I only saw it happening
on the Fedora kernel.  For the latest Fedora update kernel I backported
this change..
- x86_64: Only free PMDs and PUDs after other CPUs have been flushed
as a 'try it and see'.  At first I thought it killed the bug, but
a day or so later, it started doing it again.

In the Fedora kernel we have a patch which restricts /dev/mem reading,
so I got suspicious about this interacting with any of the changes
that had happened to drivers/char/mem.c
Out of curiousity, I backported the 3-4 patches from .12rc to
the Fedora .11 kernel, and haven't seen the problem since.

The bizarre thing is I can't explain why any of those patches would
make such a difference.  Given the bug seems to be coming and going
for me, its possible its just masked the problem.

		Dave

Re: x86-64 bad pmds in 2.6.11.6 II

[Andi Kleen]

On Fri, Apr 15, 2005 at 06:58:20PM +0100, Hugh Dickins wrote:
> On Fri, 15 Apr 2005, Chris Wright wrote:
> > * Andi Kleen (ak@suse.de) wrote:
> > > On Thu, Apr 14, 2005 at 11:27:12AM -0700, Chris Wright wrote:
> > > > Yes, I've seen it in .11 and earlier kernels.  Happen to have same
> > > > "x86_64" string on my bad pmd dumps, but can't reproduce it at all.
> > > > So, for now, I can hold off on adding the reload cr3 patch to -stable
> > > > unless you think it should be there anyway.
> > > 
> > > It is a bug fix (actually there is another related patch that fixes
> > > a similar bug), but we lived with the problems for years so I guess
> > > they can wait for .12. 
> > 
> > Sounds good.
> 
> I must confess, with all due respect to Andi, that I don't understand his
> dismissal of the possibility that load_cr3 in leave_mm might be the fix
> (to create_elf_tables writing user stack data into the pmd).

Sorry for the late answer.

Ok, lets try again. The hole fixed by this patch only covers
the case of an kernel thread with lazy mm doing some memory access
(or more likely the CPU doing a prefetch there). But ELF loading
never happens in lazy mm kernel threads.AFAIK in a "real" process
the TLB is always fully consistent.

Does that explanation satisfy you? 

I agree that my earlier one was a bit dubious because I argued about
the direct mapping, but the argv setup actually uses user addresses.
But I still think it must be something else.

-Andi

Re: x86-64 bad pmds in 2.6.11.6 II

[Hugh Dickins]

On Tue, 19 Apr 2005, Andi Kleen wrote:
> On Fri, Apr 15, 2005 at 06:58:20PM +0100, Hugh Dickins wrote:
> > 
> > I must confess, with all due respect to Andi, that I don't understand his
> > dismissal of the possibility that load_cr3 in leave_mm might be the fix
> > (to create_elf_tables writing user stack data into the pmd).
> 
> Sorry for the late answer.

Not at all.  I didn't expect you to persist in trying to persuade me,
thank you for doing so, and I apologize for taking your time on this.

> Ok, lets try again. The hole fixed by this patch only covers
> the case of an kernel thread with lazy mm doing some memory access
> (or more likely the CPU doing a prefetch there). But ELF loading
> never happens in lazy mm kernel threads.AFAIK in a "real" process
> the TLB is always fully consistent.
> 
> Does that explanation satisfy you? 

It does.  Well, I needed to restudy exec_mmap and switch_mm in detail,
and having done so, I agree that the only way you can get through
exec_mmap's activate_mm without fully flushing the cpu's TLB, is if
the active_mm matches the newly allocated mm (itself impossible since
there's a reference on the active_mm), and the cpu bit is still set
in cpu_vm_mask - precisely not the case if we went through leave_mm.
Yet I was claiming your leave_mm fix could flush TLB for exec_mmap
where it wasn't already done.

Sorry for letting the neatness of my pmd/stack story blind me
to its impossibility, and for wasting your time.

Hugh

Debugging patch was Re: x86-64 bad pmds in 2.6.11.6 II

[Andi Kleen]

Can people who can reproduce the x86-64 2.6.11 pmd bad  problem please apply
the following patch and see (a) if it can be still reprocuded with it 
and send the output generated. Also a strace of the program that showed
it (pid and name of it should be dumped) would be useful if not too big.

After staring some time at the code I cant find the problem, but 
I somehow suspect it has to do with early page table frees. That is
why they were disabled. This should not cause any memory leaks,
the page tables will be always freed at process exit, so it is
safe to apply even for production machines.

Thanks,

-Andi


diff -u linux-2.6.11/mm/memory.c-o linux-2.6.11/mm/memory.c
--- linux-2.6.11/mm/memory.c-o	2005-03-02 08:38:08.000000000 +0100
+++ linux-2.6.11/mm/memory.c	2005-04-22 19:32:30.305402456 +0200
@@ -94,6 +94,7 @@
 	if (pmd_none(*pmd))
 		return;
 	if (unlikely(pmd_bad(*pmd))) {
+		printk("%s:%d: ", current->comm, current->pid);
 		pmd_ERROR(*pmd);
 		pmd_clear(pmd);
 		return;
diff -u linux-2.6.11/mm/mmap.c-o linux-2.6.11/mm/mmap.c
--- linux-2.6.11/mm/mmap.c-o	2005-03-02 08:38:12.000000000 +0100
+++ linux-2.6.11/mm/mmap.c	2005-04-22 19:33:10.354580428 +0200
@@ -1645,11 +1645,13 @@
 		return;
 	if (first < FIRST_USER_PGD_NR * PGDIR_SIZE)
 		first = FIRST_USER_PGD_NR * PGDIR_SIZE;
+#if 0
 	/* No point trying to free anything if we're in the same pte page */
 	if ((first & PMD_MASK) < (last & PMD_MASK)) {
 		clear_page_range(tlb, first, last);
 		flush_tlb_pgtables(mm, first, last);
 	}
+#endif
 }
 
 /* Normal function to fix up a mapping

New debugging patch was Re: x86-64 bad pmds in 2.6.11.6 II

[Andi Kleen]

Could someone who reproduces this problem apply the following
patch and see if the WARN_ON triggers?


diff -u linux-2.6.11/mm/memory.c-o linux-2.6.11/mm/memory.c
--- linux-2.6.11/mm/memory.c-o	2005-03-02 08:38:08.000000000 +0100
+++ linux-2.6.11/mm/memory.c	2005-04-27 15:48:19.777104735 +0200
@@ -94,6 +94,7 @@
 	if (pmd_none(*pmd))
 		return;
 	if (unlikely(pmd_bad(*pmd))) {
+		printk("%s:%d: ", current->comm, current->pid);
 		pmd_ERROR(*pmd);
 		pmd_clear(pmd);
 		return;
@@ -113,6 +114,7 @@
 	unsigned long addr = start, next;
 	pmd_t *pmd, *__pmd;
 
+	WARN_ON(start == end);
 	if (pud_none(*pud))
 		return;
 	if (unlikely(pud_bad(*pud))) {

Re: New debugging patch was Re: x86-64 bad pmds in 2.6.11.6 II

[Dave Jones]

On Wed, Apr 27, 2005 at 04:23:44PM +0200, Andi Kleen wrote:
 > 
 > Could someone who reproduces this problem apply the following
 > patch and see if the WARN_ON triggers?
 > 
 > 
 > diff -u linux-2.6.11/mm/memory.c-o linux-2.6.11/mm/memory.c
 > --- linux-2.6.11/mm/memory.c-o	2005-03-02 08:38:08.000000000 +0100
 > +++ linux-2.6.11/mm/memory.c	2005-04-27 15:48:19.777104735 +0200
 > @@ -94,6 +94,7 @@
 >  	if (pmd_none(*pmd))
 >  		return;
 >  	if (unlikely(pmd_bad(*pmd))) {
 > +		printk("%s:%d: ", current->comm, current->pid);
 >  		pmd_ERROR(*pmd);
 >  		pmd_clear(pmd);
 >  		return;
 > @@ -113,6 +114,7 @@
 >  	unsigned long addr = start, next;
 >  	pmd_t *pmd, *__pmd;
 >  
 > +	WARN_ON(start == end);
 >  	if (pud_none(*pud))
 >  		return;
 >  	if (unlikely(pud_bad(*pud))) {

I'm up to my eyeballs in other stuff right now, so probably won't
get a chance to test this personally. I'll add it to the Fedora
testing rpm however, as 1-2 users are also hitting it.

I'll let you know if I hear anything back.

		Dave

Re: New debugging patch was Re: x86-64 bad pmds in 2.6.11.6 II

[Hans Kristian Rosbach]

On Wed, 2005-04-27 at 19:37, Dave Jones wrote: 
> On Wed, Apr 27, 2005 at 04:23:44PM +0200, Andi Kleen wrote:
>  > 
>  > Could someone who reproduces this problem apply the following
>  > patch and see if the WARN_ON triggers?
>  > 
*snip* 
> I'm up to my eyeballs in other stuff right now, so probably won't
> get a chance to test this personally. I'll add it to the Fedora
> testing rpm however, as 1-2 users are also hitting it.
> 
> I'll let you know if I hear anything back.
> 
> 		Dave

I'm seeing this problem on 2.6.11-1.14_FC3smp, a part of dmesg is
supplied below. (these messages have flooded the buffer).
Kernel was updated from kernel-smp-2.6.9-1.667 yesterday, and no
similar messages was in dmesg at that time after 8 days uptime.

Fedora Core 3 w/latest updates atm
Running Mysql, exim, http + perl, clamav, spamassasin, courier.

Tyan motherboard, dual AMD 246, 4x512MB ram (one on each bus),
two 3ware 9xx, dual broadcom nic. (lspci output below)

I have not tried the testing kernel you suggested, but will see what
time will allow today. We already have a scheduled reboot on it so
maybe I'll just sneak in a kernel update again.

More info available on request.

-HK


# uptime
12:57:50 up  4:03,  1 user,  load average: 1.85, 1.87, 1.74

# lspci
00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07)
00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05)
00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev
03)
00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02)
00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge
(rev 12)
00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge
(rev 12)
00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
01:03.0 RAID bus controller: 3ware Inc 3ware Inc 3ware 9xxx-series
SATA-RAID
02:03.0 RAID bus controller: 3ware Inc 3ware Inc 3ware 9xxx-series
SATA-RAID
02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
Gigabit Ethernet (rev 03)
02:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
Gigabit Ethernet (rev 03)
03:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
03:08.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100]
(rev 10)


# lsmod
Module                  Size  Used by
md5                     5953  1
ipv6                  297665  52
autofs4                24521  0
sunrpc                169017  1
xfs                   563953  1
exportfs                8385  1 xfs
dm_mod                 69761  0
video                  20169  0
button                  9185  0
battery                12233  0
ac                      6857  0
i2c_amd8111             8129  0
i2c_core               28353  1 i2c_amd8111
hw_random               7393  0
e100                   42305  0
mii                     7105  1 e100
tg3                    91717  0
reiserfs              276665  3
3w_9xxx                38725  6
sd_mod                 20929  8
scsi_mod              155665  2 3w_9xxx,sd_mod


Part of dmesg:
mm/memory.c:97: bad pmd ffff810059861a80(000000000000000b).
mm/memory.c:97: bad pmd ffff810059861a90(000000000000000c).
mm/memory.c:97: bad pmd ffff810059861aa0(000000000000000d).
mm/memory.c:97: bad pmd ffff810059861ab0(000000000000000e).
mm/memory.c:97: bad pmd ffff810059861ac0(0000000000000017).
mm/memory.c:97: bad pmd ffff810059861ad0(000000000000000f).
mm/memory.c:97: bad pmd ffff810059861ad8(00007ffffffffaf8).
mm/memory.c:97: bad pmd ffff810059861af8(000034365f363878).
mm/memory.c:97: bad pmd ffff810074cb7878(0000003926400a88).
mm/memory.c:97: bad pmd ffff810074cb7880(0000000000000004).
mm/memory.c:97: bad pmd ffff810074cb7888(00007ffffffffb00).
mm/memory.c:97: bad pmd ffff810074cb7890(00007ffffffffb01).
mm/memory.c:97: bad pmd ffff810074cb7898(00007ffffffffb02).
mm/memory.c:97: bad pmd ffff810074cb78a0(00007ffffffffb03).
mm/memory.c:97: bad pmd ffff810074cb78b0(00007ffffffffb04).
mm/memory.c:97: bad pmd ffff810074cb78b8(00007ffffffffb05).
mm/memory.c:97: bad pmd ffff810074cb78c0(00007ffffffffb06).
mm/memory.c:97: bad pmd ffff810074cb78c8(00007ffffffffb07).
mm/memory.c:97: bad pmd ffff810074cb78d0(00007ffffffffb08).
mm/memory.c:97: bad pmd ffff810074cb78d8(00007ffffffffb09).
mm/memory.c:97: bad pmd ffff810074cb78e0(00007ffffffffb0a).
mm/memory.c:97: bad pmd ffff810074cb78e8(00007ffffffffb0b).
mm/memory.c:97: bad pmd ffff810074cb78f0(00007ffffffffb0c).
mm/memory.c:97: bad pmd ffff810074cb78f8(00007ffffffffb0d).
mm/memory.c:97: bad pmd ffff810074cb7900(00007ffffffffb0e).
mm/memory.c:97: bad pmd ffff810074cb7908(00007ffffffffb0f).
mm/memory.c:97: bad pmd ffff810074cb7910(00007ffffffffb10).
mm/memory.c:97: bad pmd ffff810074cb7918(00007ffffffffb11).
mm/memory.c:97: bad pmd ffff810074cb7920(00007ffffffffb12).
mm/memory.c:97: bad pmd ffff810074cb7928(00007ffffffffb13).
mm/memory.c:97: bad pmd ffff810074cb7930(00007ffffffffb14).
mm/memory.c:97: bad pmd ffff810074cb7938(00007ffffffffb15).
mm/memory.c:97: bad pmd ffff810074cb7940(00007ffffffffb16).
mm/memory.c:97: bad pmd ffff810074cb7948(00007ffffffffb17).
mm/memory.c:97: bad pmd ffff810074cb7950(00007ffffffffb18).
mm/memory.c:97: bad pmd ffff810074cb7958(00007ffffffffb19).
mm/memory.c:97: bad pmd ffff810074cb7960(00007ffffffffb1a).
mm/memory.c:97: bad pmd ffff810074cb7968(00007ffffffffb1b).
mm/memory.c:97: bad pmd ffff810074cb7970(00007ffffffffb1c).
mm/memory.c:97: bad pmd ffff810074cb7978(00007ffffffffb1d).
mm/memory.c:97: bad pmd ffff810074cb7980(00007ffffffffb1e).
mm/memory.c:97: bad pmd ffff810074cb7988(00007ffffffffb1f).
mm/memory.c:97: bad pmd ffff810074cb7990(00007ffffffffb20).
mm/memory.c:97: bad pmd ffff810074cb7998(00007ffffffffb21).
mm/memory.c:97: bad pmd ffff810074cb79a0(00007ffffffffb22).
mm/memory.c:97: bad pmd ffff810074cb79a8(00007ffffffffb23).
mm/memory.c:97: bad pmd ffff810074cb79b0(00007ffffffffb24).
mm/memory.c:97: bad pmd ffff810074cb79b8(00007ffffffffb25).
mm/memory.c:97: bad pmd ffff810074cb79c0(00007ffffffffb26).
mm/memory.c:97: bad pmd ffff810074cb79c8(00007ffffffffb27).
mm/memory.c:97: bad pmd ffff810074cb79d0(00007ffffffffb28).
mm/memory.c:97: bad pmd ffff810074cb79d8(00007ffffffffb29).
mm/memory.c:97: bad pmd ffff810074cb79e0(00007ffffffffb2a).
mm/memory.c:97: bad pmd ffff810074cb79f0(0000000000000010).
mm/memory.c:97: bad pmd ffff810074cb79f8(00000000078bfbff).
mm/memory.c:97: bad pmd ffff810074cb7a00(0000000000000006).
mm/memory.c:97: bad pmd ffff810074cb7a08(0000000000001000).
mm/memory.c:97: bad pmd ffff810074cb7a10(0000000000000011).
mm/memory.c:97: bad pmd ffff810074cb7a18(0000000000000064).
mm/memory.c:97: bad pmd ffff810074cb7a20(0000000000000003).
mm/memory.c:97: bad pmd ffff810074cb7a28(0000000000400040).
mm/memory.c:97: bad pmd ffff810074cb7a30(0000000000000004).
mm/memory.c:97: bad pmd ffff810074cb7a38(0000000000000038).
mm/memory.c:97: bad pmd ffff810074cb7a40(0000000000000005).
mm/memory.c:97: bad pmd ffff810074cb7a48(0000000000000008).
mm/memory.c:97: bad pmd ffff810074cb7a50(0000000000000007).
mm/memory.c:97: bad pmd ffff810074cb7a60(0000000000000008).
mm/memory.c:97: bad pmd ffff810074cb7a70(0000000000000009).
mm/memory.c:97: bad pmd ffff810074cb7a78(0000000000401480).
mm/memory.c:97: bad pmd ffff810074cb7a80(000000000000000b).
mm/memory.c:97: bad pmd ffff810074cb7a90(000000000000000c).
mm/memory.c:97: bad pmd ffff810074cb7aa0(000000000000000d).
mm/memory.c:97: bad pmd ffff810074cb7ab0(000000000000000e).
mm/memory.c:97: bad pmd ffff810074cb7ac0(0000000000000017).
mm/memory.c:97: bad pmd ffff810074cb7ad0(000000000000000f).
mm/memory.c:97: bad pmd ffff810074cb7ad8(00007ffffffffaf9).
mm/memory.c:97: bad pmd ffff810074cb7af8(0034365f36387800).

Re: x86-64 bad pmds in 2.6.11.6 II

[Christopher Warner]

> It does.  Well, I needed to restudy exec_mmap and switch_mm in detail,
> and having done so, I agree that the only way you can get through
> exec_mmap's activate_mm without fully flushing the cpu's TLB, is if
> the active_mm matches the newly allocated mm (itself impossible since
> there's a reference on the active_mm), and the cpu bit is still set
> in cpu_vm_mask - precisely not the case if we went through leave_mm.
> Yet I was claiming your leave_mm fix could flush TLB for exec_mmap
> where it wasn't already done.
> 
> Sorry for letting the neatness of my pmd/stack story blind me
> to its impossibility, and for wasting your time.
> 
> Hugh
> -

Any updated information one should know about this before testing?

I'm getting bad pmds in 2.6.11.5; Tyan S2882/dual AMD 246 opterons. The
problem only occurs when doing some thread intensive task. I'm going to
try and strace/bt and send some information as it occurs. Hardware is
almost identical to the setup above.

-Christopher Warner

Re: x86-64 bad pmds in 2.6.11.6 II

[Chris Wright]

* Christopher Warner (chris@servertogo.com) wrote:
> Any updated information one should know about this before testing?

Andi would like to see output from this patch:

http://marc.theaimsgroup.com/?l=linux-kernel&m=111461231610952&w=2

thanks,
-chris

Re: x86-64 bad pmds in 2.6.11.6 II

[Dave Jones]

On Fri, Apr 29, 2005 at 07:12:59AM -0400, Christopher Warner wrote:
 > 
 > > It does.  Well, I needed to restudy exec_mmap and switch_mm in detail,
 > > and having done so, I agree that the only way you can get through
 > > exec_mmap's activate_mm without fully flushing the cpu's TLB, is if
 > > the active_mm matches the newly allocated mm (itself impossible since
 > > there's a reference on the active_mm), and the cpu bit is still set
 > > in cpu_vm_mask - precisely not the case if we went through leave_mm.
 > > Yet I was claiming your leave_mm fix could flush TLB for exec_mmap
 > > where it wasn't already done.
 > > 
 > > Sorry for letting the neatness of my pmd/stack story blind me
 > > to its impossibility, and for wasting your time.
 > > 
 > > Hugh
 > > -
 > 
 > Any updated information one should know about this before testing?
 > 
 > I'm getting bad pmds in 2.6.11.5; Tyan S2882/dual AMD 246 opterons.

Datapoint: exactly the same model as my workstation which showed
this problem recently.

		Dave

Re: x86-64 bad pmds in 2.6.11.6 II

[Christopher Warner]

Actually I am testing your patches. Its just going to take some time.
The problem occurs under severe load and I'm in the process of doing
load testing this for an inhouse app this week. Soon as i'm able to send
debug information I will.

-Christopher Warner

On Mon, 2005-05-02 at 19:00 +0200, Andi Kleen wrote:
> > Datapoint: exactly the same model as my workstation which showed
> > this problem recently.
> 
> FYI
> 
> Since all the people who run into this are refusing to test my 
> debugging/test patches (no feedback at all for them so far) I give up on this 
> now.
> 
> -Andi
> 

Re: x86-64 bad pmds in 2.6.11.6 II

[Andi Kleen]

> Datapoint: exactly the same model as my workstation which showed
> this problem recently.

FYI

Since all the people who run into this are refusing to test my 
debugging/test patches (no feedback at all for them so far) I give up on this 
now.

-Andi

Re: x86-64 bad pmds in 2.6.11.6 II

[Chris Wright]

* Christopher Warner (chris@servertogo.com) wrote:
> Actually I am testing your patches. Its just going to take some time.
> The problem occurs under severe load and I'm in the process of doing
> load testing this for an inhouse app this week. Soon as i'm able to send
> debug information I will.

Same here.  I've just never found a way to trigger other than wait.

thanks,
-chris

Re: x86-64 bad pmds in 2.6.11.6 II

[Dave Jones]

On Mon, May 02, 2005 at 01:33:59PM -0700, Chris Wright wrote:
 > * Christopher Warner (chris@servertogo.com) wrote:
 > > Actually I am testing your patches. Its just going to take some time.
 > > The problem occurs under severe load and I'm in the process of doing
 > > load testing this for an inhouse app this week. Soon as i'm able to send
 > > debug information I will.
 > 
 > Same here.  I've just never found a way to trigger other than wait.

*nod*, the current test-kernel update for Fedora also has your
debugging patches, but none of the users have hit them (or reported
them) yet.

		Dave

Re: x86-64 bad pmds in 2.6.11.6 II

[Andi Kleen]

On Mon, May 02, 2005 at 05:08:39PM -0400, Dave Jones wrote:
> On Mon, May 02, 2005 at 01:33:59PM -0700, Chris Wright wrote:
>  > * Christopher Warner (chris@servertogo.com) wrote:
>  > > Actually I am testing your patches. Its just going to take some time.
>  > > The problem occurs under severe load and I'm in the process of doing
>  > > load testing this for an inhouse app this week. Soon as i'm able to send
>  > > debug information I will.
>  > 
>  > Same here.  I've just never found a way to trigger other than wait.
> 
> *nod*, the current test-kernel update for Fedora also has your
> debugging patches, but none of the users have hit them (or reported
> them) yet.

The second version with the WARN_ON? If not please update to that one.

-Andi

Re: x86-64 bad pmds in 2.6.11.6 II

[Dave Jones]

On Tue, May 03, 2005 at 04:28:58PM +0200, Andi Kleen wrote:
 > On Mon, May 02, 2005 at 05:08:39PM -0400, Dave Jones wrote:
 > > On Mon, May 02, 2005 at 01:33:59PM -0700, Chris Wright wrote:
 > >  > * Christopher Warner (chris@servertogo.com) wrote:
 > >  > > Actually I am testing your patches. Its just going to take some time.
 > >  > > The problem occurs under severe load and I'm in the process of doing
 > >  > > load testing this for an inhouse app this week. Soon as i'm able to send
 > >  > > debug information I will.
 > >  > 
 > >  > Same here.  I've just never found a way to trigger other than wait.
 > > 
 > > *nod*, the current test-kernel update for Fedora also has your
 > > debugging patches, but none of the users have hit them (or reported
 > > them) yet.
 > 
 > The second version with the WARN_ON? If not please update to that one.

I lost track.. Here's what I included..

--- linux-2.6.11/mm/memory.c~	2005-04-27 13:37:20.000000000 -0400
+++ linux-2.6.11/mm/memory.c	2005-04-27 13:37:45.000000000 -0400
@@ -94,6 +94,7 @@ static inline void clear_pmd_range(struc
 	if (pmd_none(*pmd))
 		return;
 	if (unlikely(pmd_bad(*pmd))) {
+		printk("%s:%d: ", current->comm, current->pid);
 		pmd_ERROR(*pmd);
 		pmd_clear(pmd);
 		return;
@@ -113,6 +114,7 @@ static inline void clear_pud_range(struc
 	unsigned long addr = start, next;
 	pmd_t *pmd, *__pmd;
 
+	WARN_ON(start == end);
 	if (pud_none(*pud))
 		return;
 	if (unlikely(pud_bad(*pud))) {



		Dave

Re: x86-64 bad pmds in 2.6.11.6 II

[Christopher Warner]

2.6.11.5 kernel,
Tyan S2882/dual AMD 246 opterons

sh:18983: mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). 
sh:18983: mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).
sh:18983: mm/memory.c:99: bad pmd ffff810005974cd8(00007ffffffffe48).
sh:18983: mm/memory.c:99: bad pmd ffff810005974ce0(00007ffffffffe49).
sh:18983: mm/memory.c:99: bad pmd ffff810005974ce8(00007ffffffffe4a).
sh:18983: mm/memory.c:99: bad pmd ffff810005974cf0(00007ffffffffe4b).
sh:18983: mm/memory.c:99: bad pmd ffff810005974cf8(00007ffffffffe4c).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d00(00007ffffffffe4d).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d08(00007ffffffffe4e).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d10(00007ffffffffe4f).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d18(00007ffffffffe50).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d20(00007ffffffffe51).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d30(0000000000000010).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d38(00000000078bfbff).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d40(0000000000000006).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d48(0000000000001000).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d50(0000000000000011).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d58(0000000000000064).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d60(0000000000000003).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d68(0000000000400040).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d70(0000000000000004).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d78(0000000000000038).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d80(0000000000000005).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d88(0000000000000008).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d90(0000000000000007).
sh:18983: mm/memory.c:99: bad pmd ffff810005974d98(00002aaaaaaab000).
sh:18983: mm/memory.c:99: bad pmd ffff810005974da0(0000000000000008).
sh:18983: mm/memory.c:99: bad pmd ffff810005974db0(0000000000000009).
sh:18983: mm/memory.c:99: bad pmd ffff810005974db8(0000000000413a00).
sh:18983: mm/memory.c:99: bad pmd ffff810005974dc0(000000000000000b).
sh:18983: mm/memory.c:99: bad pmd ffff810005974dc8(00000000000007d3).
sh:18983: mm/memory.c:99: bad pmd ffff810005974dd0(000000000000000c).
sh:18983: mm/memory.c:99: bad pmd ffff810005974dd8(00000000000007d3).
sh:18983: mm/memory.c:99: bad pmd ffff810005974de0(000000000000000d).
sh:18983: mm/memory.c:99: bad pmd ffff810005974df0(000000000000000e).
sh:18983: mm/memory.c:99: bad pmd ffff810005974e00(0000000000000017).
sh:18983: mm/memory.c:99: bad pmd ffff810005974e10(000000000000000f).
sh:18983: mm/memory.c:99: bad pmd ffff810005974e18(00007ffffffffe3a).
sh:18983: mm/memory.c:99: bad pmd ffff810005974e38(34365f3638780000)

Re: x86-64 bad pmds in 2.6.11.6 II

[Christopher Warner]

Thats from a couple of days ago. The bug used to be easily reproducible
and then it just stopped suddenly.

I've thrown the loads of the server machines in question way into the
2000 range with lots of threads, io, cpu usage. I'm really confused as
to what exactly it could be. I'm going to try on a couple of different
machines this week.

I'm starting to suspect it has something to do with the mobo itself,
since I have two or three next to me I'll try on those other machines
and then try on a completely different machine/motherboard.

It's really sneaky, so we'll see what happens.
 
On Tue, 2005-05-10 at 09:26 -0700, Chris Wright wrote:
> * Christopher Warner (chris@servertogo.com) wrote:
> > 2.6.11.5 kernel,
> > Tyan S2882/dual AMD 246 opterons
> 
> Got a time stamp by any chance (or a clue re: what was going on at the
> time)?
> 
> thanks,
> -chris

Re: x86-64 bad pmds in 2.6.11.6 II

[Chris Wright]

* Christopher Warner (chris@servertogo.com) wrote:
> 2.6.11.5 kernel,
> Tyan S2882/dual AMD 246 opterons

Got a time stamp by any chance (or a clue re: what was going on at the
time)?

thanks,
-chris

Re: x86-64 bad pmds in 2.6.11.6 II

[Dave Jones]

On Tue, May 10, 2005 at 05:36:54AM -0400, Christopher Warner wrote:
 > 2.6.11.5 kernel,
 > Tyan S2882/dual AMD 246 opterons
 > sh:18983: mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). 
 > sh:18983: mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).

That's the 3rd or 4th time I've seen this reported on this hardware.
It's not exclusive to it, but it does seem more susceptible
for some reason. Spooky.

		Dave

Re: x86-64 bad pmds in 2.6.11.6 II

[Andi Kleen]

On Tue, May 10, 2005 at 12:38:51PM -0400, Dave Jones wrote:
> On Tue, May 10, 2005 at 05:36:54AM -0400, Christopher Warner wrote:
>  > 2.6.11.5 kernel,
>  > Tyan S2882/dual AMD 246 opterons
>  > sh:18983: mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). 
>  > sh:18983: mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).
> 
> That's the 3rd or 4th time I've seen this reported on this hardware.
> It's not exclusive to it, but it does seem more susceptible
> for some reason. Spooky.

It seems to be clear now that it is hardware independent.

I actually got it once now too, but only after 24+h stress test :/

I have a better debugging patch now that I will be testing soon,
hopefully that turns something up.

-Andi

Re: x86-64 bad pmds in 2.6.11.6 II

[Dave Jones]

On Tue, May 10, 2005 at 06:46:49PM +0200, Andi Kleen wrote:
 > On Tue, May 10, 2005 at 12:38:51PM -0400, Dave Jones wrote:
 > > On Tue, May 10, 2005 at 05:36:54AM -0400, Christopher Warner wrote:
 > >  > 2.6.11.5 kernel,
 > >  > Tyan S2882/dual AMD 246 opterons
 > >  > sh:18983: mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). 
 > >  > sh:18983: mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).
 > > 
 > > That's the 3rd or 4th time I've seen this reported on this hardware.
 > > It's not exclusive to it, but it does seem more susceptible
 > > for some reason. Spooky.
 > 
 > It seems to be clear now that it is hardware independent.
 > 
 > I actually got it once now too, but only after 24+h stress test :/
 > 
 > I have a better debugging patch now that I will be testing soon,
 > hopefully that turns something up.

Ok, I'm respinning the Fedora update kernel today for other
reasons, if you have that patch in time, I'll toss it in too.

Though as yet, no further reports from our users.

		Dave

Re: x86-64 bad pmds in 2.6.11.6 II

[Andi Kleen]

On Tue, May 10, 2005 at 12:59:38PM -0400, Dave Jones wrote:
> On Tue, May 10, 2005 at 06:46:49PM +0200, Andi Kleen wrote:
>  > On Tue, May 10, 2005 at 12:38:51PM -0400, Dave Jones wrote:
>  > > On Tue, May 10, 2005 at 05:36:54AM -0400, Christopher Warner wrote:
>  > >  > 2.6.11.5 kernel,
>  > >  > Tyan S2882/dual AMD 246 opterons
>  > >  > sh:18983: mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). 
>  > >  > sh:18983: mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).
>  > > 
>  > > That's the 3rd or 4th time I've seen this reported on this hardware.
>  > > It's not exclusive to it, but it does seem more susceptible
>  > > for some reason. Spooky.
>  > 
>  > It seems to be clear now that it is hardware independent.
>  > 
>  > I actually got it once now too, but only after 24+h stress test :/
>  > 
>  > I have a better debugging patch now that I will be testing soon,
>  > hopefully that turns something up.
> 
> Ok, I'm respinning the Fedora update kernel today for other
> reasons, if you have that patch in time, I'll toss it in too.

The patch has considerable overhead, probably not  good idea
for a production rpm.

-Andi

Re: x86-64 bad pmds in 2.6.11.6 II

[Chris Wright]

* Andi Kleen (ak@suse.de) wrote:
> On Tue, May 10, 2005 at 12:59:38PM -0400, Dave Jones wrote:
> > On Tue, May 10, 2005 at 06:46:49PM +0200, Andi Kleen wrote:
> >  > On Tue, May 10, 2005 at 12:38:51PM -0400, Dave Jones wrote:
> >  > > On Tue, May 10, 2005 at 05:36:54AM -0400, Christopher Warner wrote:
> >  > >  > 2.6.11.5 kernel,
> >  > >  > Tyan S2882/dual AMD 246 opterons
> >  > >  > sh:18983: mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). 
> >  > >  > sh:18983: mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).
> >  > > 
> >  > > That's the 3rd or 4th time I've seen this reported on this hardware.
> >  > > It's not exclusive to it, but it does seem more susceptible
> >  > > for some reason. Spooky.
> >  > 
> >  > It seems to be clear now that it is hardware independent.
> >  > 
> >  > I actually got it once now too, but only after 24+h stress test :/
> >  > 
> >  > I have a better debugging patch now that I will be testing soon,
> >  > hopefully that turns something up.
> > 
> > Ok, I'm respinning the Fedora update kernel today for other
> > reasons, if you have that patch in time, I'll toss it in too.
> 
> The patch has considerable overhead, probably not  good idea
> for a production rpm.

I don't mind running it here.  I've triggered it once, and not hit the
WARN_ON(start == end).  Current was "sh", not that helpful.

thanks,
-chris

Re: x86-64 bad pmds in 2.6.11.6 II

[Andi Kleen]

On Tue, May 10, 2005 at 12:59:38PM -0400, Dave Jones wrote:
> On Tue, May 10, 2005 at 06:46:49PM +0200, Andi Kleen wrote:
>  > On Tue, May 10, 2005 at 12:38:51PM -0400, Dave Jones wrote:
>  > > On Tue, May 10, 2005 at 05:36:54AM -0400, Christopher Warner wrote:
>  > >  > 2.6.11.5 kernel,
>  > >  > Tyan S2882/dual AMD 246 opterons
>  > >  > sh:18983: mm/memory.c:99: bad pmd ffff810005974cc8(00007ffffffffe46). 
>  > >  > sh:18983: mm/memory.c:99: bad pmd ffff810005974cd0(00007ffffffffe47).
>  > > 
>  > > That's the 3rd or 4th time I've seen this reported on this hardware.
>  > > It's not exclusive to it, but it does seem more susceptible
>  > > for some reason. Spooky.
>  > 
>  > It seems to be clear now that it is hardware independent.
>  > 
>  > I actually got it once now too, but only after 24+h stress test :/
>  > 
>  > I have a better debugging patch now that I will be testing soon,
>  > hopefully that turns something up.
> 
> Ok, I'm respinning the Fedora update kernel today for other
> reasons, if you have that patch in time, I'll toss it in too.
> 
> Though as yet, no further reports from our users.

Here's the new patch. However it costs some memory bloat because
I added a new field to struct page 

-Andi


Index: linux/mm/page_alloc.c
===================================================================
--- linux.orig/mm/page_alloc.c
+++ linux/mm/page_alloc.c
@@ -912,6 +912,7 @@ nopage:
 	return NULL;
 got_pg:
 	zone_statistics(zonelist, z);
+	page->freer = (void *)-1ULL; 
 	return page;
 }
 
@@ -962,13 +963,16 @@ void __pagevec_free(struct pagevec *pvec
 {
 	int i = pagevec_count(pvec);
 
-	while (--i >= 0)
+	while (--i >= 0) {
+		pvec->pages[i]->freer = (void *)__builtin_return_address(0);
 		free_hot_cold_page(pvec->pages[i], pvec->cold);
+	}
 }
 
 fastcall void __free_pages(struct page *page, unsigned int order)
 {
 	if (!PageReserved(page) && put_page_testzero(page)) {
+		page->freer = (void *)__builtin_return_address(0);
 		if (order == 0)
 			free_hot_page(page);
 		else
@@ -1595,6 +1599,7 @@ void __init memmap_init_zone(unsigned lo
 	struct page *page;
 
 	for (page = start; page < (start + size); page++) {
+		page->freer = NULL;
 		set_page_zone(page, NODEZONE(nid, zone));
 		set_page_count(page, 0);
 		reset_page_mapcount(page);
Index: linux/include/linux/mm.h
===================================================================
--- linux.orig/include/linux/mm.h
+++ linux/include/linux/mm.h
@@ -261,6 +261,7 @@ struct page {
 	void *virtual;			/* Kernel virtual address (NULL if
 					   not kmapped, ie. highmem) */
 #endif /* WANT_PAGE_VIRTUAL */
+	void *freer;
 };
 
 /*
Index: linux/mm/memory.c
===================================================================
--- linux.orig/mm/memory.c
+++ linux/mm/memory.c
@@ -48,6 +48,7 @@
 #include <linux/rmap.h>
 #include <linux/module.h>
 #include <linux/init.h>
+#include <linux/kallsyms.h>
 
 #include <asm/pgalloc.h>
 #include <asm/uaccess.h>
@@ -108,6 +109,14 @@ static inline void clear_pmd_range(struc
 
 	pmd = pmd_offset(pud, addr);
 
+	{
+	struct page *p = virt_to_page(pmd); 
+	if (page_count(p) < 1) {
+		printk("%s:%d free pmd %lx ", current->comm, current->pid, addr); 
+		print_symbol("freed by %s\n", (unsigned long)p->freer); 
+	}
+	}
+
 	/* Only free fully aligned ranges */
 	if (!((addr | end) & ~PUD_MASK))
 		empty_pmd = pmd;

Re: x86-64 bad pmds in 2.6.11.6 II

[Peter J. Stieber]

CW = Christopher Warner
CW>>>>> 2.6.11.5 kernel,
CW>>>>> Tyan S2882/dual AMD 246 opterons
CW>>>>> sh:18983: mm/memory.c:99: bad pmd
ffff810005974cc8(00007ffffffffe46).
CW>>>>> sh:18983: mm/memory.c:99: bad pmd
ffff810005974cd0(00007ffffffffe47).

DJ = Dave Jones
DJ>>>> That's the 3rd or 4th time I've seen this
DJ>>>> reported on this hardware.
DJ>>>> It's not exclusive to it, but it does seem more
DJ>>>> susceptible for some reason. Spooky.

AK = Andi Kleen wrote:
AK>>> It seems to be clear now that it is hardware
AK>>> independent.
AK>>>
AK>>> I actually got it once now too, but only after
AK>>> 24+h stress test :/
AK>>>
AK>>> I have a better debugging patch now that I will be
AK>>> testing soon, hopefully that turns something up.

DJ = Dave Jones
DJ>> Ok, I'm respinning the Fedora update kernel today
DJ>> for other reasons, if you have that patch in time,
DJ>> I'll toss it in too.
DJ>>
DJ>> Though as yet, no further reports from our users.

AK = Andi Kleen
AK> Here's the new patch. However it costs some memory
AK> bloat because I added a new field to struct page

I posted some information on the fedora-list concerning my experience
with this problem. I am using a Tyan S2885/dual 244 Opterons. For HW and
driver details see:
https://www.redhat.com/archives/fedora-list/2005-May/msg01690.html

I have been using Dave's FC3 test kernel (2.6.11-1.24_FC3smp) for a
little over a day and have been unable to generate the problem with the
computer under a larger than normal load.

Prior to May 12, I had been seeing the problem very regularly. It
started around April 14. I believe this is about the time I first
started using the 2.6.11-1.14_FC3smp kernel. I remember I had to get a
BIOS upgrade from Tyan (http://www.tyan.com/support/html/b_s2885.html
unfortunately a Beta release) to get my network back once I started
using the new kernel. After that the memory.c messages started showing
up. I though it might be BIOS related (see the note on the referenced
Tyan page), but when I saw Christopher's post I thought maybe it was the
kernel because his MOBO doesn't have a Beta BIOS release. I googled and
found this thread, and subscribed to this list.

I have "memory.c:97 bad pmd" entries in my /var/log/messages files going
back to April 14. The only days I don't have them are April 20, 22, 23,
24 and April 29 (22, 23, and 24 are a weekend with less activity). I
have had them every day in May until I installed Dave's test kernel.

I am very computer literate, but not a kernel developer. I hope I didn't
offend you guys by posting here. I would be willing to be your guinea
pig for testing. Currently I am unable to reproduce the problem. If I am
able to reproduce the problem, would you prefer I post here or on the
fedora list?

Thanks for all of your efforts,
Pete

Re: x86-64 bad pmds in 2.6.11.6 II

[Peter J. Stieber]

CW = Christopher Warner
CW>>>>>> 2.6.11.5 kernel,
CW>>>>>> Tyan S2882/dual AMD 246 opterons
CW>>>>>> sh:18983: mm/memory.c:99: bad pmd
CW>>>>>> ffff810005974cc8(00007ffffffffe46).
CW>>>>>> sh:18983: mm/memory.c:99: bad pmd
CW>>>>>> ffff810005974cd0(00007ffffffffe47).

DJ = Dave Jones
DJ>>>>> That's the 3rd or 4th time I've seen this
DJ>>>>> reported on this hardware.
DJ>>>>> It's not exclusive to it, but it does seem more
DJ>>>>> susceptible for some reason. Spooky.

AK = Andi Kleen wrote:
AK>>>> It seems to be clear now that it is hardware
AK>>>> independent.
AK>>>>
AK>>>> I actually got it once now too, but only after
AK>>>> 24+h stress test :/
AK>>>>
AK>>>> I have a better debugging patch now that I will be
AK>>>> testing soon, hopefully that turns something up.

DJ = Dave Jones
DJ>>> Ok, I'm respinning the Fedora update kernel today
DJ>>> for other reasons, if you have that patch in time,
DJ>>> I'll toss it in too.
DJ>>>
DJ>>> Though as yet, no further reports from our users.

PJS = Peter J. Stieber
PJS> I posted some information on the fedora-list
PJS> concerning my experience with this problem. I
PJS> am using a Tyan S2885/dual 244 Opterons. For HW and
PJS> driver details see:
PJS> https://www.redhat.com/archives/fedora-list/2005-May/msg01690.html
PJS> I have been using Dave's FC3 test kernel
PJS> (2.6.11-1.24_FC3smp) for a little over a day and
PJS> have been unable to generate the problem with the
PJS> computer under a larger than normal load.
PJS>
PJS> Prior to May 12, I had been seeing the problem very
PJS> regularly. It started around April 14. I believe this
PJS> is about the time I first started using the
PJS> 2.6.11-1.14_FC3smp kernel. I remember I had to get a
PJS> BIOS upgrade from Tyan
PJS> (http://www.tyan.com/support/html/b_s2885.html
PJS> unfortunately a Beta release) to get my network back
PJS> once I started using the new kernel. After that the
PJS> memory.c messages started showing up. I though it
PJS> might be BIOS related (see the note on the referenced
PJS> Tyan page), but when I saw Christopher's post I
PJS> thought maybe it was the kernel because his MOBO
PJS> doesn't have a Beta BIOS release. I googled and
PJS> found this thread, and subscribed to this list.
PJS>
PJS> I have "memory.c:97 bad pmd" entries in my
PJS> /var/log/messages files going back to April 14.
PJS> The only days I don't have them are April 20, 22, 23,
PJS> 24 and April 29 (22, 23, and 24 are a weekend with
PJS> less activity). I have had them every day in May
PJS> until I installed Dave's test kernel.
PJS>
PJS> I am very computer literate, but not a kernel
PJS> developer. I hope I didn't offend you guys by
PJS> posting here. I would be willing to be your guinea
PJS> pig for testing. Currently I am unable to reproduce
PJS> the problem. If I am able to reproduce the problem,
PJS> would you prefer I post here or on the
PJS> fedora list?
PJS>
PJS> Thanks for all of your efforts,

My Tyan S2885 just exhibited the problem with Dave's test FC3 kernel
(2.6.11-1.24_FC3smp):

May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d008(0000000000000008).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d018(0000000000000009).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d020(0000000000401b80).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d028(000000000000000b).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d030(00000000000001f4).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d038(000000000000000c).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d040(00000000000001f4).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d048(000000000000000d).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d050(00000000000001f7).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d058(000000000000000e).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d060(00000000000001f7).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d068(0000000000000017).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d078(000000000000000f).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d080(00007ffffffff0a4).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d0a0(5f36387800000000).
May 14 10:00:18 maggie kernel: collect2:14167: mm/memory.c:98: bad pmd
ffff81005856d0a8(0000000000003436).

It died while running configure. It was attempting to use gcc to make
sure C compiling was possible. In the past, if I run the script a second
time it works.

Sorry for my ignorance, but what other information do you need?
Pete