* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
@ 2012-10-08 4:53 Zhang, Lin-Bao (Linux Kernel R&D)
2012-10-08 23:57 ` Suresh Siddha
0 siblings, 1 reply; 11+ messages in thread
From: Zhang, Lin-Bao (Linux Kernel R&D) @ 2012-10-08 4:53 UTC (permalink / raw)
To: Suresh Siddha
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, hpa, x86,
a.p.zijlstra, jarkko.sakkinen, joerg.roedel, agordeev, yinghai,
stable
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 10540 bytes --]
Hi Suresh,
Could you please update current status about these 2 files and patch?
I am not sure if I have answered your questions , if not ,feel free to let me know.
This is my first time to submit patch to LKML, so what should I do next step ?
About this patch , where needs to be enhanced ?
Thanks very much!
-- Bob(LinBao Zhang)
HP linux kernel enginner
> -----Original Message-----
> From: Zhang, Lin-Bao (Linux Kernel R&D)
> Sent: 2012å¹´9æ21æ¥ 1:16
> To: 'Suresh Siddha'
> Cc: linux-kernel@vger.kernel.org; alan@lxorguk.ukuu.org.uk;
> mingo@redhat.com; Croxon, Nigel; 'tglx@linutronix.de'; 'hpa@zytor.com';
> 'x86@kernel.org'; 'a.p.zijlstra@chello.nl'; 'jarkko.sakkinen@intel.com';
> 'joerg.roedel@amd.com'; 'agordeev@redhat.com'; 'yinghai@kernel.org';
> 'stable@kernel.org'
> Subject: RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A
> interrupt during the time window between changing VT-d table base address
> and initializing these VT-d entries(smpboot.c and apic.c )
>
> Hi suresh,
>
> Thanks for your reply and review this patch.
> I also cc other maintainers of arch/x86/kernel/smpboot.c and
> arch/x86/kernel/apic/apic.c(getting them by get_maintainer.pl script,
> hopefully I have not disturbed many people , if yes ,sorry first)
>
>
> > -----Original Message-----
> > From: Suresh Siddha [mailto:suresh.b.siddha@intel.com]
> > Sent: 2012å¹´9æ21æ¥ 6:23
> > To: Zhang, Lin-Bao (Linux Kernel R&D)
> > Cc: linux-kernel@vger.kernel.org; alan@lxorguk.ukuu.org.uk;
> > mingo@redhat.com; Croxon, Nigel
> > Subject: Re: [PATCH] fix x2apic defect that Linux kernel doesn't mask
> > 8259A interrupt during the time window between changing VT-d table
> > base address and initializing these VT-d entries
> >
> > On Wed, 2012-09-12 at 07:02 +0000, Zhang, Lin-Bao (ESSN-MCXS-Linux
> > Kernel
> > R&D) wrote:
> > > Hi all,
> > > This defect can be observed when the x2apic setting in BIOS is set
> > > to "auto" and the BIOS has virtual wire mode enabled on a power up.
> > > This defect was found on a 2.6.32 based kernel.
> >
> > I assume you are able to reproduce the issue with the latest kernel aswell?
> >
> In fact , this is what I want to further discussion. Thanks for your comments
> about 3.x on x2apic.
> We can only reproduce this issue on 2.6.x kernel, including RHEL6.1/6.2/6.3
> and sles11sp1, they are all of 2.6.xx series.
> In 3.x upstream series , we didn't reproduce this problem, I ever tested
> upstream version : 3.0.0 , 3.0.38 , 3.1.10 ,3.3.8,3.4.4, we can't reproduce it.
> But I don't think this can prove that 3.x.x doesn't have potential problem
> similar with 2.6.x .
> By reviewing the 3.x kernel source , I found that 3.xx source have the same
> design defect ,but we don't know why it doesn't trigger this problem as 2.6 ,
> maybe other part work around this issue , so welcome comments ,we need to
> know the real reason.
> Anyway , from 3.x.x kernel source , it still first change VT-d table base address ,
> after some time, linux kernel then initialize RTEs. So during the window ,
> present bit must 0.
> During this window slot , if a interrupt is coming , platform will check VT-d
> entry 's present bit is 0 , cause non-fatal error and send NMI to OS. By intel's
> ITP we can clearly watch this error is caused :
> 0x8000_0022_0000_00F1_0000_0000_0000_0000 -> [22] Bit 103:96: FR
> Fault Reason is 22h The Present (P) field in the IRTE entry corresponding to the
> interrupt_index of the interrupt request is Clear. (Appendix A Fault Reason
> Encodings)
>
> In fact, this error is just non-fatal , if firmware designed well, it should depress
> this error , I think after some time, VT-d entry has been initialized successfully ,
> this error won't exist again.
> I think the direction for kernel source to avoid this problem regardless
> firmware is :
> a) mask all 8259A interrupt -> b) create a new VT-d table ,and initialize all
> entries (RTEs) -> c) take over BIOS's simple VT-d table by kernel's VT-d table
> base address --> d) unmask 8259A
> thus linux kernel can correctly handle interrupt. I think this should be safe.
> How do you think about it ?
>
>
> > What virtual wire mode is it?
> > Virtual wire mode-A (where the PIC output is connected to LINT0 of the
> > Local
> > APIC) doesn't go through interrupt-remapping and virtual wire mode-B
> > (where the PIC output is routed through the IO-APIC RTE) will be
> > completely disabled as all the BIOS setup IO-APIC RTE's are masked by
> > the Linux kernel from the time we enable interrupt-remapping to the
> > time IO-APIC RTE's are properly re-configured by the Linux kernel again.
> >
> > So I am at a loss to understand what is causing this.
> >
> Yeah , Virtual wire mob B need to use io-apic .
> If no io-apic , this issue will never occur.
>
>
> > >
> > > The kernel code (smpboot.c, apic.c) does not mask 8259A interrupts
> > > before changing and initializing the new VT-d table when x2apic
> > > virtual wire mode is enable on power up. The Linux Kernel expects
> > > virtual wire mode to be disabled when booting and enables it when
> > > interrupts are masked.
> > >
> > > The BIOS code builds a simple VT-d table on power up. While the
> > > Linux Kernel boots, it first builds an empty VT-d table and use it.
> > > After some time, the Linux Kernel then initializes the IO-APIC
> > > redirect table, and then initializes the VT-d entries. The window
> > > between initializing the redirect table and the VT-d entries, the
> > > 8259A interrupts are not masked. If an interrupt occurs in this
> > > window, the Linux Kernel will not find a valid entry for this
> > > interrupt. The kernel treats it to be a fatal error and panics. If
> > > the error never gets cleared, the Linux kernel continuously print this error:
> > > "NMI: IOCK error (debug interrupt?) for reason"
> >
> > Not sure why we get a NMI instead of a vt-d fault? Perhaps the vt-d
> > fault is also getting reported via NMI in this platform?
> >
> Yes, you are right.
> When VT-d entry is Present bit is 0 , it will cause platform non-fatal error , and
> platform will send NMI(NMI reason is IOCHK,you know NMI can have many
> reasons) .
> Because this non-fatal err exists forever , so platform will send NMI looply to
> OS , so OS will receive many NMI , so linux kernel will print looply
> "NMI: IOCK error (debug interrupt?)" , linux kernel can't do any other things.
>
> Following is error messages : in 2.6.32 kernel , we always reproduce it every
> time( adding x2apic_phys is reasonable)
>
>
> ------------error logs: -------------------------------------------
> IOAPIC id 10 under DRHD base 0xace00000
> IOAPIC id 8 under DRHD base 0xa8000000
> IOAPIC id 0 under DRHD base 0xa8000000
> Enabled IRQ remapping in x2apic mode
> NMI: IOCK error (debug interrupt?)
> CPU 0
> Modules linked in:
>
> Pid: 1, comm: swapper Not tainted 2.6.32rhel6.2-Bob #1 HP ProLiant DL980
> G7
> RIP: 0010:[<ffffffff810de39e>] [<ffffffff810de39e>]
> check_for_new_grace_period+0x2e/0xd0
> RSP: 0018:ffff880046003e40 EFLAGS: 00000082
> RAX: 0000000000000282 RBX: 0000000000000282 RCX: 0000000000000000
> RDX: fffffffffffffed4 RSI: ffff880046011400 RDI: ffffffff81aaf640
> RBP: ffff880046003e60 R08: 0000000000989680 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaf640
> R13: ffff880046011400 R14: 0000000000000100 R15: 0000000000000009
> FS: 0000000000000000(0000) GS:ffff880046000000(0000)
> knlGS:0000000000000000
> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 0000000001a85000 CR4: 00000000000006f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process swapper (pid: 1, threadinfo ffff88c7ebe4e000, task ffff88086b4694c0)
> Stack:
> ffff880046011400 ffffffff81aaf640 0000000000000048 0000000000000100
> <0> ffff880046003eb0 ffffffff810dedb4 ffffffff81a8ebe0 ffffffff81ea2120 <0>
> ffff880046003e80 0000000000000001 ffffffff81a830c8 0000000000000048
> Call Trace:
> <IRQ>
> [<ffffffff810dedb4>] __rcu_process_callbacks+0x54/0x330
> [<ffffffff810df0da>] rcu_process_callbacks+0x4a/0x50 [<ffffffff81072161>]
> __do_softirq+0xc1/0x1d0 [<ffffff100e75e>] ? timer_interrupt+0x1e/0x30
> [<ffffffff8100c24c>] call_softirq+0x1c/0x30 [<ffffffff8100de85>]
> do_softirq+0x65/0xa0 [<ffffffff81071f45>] irq_exit+0x85/0x90
> [<ffffffff814f4cf5>] do_IRQ+0x75/0xf0 [<ffffffff8100ba53>]
> ret_from_intr+0x0/0x11 <EOI> [<ffffffff81c303ec>] ?
> enable_IR_x2apic+0x18a/0x221 [<ffffffff81c2e189>]
> native_smp_prepare_cpus+0x143/0x389
> [<ffffffff81c1f740>] kernel_init+0x112/0x2f9 [<ffffffff8100c14a>]
> child_rip+0xa/0x20 [<ffffffff81c1f62e>] ? kernel_init+0x0/0x2f9
> [<ffffffff8100c140>] ? child_rip+0x0/0x20
> Code: e5 48 83 ec 20 48 89 1c 24 4c 89 64 24 08 4c 89 6c 24 10 4c 89 74 24
> 18 0f 1f 44 00 00 49 89 f5 9c 58 0f 1f 44 00 00 48 89 c3 fa <66> 0f 1f 44 00 00
> 31 d2 48 8b 87 c8 a0 00 00 48 39 46 08 74 6c
> NMI: IOCK error (debug interrupt?)
>
>
>
> > Does your tested kernel has this fix?
> > commit 254e42006c893f45bca48f313536fcba12206418
> > Author: Suresh Siddha <suresh.b.siddha@intel.com>
> > Date: Mon Dec 6 12:26:30 2010 -0800
> >
> > x86, vt-d: Quirk for masking vtd spec errors to platform error
> > handling logic
> >
> Let me take some time to research it, I think it seems that you would
> mask/depress VT-d spec errors( for example , The Present (P) field in the IRTE
> entry corresponding to the interrupt_index of the interrupt request is Clear.)
> But I think ,this is just disable error reporting or disable error handling. But in
> our machine , if we found this error , platform will send NMI to OS.
> Maybe other platform don't send NMI to OS.
> But for linux kernel , we need to assure no this error occur , not depress
> error(certainly, if error is non-fatal , we can depress it ; if fatal error , we must
> stop machine and restart).
> For OS , how to differ fatal and non-fatal error ?
>
> > Will you be able to provide the failing kernel log so that I can
> > better understand the issue?
> >
> I have pasted error logs above , if you need all booting log , I can send it to
> public location ,and give you a link. I don't want to paste it all here, too long. :)
> Or need I submit a bug in bugzilla.kernel.org ?
>
>
> > thanks,
> > suresh
> >
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
2012-10-08 4:53 [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c ) Zhang, Lin-Bao (Linux Kernel R&D)
@ 2012-10-08 23:57 ` Suresh Siddha
2012-10-10 0:26 ` Zhang, Lin-Bao (Linux Kernel R&D)
2012-10-10 23:02 ` Zhang, Lin-Bao (Linux Kernel R&D)
0 siblings, 2 replies; 11+ messages in thread
From: Suresh Siddha @ 2012-10-08 23:57 UTC (permalink / raw)
To: Zhang, Lin-Bao (Linux Kernel R&D)
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, hpa, x86,
a.p.zijlstra, Sakkinen, Jarkko, joerg.roedel, agordeev, yinghai,
stable
On Sun, 2012-10-07 at 21:53 -0700, Zhang, Lin-Bao (Linux Kernel R&D)
wrote:
> Hi Suresh,
> Could you please update current status about these 2 files and patch?
> I am not sure if I have answered your questions , if not ,feel free to let me know.
> This is my first time to submit patch to LKML, so what should I do next step ?
As I mentioned earlier, the current design already ensures that all the
IO-APIC RTE's are masked between the time we enable interrupt-remapping
to the time when the IO-APIC RTE's are configured correctly.
So I looked at why you are seeing the problem with v2.6.32 but not with
the recent kernels. And I think I found out the reason.
2.6.32 kernel is missing this fix,
http://marc.info/?l=linux-acpi&m=126993666715081&w=2
commit 7716a5c4ff5f1f3dc5e9edcab125cbf7fceef0af
Author: Eric W. Biederman <ebiederm@xmission.com>
Date: Tue Mar 30 01:07:12 2010 -0700
x86, ioapic: Move nr_ioapic_registers calculation to mp_register_ioapic.
Now that all ioapic registration happens in mp_register_ioapic we can
move the calculation of nr_ioapic_registers there from enable_IO_APIC.
The number of ioapic registers is already calucated in mp_register_ioapic
so all that really needs to be done is to save the caluclated value
in nr_ioapic_registers.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
LKML-Reference: <1269936436-7039-11-git-send-email-ebiederm@xmission.com>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Because of this, in v2.6.32, mask_IO_APIC_setup() is not working as
expected as nr_ioapic_registers[] are not yet initialized and thus the
io-apic RTE's are not masked as expected.
We just need the last hunk of that patch, I think.
Can you please apply the appended patch to 2.6.32 kernel and see if the
issue you mentioned gets fixed? If so, we can ask the -stable and OSV's
teams to pick up this fix.
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index f807255..dae9240 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -4293,6 +4281,7 @@ static int bad_ioapic(unsigned long address)
void __init mp_register_ioapic(int id, u32 address, u32 gsi_base)
{
int idx = 0;
+ int entries;
if (bad_ioapic(address))
return;
@@ -4311,9 +4300,14 @@ void __init mp_register_ioapic(int id, u32 address, u32 gsi_base)
* Build basic GSI lookup table to facilitate gsi->io_apic lookups
* and to prevent reprogramming of IOAPIC pins (PCI GSIs).
*/
+ entries = io_apic_get_redir_entries(idx);
mp_gsi_routing[idx].gsi_base = gsi_base;
- mp_gsi_routing[idx].gsi_end = gsi_base +
- io_apic_get_redir_entries(idx) - 1;
+ mp_gsi_routing[idx].gsi_end = gsi_base + entries - 1;
+
+ /*
+ * The number of IO-APIC IRQ registers (== #pins):
+ */
+ nr_ioapic_registers[idx] = entries;
if (mp_gsi_routing[idx].gsi_end > gsi_end)
gsi_end = mp_gsi_routing[idx].gsi_end;
^ permalink raw reply related [flat|nested] 11+ messages in thread
* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
2012-10-08 23:57 ` Suresh Siddha
@ 2012-10-10 0:26 ` Zhang, Lin-Bao (Linux Kernel R&D)
2012-10-11 0:07 ` Suresh Siddha
2012-10-10 23:02 ` Zhang, Lin-Bao (Linux Kernel R&D)
1 sibling, 1 reply; 11+ messages in thread
From: Zhang, Lin-Bao (Linux Kernel R&D) @ 2012-10-10 0:26 UTC (permalink / raw)
To: Suresh Siddha
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, hpa, x86,
a.p.zijlstra, Sakkinen, Jarkko, joerg.roedel, agordeev, yinghai,
stable
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2581 bytes --]
Hi Suresh,
Thanks very much for your reply!
I tested this patch in 2.6.32 with our machine , it can indeed can resolve current issue. that 's good.
I also check 3.x version of kernel source , for example , 3.3.8 and 3.0.0 , they indeed include similar patch ,
I think the key action is this line :
+ /*
+ * The number of IO-APIC IRQ registers (== #pins):
+ */
+ nr_ioapic_registers[idx] = entries;
In 3.3.8 , it is like this:
/*
4011 * The number of IO-APIC IRQ registers (== #pins):
4012 */
4013 ioapics[idx].nr_registers = entries;
I also use 3.3.8 to test , no modification don't reproduce our issue.
If I comment this line (ioapics[idx].nr_registers = entries;) , it will reproduce the problem that occurs in 2.6.32.
so this can prove that your patch should work for 2.6.x kernel.
But I am not sure why it can work. Let's discuss again. I am researching the whole source again.
It seems that we have 2 directions to fix current problem :
a) your patch : all the IO-APIC RTE's are masked between the time we enable interrupt-remapping to the time
> when the IO-APIC RTE's are configured correctly.
b) my patch mask interrupt during VT-d table base address changing and VT-d entries initialized successfully.
I suppose during the time window , we should disable 8259A interrupt.
> As I mentioned earlier, the current design already ensures that all the IO-APIC
> RTE's are masked between the time we enable interrupt-remapping to the time
> when the IO-APIC RTE's are configured correctly.
>
So , we can think ,as your patch , during the window , IO-apic is useless or we can think IO-APIC doesn't exist ?
Could you mind please sharing your design details ? thanks very much!
> So I looked at why you are seeing the problem with v2.6.32 but not with the
> recent kernels. And I think I found out the reason.
>
> 2.6.32 kernel is missing this fix,
>
Yes, 2.6.32 doesn't have this patch. 3.x.x indeed have it.
> Because of this, in v2.6.32, mask_IO_APIC_setup() is not working as expected
> as nr_ioapic_registers[] are not yet initialized and thus the io-apic RTE's are not
> masked as expected.
>
> We just need the last hunk of that patch, I think.
>
> Can you please apply the appended patch to 2.6.32 kernel and see if the issue
> you mentioned gets fixed? If so, we can ask the -stable and OSV's teams to
> pick up this fix.
Yes , it can resolve current issue.
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
2012-10-08 23:57 ` Suresh Siddha
2012-10-10 0:26 ` Zhang, Lin-Bao (Linux Kernel R&D)
@ 2012-10-10 23:02 ` Zhang, Lin-Bao (Linux Kernel R&D)
2012-10-11 0:01 ` Suresh Siddha
1 sibling, 1 reply; 11+ messages in thread
From: Zhang, Lin-Bao (Linux Kernel R&D) @ 2012-10-10 23:02 UTC (permalink / raw)
To: Suresh Siddha
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, hpa, x86,
a.p.zijlstra, Sakkinen, Jarkko, joerg.roedel, agordeev, yinghai,
Zhang, Lin-Bao (Linux Kernel R&D)
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2646 bytes --]
Hi Suresh,
Now , I want to clarify 2 type of errors :
a) some interrupt doesn't work through Interrupt -remapped IO-APIC.
b) an interrupt is coming , there is an IO-APIC entry to know how to route it to which CPU,
but VT-d entry is empty, ITP will report Present bit is clear.
BIOS should provide simple IO-APIC entries and VT-d entries ,although we don't trust them.
in the issue that we have happened to is of the second case.
1 ,
> As I mentioned earlier, the current design already ensures that all the IO-APIC
> RTE's are masked between the time we enable interrupt-remapping to the time
> when the IO-APIC RTE's are configured correctly.
>
> So I looked at why you are seeing the problem with v2.6.32 but not with the
> recent kernels. And I think I found out the reason.
I want to know what masking IO-APIC means?
I know from BIOS point of view ,they can disable IO-APIC, just like no IO-APIC equipment.
So masking IO-APIC entries equals to disable IO-APIC ? when a interrupt is coming ,
System doesn't make it through IO-APIC ,and directly to Local APIC ?
If yes , I have a question :
I first draw a time line :
a) Clear IO-APIC table c)
|-------------------------------------------------|------------------------------------------------------|
Change VT-d table b) create IO-APIC table and VT-d table and initialize them
Empty table empty IO-APIC table
When interrupt is coming between a) and b) ? I suppose the error : "VT-d entry 's Present bit is clear ).
When interrupt is coming between b) and c) ? ??
When interrupt is coming after c) point ? I suppose everything will work fine.
2,
We know maybe bios will provide a simple IO-APIC table for us, although we OS doesn't believe it.
I also did a test :
- disable x2apic in BIOS.
find kernel source 3.3.8 , and delete that patch (comment that key line) .
I found that it would display :
Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC.
But if we disable x2apic in BIOS , rhel6.2 doesn't need any patch, it can run. Very strange.
- enable x2apic in BIOS
delete your patch from 3.3.8 , same issue with us. VT-d entry error (Present bit0 is clear ).
But it doesn't display the error above(don't tell me it doesn't work through IO-APIC ,
it represents io-apic entry exists) .
-- Bob(LinBao Zhang)
HP linux kernel enginner
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
2012-10-10 23:02 ` Zhang, Lin-Bao (Linux Kernel R&D)
@ 2012-10-11 0:01 ` Suresh Siddha
2012-10-11 16:46 ` Zhang, Lin-Bao (Linux Kernel R&D)
0 siblings, 1 reply; 11+ messages in thread
From: Suresh Siddha @ 2012-10-11 0:01 UTC (permalink / raw)
To: Zhang, Lin-Bao (Linux Kernel R&D)
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, hpa, x86,
a.p.zijlstra, Sakkinen, Jarkko, joerg.roedel, agordeev, yinghai
On Wed, 2012-10-10 at 16:02 -0700, Zhang, Lin-Bao (Linux Kernel R&D)
wrote:
> > As I mentioned earlier, the current design already ensures that all the IO-APIC
> > RTE's are masked between the time we enable interrupt-remapping to the time
> > when the IO-APIC RTE's are configured correctly.
> >
> > So I looked at why you are seeing the problem with v2.6.32 but not with the
> > recent kernels. And I think I found out the reason.
>
> I want to know what masking IO-APIC means?
As the platform is configured to use virtual-wire B and the
corresponding IO-APIC RTE is masked, that interrupt will be dropped.
thanks,
suresh
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
2012-10-10 0:26 ` Zhang, Lin-Bao (Linux Kernel R&D)
@ 2012-10-11 0:07 ` Suresh Siddha
2012-10-11 3:52 ` Zhang, Lin-Bao (Linux Kernel R&D)
0 siblings, 1 reply; 11+ messages in thread
From: Suresh Siddha @ 2012-10-11 0:07 UTC (permalink / raw)
To: Zhang, Lin-Bao (Linux Kernel R&D)
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, hpa, x86,
a.p.zijlstra, Sakkinen, Jarkko, joerg.roedel, agordeev, yinghai,
stable
On Wed, 2012-10-10 at 00:26 +0000, Zhang, Lin-Bao (Linux Kernel R&D)
wrote:
> So , we can think ,as your patch , during the window , IO-apic is
> useless or we can think IO-APIC doesn't exist ?
> Could you mind please sharing your design details ? thanks very much!
Between the window of interrupt-remapping enabled and the masked IO-APIC
RTE's are configured properly, linux kernel doesn't wait/depend on any
external interrupts.
> > Can you please apply the appended patch to 2.6.32 kernel and see if the issue
> > you mentioned gets fixed? If so, we can ask the -stable and OSV's teams to
> > pick up this fix.
> Yes , it can resolve current issue.
Thanks for testing it out.
I will add the appropriate changelog and send the patch out (to 2.6.32
stable and OSV kernels) with your "Tested-by:" if you are ok.
thanks,
suresh
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
2012-10-11 0:07 ` Suresh Siddha
@ 2012-10-11 3:52 ` Zhang, Lin-Bao (Linux Kernel R&D)
2012-10-11 5:26 ` H. Peter Anvin
0 siblings, 1 reply; 11+ messages in thread
From: Zhang, Lin-Bao (Linux Kernel R&D) @ 2012-10-11 3:52 UTC (permalink / raw)
To: Suresh Siddha
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, hpa, x86,
a.p.zijlstra, Sakkinen, Jarkko, joerg.roedel, agordeev, yinghai,
Zhang, Lin-Bao (Linux Kernel R&D)
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 880 bytes --]
> -----Original Message-----
> From: Suresh Siddha [mailto:suresh.b.siddha@intel.com]
> > > Can you please apply the appended patch to 2.6.32 kernel and see if
> > > the issue you mentioned gets fixed? If so, we can ask the -stable
> > > and OSV's teams to pick up this fix.
> > Yes , it can resolve current issue.
>
> Thanks for testing it out.
>
You are welcome!
> I will add the appropriate changelog and send the patch out (to 2.6.32 stable
> and OSV kernels) with your "Tested-by:" if you are ok.
>
Sure, it is my pleasure .Please go ahead!
BTW , what's OSV kernels ? I can't find its meaning by searching google.
Once your patch has been included by 2.6 git, kindly inform me. Thanks.
> thanks,
> suresh
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
2012-10-11 3:52 ` Zhang, Lin-Bao (Linux Kernel R&D)
@ 2012-10-11 5:26 ` H. Peter Anvin
2012-10-11 15:32 ` Zhang, Lin-Bao (Linux Kernel R&D)
0 siblings, 1 reply; 11+ messages in thread
From: H. Peter Anvin @ 2012-10-11 5:26 UTC (permalink / raw)
To: Zhang, Lin-Bao (Linux Kernel R&D), Suresh Siddha
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, x86,
a.p.zijlstra, Sakkinen, Jarkko, joerg.roedel, agordeev, yinghai
OSV = Operating System Vendor ime. red Hat, SUSE etc.
"Zhang, Lin-Bao (Linux Kernel R&D)" <linbao.zhang@hp.com> wrote:
>
>> -----Original Message-----
>> From: Suresh Siddha [mailto:suresh.b.siddha@intel.com]
>> > > Can you please apply the appended patch to 2.6.32 kernel and see
>if
>> > > the issue you mentioned gets fixed? If so, we can ask the -stable
>> > > and OSV's teams to pick up this fix.
>> > Yes , it can resolve current issue.
>>
>> Thanks for testing it out.
>>
>You are welcome!
>
>> I will add the appropriate changelog and send the patch out (to
>2.6.32 stable
>> and OSV kernels) with your "Tested-by:" if you are ok.
>>
>Sure, it is my pleasure .Please go ahead!
>BTW , what's OSV kernels ? I can't find its meaning by searching
>google.
>Once your patch has been included by 2.6 git, kindly inform me. Thanks.
>
>
>> thanks,
>> suresh
--
Sent from my mobile phone. Please excuse brevity and lack of formatting.
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
2012-10-11 5:26 ` H. Peter Anvin
@ 2012-10-11 15:32 ` Zhang, Lin-Bao (Linux Kernel R&D)
0 siblings, 0 replies; 11+ messages in thread
From: Zhang, Lin-Bao (Linux Kernel R&D) @ 2012-10-11 15:32 UTC (permalink / raw)
To: H. Peter Anvin, Suresh Siddha
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, x86,
a.p.zijlstra, Sakkinen, Jarkko, joerg.roedel, agordeev, yinghai
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 869 bytes --]
> -----Original Message-----
> From: H. Peter Anvin [mailto:hpa@zytor.com]
> Sent: 2012å¹´10æ10æ¥ 23:27
> To: Zhang, Lin-Bao (Linux Kernel R&D); Suresh Siddha
> Cc: linux-kernel@vger.kernel.org; alan@lxorguk.ukuu.org.uk;
> mingo@redhat.com; Croxon, Nigel; tglx@linutronix.de; x86@kernel.org;
> a.p.zijlstra@chello.nl; Sakkinen, Jarkko; joerg.roedel@amd.com;
> agordeev@redhat.com; yinghai@kernel.org
> Subject: RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A
> interrupt during the time window between changing VT-d table base address
> and initializing these VT-d entries(smpboot.c and apic.c )
>
> OSV = Operating System Vendor ime. red Hat, SUSE etc.
>
Thanks Peter very much.
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
2012-10-11 0:01 ` Suresh Siddha
@ 2012-10-11 16:46 ` Zhang, Lin-Bao (Linux Kernel R&D)
0 siblings, 0 replies; 11+ messages in thread
From: Zhang, Lin-Bao (Linux Kernel R&D) @ 2012-10-11 16:46 UTC (permalink / raw)
To: Suresh Siddha
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, hpa, x86,
a.p.zijlstra, Sakkinen, Jarkko, joerg.roedel, agordeev, yinghai,
Zhang, Lin-Bao (Linux Kernel R&D)
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1032 bytes --]
> -----Original Message-----
> From: Suresh Siddha [mailto:suresh.b.siddha@intel.com]
> Sent: 2012å¹´10æ10æ¥ 18:02
> > > So I looked at why you are seeing the problem with v2.6.32 but not
> > > with the recent kernels. And I think I found out the reason.
> >
> > I want to know what masking IO-APIC means?
>
> As the platform is configured to use virtual-wire B and the corresponding
> IO-APIC RTE is masked, that interrupt will be dropped.
>
thanks for your explanation and confirm, I also consulted BIOS guys ,
yes, interrupt will be dropped , software didn't see them,and don't go to handle them.
So we OS don't need to disable 8259A interrupt during the window
, only if 2.6.32 has applied your patch(triggered some actions to mask IO-apic)
That's good. Perfect design . thanks.
Here we go!
---------------
-- Bob(LinBao Zhang)
HP linux kernel enginner
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c )
@ 2012-09-21 7:20 Zhang, Lin-Bao (Linux Kernel R&D)
0 siblings, 0 replies; 11+ messages in thread
From: Zhang, Lin-Bao (Linux Kernel R&D) @ 2012-09-21 7:20 UTC (permalink / raw)
To: Suresh Siddha
Cc: linux-kernel, alan, mingo, Croxon, Nigel, tglx, hpa, x86,
a.p.zijlstra, jarkko.sakkinen, joerg.roedel, agordeev, yinghai,
stable
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 9190 bytes --]
Hi suresh,
Thanks for your reply and review this patch.
I also cc other maintainers of arch/x86/kernel/smpboot.c and arch/x86/kernel/apic/apic.c(getting them by get_maintainer.pl script, hopefully I have not disturbed many people , if yes ,sorry first)
> -----Original Message-----
> From: Suresh Siddha [mailto:suresh.b.siddha@intel.com]
> Sent: 2012å¹´9æ21æ¥ 6:23
> To: Zhang, Lin-Bao (Linux Kernel R&D)
> Cc: linux-kernel@vger.kernel.org; alan@lxorguk.ukuu.org.uk;
> mingo@redhat.com; Croxon, Nigel
> Subject: Re: [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A
> interrupt during the time window between changing VT-d table base address
> and initializing these VT-d entries
>
> On Wed, 2012-09-12 at 07:02 +0000, Zhang, Lin-Bao (ESSN-MCXS-Linux Kernel
> R&D) wrote:
> > Hi all,
> > This defect can be observed when the x2apic setting in BIOS is set to
> > "auto" and the BIOS has virtual wire mode enabled on a power up. This
> > defect was found on a 2.6.32 based kernel.
>
> I assume you are able to reproduce the issue with the latest kernel aswell?
>
In fact , this is what I want to further discussion. Thanks for your comments about 3.x on x2apic.
We can only reproduce this issue on 2.6.x kernel, including RHEL6.1/6.2/6.3 and sles11sp1, they are all of 2.6.xx series.
In 3.x upstream series , we didn't reproduce this problem, I ever tested upstream version : 3.0.0 , 3.0.38 , 3.1.10 ,3.3.8,3.4.4, we can't reproduce it.
But I don't think this can prove that 3.x.x doesn't have potential problem similar with 2.6.x .
By reviewing the 3.x kernel source , I found that 3.xx source have the same design defect ,but we don't know why it doesn't trigger this problem as 2.6 , maybe other part work around this issue , so welcome comments ,we need to know the real reason.
Anyway , from 3.x.x kernel source , it still first change VT-d table base address , after some time, linux kernel then initialize RTEs. So during the window , present bit must 0.
During this window slot , if a interrupt is coming , platform will check VT-d entry 's present bit is 0 , cause non-fatal error and send NMI to OS. By intel's ITP we can clearly watch this error is caused :
0x8000_0022_0000_00F1_0000_0000_0000_0000 -> [22] Bit 103:96: FR Fault Reason is 22h The Present (P) field in the IRTE entry corresponding to the interrupt_index of the interrupt request is Clear. (Appendix A Fault Reason Encodings)
In fact, this error is just non-fatal , if firmware designed well, it should depress this error , I think after some time, VT-d entry has been initialized successfully , this error won't exist again.
I think the direction for kernel source to avoid this problem regardless firmware is :
a) mask all 8259A interrupt -> b) create a new VT-d table ,and initialize all entries (RTEs) -> c) take over BIOS's simple VT-d table by kernel's VT-d table base address --> d) unmask 8259A
thus linux kernel can correctly handle interrupt. I think this should be safe. How do you think about it ?
> What virtual wire mode is it?
> Virtual wire mode-A (where the PIC output is connected to LINT0 of the Local
> APIC) doesn't go through interrupt-remapping and virtual wire mode-B (where
> the PIC output is routed through the IO-APIC RTE) will be completely disabled
> as all the BIOS setup IO-APIC RTE's are masked by the Linux kernel from the
> time we enable interrupt-remapping to the time IO-APIC RTE's are properly
> re-configured by the Linux kernel again.
>
> So I am at a loss to understand what is causing this.
>
Yeah , Virtual wire mob B need to use io-apic .
If no io-apic , this issue will never occur.
> >
> > The kernel code (smpboot.c, apic.c) does not mask 8259A interrupts
> > before changing and initializing the new VT-d table when x2apic
> > virtual wire mode is enable on power up. The Linux Kernel expects
> > virtual wire mode to be disabled when booting and enables it when
> > interrupts are masked.
> >
> > The BIOS code builds a simple VT-d table on power up. While the Linux
> > Kernel boots, it first builds an empty VT-d table and use it. After
> > some time, the Linux Kernel then initializes the IO-APIC redirect
> > table, and then initializes the VT-d entries. The window between
> > initializing the redirect table and the VT-d entries, the 8259A
> > interrupts are not masked. If an interrupt occurs in this window, the
> > Linux Kernel will not find a valid entry for this interrupt. The
> > kernel treats it to be a fatal error and panics. If the error never
> > gets cleared, the Linux kernel continuously print this error:
> > "NMI: IOCK error (debug interrupt?) for reason"
>
> Not sure why we get a NMI instead of a vt-d fault? Perhaps the vt-d fault is
> also getting reported via NMI in this platform?
>
Yes, you are right.
When VT-d entry is Present bit is 0 , it will cause platform non-fatal error , and platform will send NMI(NMI reason is IOCHK,you know NMI can have many reasons) .
Because this non-fatal err exists forever , so platform will send NMI looply to OS , so OS will receive many NMI , so linux kernel will print looply
"NMI: IOCK error (debug interrupt?)" , linux kernel can't do any other things.
Following is error messages : in 2.6.32 kernel , we always reproduce it every time( adding x2apic_phys is reasonable)
------------error logs: -------------------------------------------
IOAPIC id 10 under DRHD base 0xace00000
IOAPIC id 8 under DRHD base 0xa8000000
IOAPIC id 0 under DRHD base 0xa8000000
Enabled IRQ remapping in x2apic mode
NMI: IOCK error (debug interrupt?)
CPU 0
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.32rhel6.2-Bob #1 HP ProLiant DL980 G7
RIP: 0010:[<ffffffff810de39e>] [<ffffffff810de39e>] check_for_new_grace_period+0x2e/0xd0
RSP: 0018:ffff880046003e40 EFLAGS: 00000082
RAX: 0000000000000282 RBX: 0000000000000282 RCX: 0000000000000000
RDX: fffffffffffffed4 RSI: ffff880046011400 RDI: ffffffff81aaf640
RBP: ffff880046003e60 R08: 0000000000989680 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaf640
R13: ffff880046011400 R14: 0000000000000100 R15: 0000000000000009
FS: 0000000000000000(0000) GS:ffff880046000000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a85000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 1, threadinfo ffff88c7ebe4e000, task ffff88086b4694c0)
Stack:
ffff880046011400 ffffffff81aaf640 0000000000000048 0000000000000100
<0> ffff880046003eb0 ffffffff810dedb4 ffffffff81a8ebe0 ffffffff81ea2120
<0> ffff880046003e80 0000000000000001 ffffffff81a830c8 0000000000000048
Call Trace:
<IRQ>
[<ffffffff810dedb4>] __rcu_process_callbacks+0x54/0x330
[<ffffffff810df0da>] rcu_process_callbacks+0x4a/0x50
[<ffffffff81072161>] __do_softirq+0xc1/0x1d0
[<ffffff100e75e>] ? timer_interrupt+0x1e/0x30
[<ffffffff8100c24c>] call_softirq+0x1c/0x30
[<ffffffff8100de85>] do_softirq+0x65/0xa0
[<ffffffff81071f45>] irq_exit+0x85/0x90
[<ffffffff814f4cf5>] do_IRQ+0x75/0xf0
[<ffffffff8100ba53>] ret_from_intr+0x0/0x11
<EOI>
[<ffffffff81c303ec>] ? enable_IR_x2apic+0x18a/0x221
[<ffffffff81c2e189>] native_smp_prepare_cpus+0x143/0x389
[<ffffffff81c1f740>] kernel_init+0x112/0x2f9
[<ffffffff8100c14a>] child_rip+0xa/0x20
[<ffffffff81c1f62e>] ? kernel_init+0x0/0x2f9
[<ffffffff8100c140>] ? child_rip+0x0/0x20
Code: e5 48 83 ec 20 48 89 1c 24 4c 89 64 24 08 4c 89 6c 24 10 4c 89 74 24 18 0f 1f 44 00 00 49 89 f5 9c 58 0f 1f 44 00 00 48 89 c3 fa <66> 0f 1f 44 00 00 31 d2 48 8b 87 c8 a0 00 00 48 39 46 08 74 6c
NMI: IOCK error (debug interrupt?)
> Does your tested kernel has this fix?
> commit 254e42006c893f45bca48f313536fcba12206418
> Author: Suresh Siddha <suresh.b.siddha@intel.com>
> Date: Mon Dec 6 12:26:30 2010 -0800
>
> x86, vt-d: Quirk for masking vtd spec errors to platform error handling
> logic
>
Let me take some time to research it, I think it seems that you would mask/depress VT-d spec errors( for example , The Present (P) field in the IRTE entry corresponding to the interrupt_index of the interrupt request is Clear.)
But I think ,this is just disable error reporting or disable error handling. But in our machine , if we found this error , platform will send NMI to OS.
Maybe other platform don't send NMI to OS.
But for linux kernel , we need to assure no this error occur , not depress error(certainly, if error is non-fatal , we can depress it ; if fatal error , we must stop machine and restart).
For OS , how to differ fatal and non-fatal error ?
> Will you be able to provide the failing kernel log so that I can better
> understand the issue?
>
I have pasted error logs above , if you need all booting log , I can send it to public location ,and give you a link. I don't want to paste it all here, too long. :)
Or need I submit a bug in bugzilla.kernel.org ?
> thanks,
> suresh
>
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-10-11 16:47 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-10-08 4:53 [PATCH] fix x2apic defect that Linux kernel doesn't mask 8259A interrupt during the time window between changing VT-d table base address and initializing these VT-d entries(smpboot.c and apic.c ) Zhang, Lin-Bao (Linux Kernel R&D)
2012-10-08 23:57 ` Suresh Siddha
2012-10-10 0:26 ` Zhang, Lin-Bao (Linux Kernel R&D)
2012-10-11 0:07 ` Suresh Siddha
2012-10-11 3:52 ` Zhang, Lin-Bao (Linux Kernel R&D)
2012-10-11 5:26 ` H. Peter Anvin
2012-10-11 15:32 ` Zhang, Lin-Bao (Linux Kernel R&D)
2012-10-10 23:02 ` Zhang, Lin-Bao (Linux Kernel R&D)
2012-10-11 0:01 ` Suresh Siddha
2012-10-11 16:46 ` Zhang, Lin-Bao (Linux Kernel R&D)
-- strict thread matches above, loose matches on Subject: below --
2012-09-21 7:20 Zhang, Lin-Bao (Linux Kernel R&D)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).