Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Yinghai Lu]

On Sat, Feb 11, 2012 at 3:09 PM, tip-bot for Don Zickus
<dzickus@redhat.com> wrote:
> Commit-ID:  d9bc9be89629445758670220787683e37c93f6c1
> Gitweb:     http://git.kernel.org/tip/d9bc9be89629445758670220787683e37c93f6c1
> Author:     Don Zickus <dzickus@redhat.com>
> AuthorDate: Thu, 9 Feb 2012 16:53:41 -0500
> Committer:  Ingo Molnar <mingo@elte.hu>
> CommitDate: Sat, 11 Feb 2012 15:38:53 +0100
>
> x86/kdump: No need to disable ioapic/lapic in crash path
>
> A customer of ours noticed when their machine crashed, kdump did
> not work but hung instead.  Using their firmware dumping
> solution they grabbed a vmcore and decoded the stacks on the
> cpus.  What they noticed seemed to be a rare deadlock with the
> ioapic_lock.
>
>  CPU4:
>  machine_crash_shutdown
>  -> machine_ops.crash_shutdown
>    -> native_machine_crash_shutdown
>       -> kdump_nmi_shootdown_cpus ------> Send NMI to other CPUs
>       -> disable_IO_APIC
>          -> clear_IO_APIC
>             -> clear_IO_APIC_pin
>                -> ioapic_read_entry
>                   -> spin_lock_irqsave(&ioapic_lock, flags)
>                   ---Infinite loop here---
>
>  CPU0:
>  do_IRQ
>  -> handle_irq
>    -> handle_edge_irq
>        -> ack_apic_edge
>           -> move_native_irq
>               -> mask_IO_APIC_irq
>                  -> mask_IO_APIC_irq_desc
>                     -> spin_lock_irqsave(&ioapic_lock, flags)
>                     ---Receive NMI here after getting spinlock---
>                        -> nmi
>                           -> do_nmi
>                              -> crash_nmi_callback
>                              ---Infinite loop here---
>
> The problem is that although kdump tries to shutdown minimal
> hardware, it still needs to disable the IO APIC.  This requires
> spinlocks which may be held by another cpu.  This other cpu is
> being held infinitely in an NMI context by kdump in order to
> serialize the crashing path.  Instant deadlock.
>
> Eric brought up a point that because the boot code was
> restructured we may not need to disable the io apic any more in
> the crash path.  The original concern that led to the
> development of disable_IO_APIC, was that the jiffies calibration
> on boot up relied on the PIT timer for reference.  Access to the
> PIT required 8259 interrupts to be working.  This wouldn't work
> if the ioapic needed to be configured.  So on panic path, the
> ioapic was reconfigured to use virtual wire mode to allow the 8259 to passthrough.
>
> Those concerns don't hold true now, thanks to the jiffies
> calibration code not needing the PIT.  As a result, we can
> remove this call and simplify the locking needed in the panic
> path.
>
> The same work allowed us to remove the need to disable the local
> apic on shutdown too.  This should allow us to jump to the
> second a little faster.
>
> I tested kdump on an Ivy Bridge platform, a Pentium4 and an old
> athlon that did not have an ioapic.  All three were successful.
>
> I also tested using lkdtm that would use jprobes to panic the
> system when entering do_IRQ.  The idea was to see how the system
> reacted with an interrupt pending in the second kernel.  My
> core2 quad successfully kdump'd 3 times in a row with no issues.
>
> v2: removed the disable lapic code too

with this commit, kdump is not working anymore on my setups with
Nehalem, Westmere, sandbridge.
these setup all have VT-d enabled.


After reverting this commit, kdump is working again.

So assume you need to drop this patch.

Thanks

Yinghai Lu

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Eric W. Biederman]

Yinghai Lu <yinghai@kernel.org> writes:

> On Sat, Feb 11, 2012 at 3:09 PM, tip-bot for Don Zickus
> <dzickus@redhat.com> wrote:
>> Commit-ID:  d9bc9be89629445758670220787683e37c93f6c1
>> Gitweb:     http://git.kernel.org/tip/d9bc9be89629445758670220787683e37c93f6c1
>> Author:     Don Zickus <dzickus@redhat.com>
>> AuthorDate: Thu, 9 Feb 2012 16:53:41 -0500
>> Committer:  Ingo Molnar <mingo@elte.hu>
>> CommitDate: Sat, 11 Feb 2012 15:38:53 +0100
>>
>> x86/kdump: No need to disable ioapic/lapic in crash path
>>
>> A customer of ours noticed when their machine crashed, kdump did
>> not work but hung instead.  Using their firmware dumping
>> solution they grabbed a vmcore and decoded the stacks on the
>> cpus.  What they noticed seemed to be a rare deadlock with the
>> ioapic_lock.
>>
>>  CPU4:
>>  machine_crash_shutdown
>>  -> machine_ops.crash_shutdown
>>    -> native_machine_crash_shutdown
>>       -> kdump_nmi_shootdown_cpus ------> Send NMI to other CPUs
>>       -> disable_IO_APIC
>>          -> clear_IO_APIC
>>             -> clear_IO_APIC_pin
>>                -> ioapic_read_entry
>>                   -> spin_lock_irqsave(&ioapic_lock, flags)
>>                   ---Infinite loop here---
>>
>>  CPU0:
>>  do_IRQ
>>  -> handle_irq
>>    -> handle_edge_irq
>>        -> ack_apic_edge
>>           -> move_native_irq
>>               -> mask_IO_APIC_irq
>>                  -> mask_IO_APIC_irq_desc
>>                     -> spin_lock_irqsave(&ioapic_lock, flags)
>>                     ---Receive NMI here after getting spinlock---
>>                        -> nmi
>>                           -> do_nmi
>>                              -> crash_nmi_callback
>>                              ---Infinite loop here---
>>
>> The problem is that although kdump tries to shutdown minimal
>> hardware, it still needs to disable the IO APIC.  This requires
>> spinlocks which may be held by another cpu.  This other cpu is
>> being held infinitely in an NMI context by kdump in order to
>> serialize the crashing path.  Instant deadlock.
>>
>> Eric brought up a point that because the boot code was
>> restructured we may not need to disable the io apic any more in
>> the crash path.  The original concern that led to the
>> development of disable_IO_APIC, was that the jiffies calibration
>> on boot up relied on the PIT timer for reference.  Access to the
>> PIT required 8259 interrupts to be working.  This wouldn't work
>> if the ioapic needed to be configured.  So on panic path, the
>> ioapic was reconfigured to use virtual wire mode to allow the 8259 to passthrough.
>>
>> Those concerns don't hold true now, thanks to the jiffies
>> calibration code not needing the PIT.  As a result, we can
>> remove this call and simplify the locking needed in the panic
>> path.
>>
>> The same work allowed us to remove the need to disable the local
>> apic on shutdown too.  This should allow us to jump to the
>> second a little faster.
>>
>> I tested kdump on an Ivy Bridge platform, a Pentium4 and an old
>> athlon that did not have an ioapic.  All three were successful.
>>
>> I also tested using lkdtm that would use jprobes to panic the
>> system when entering do_IRQ.  The idea was to see how the system
>> reacted with an interrupt pending in the second kernel.  My
>> core2 quad successfully kdump'd 3 times in a row with no issues.
>>
>> v2: removed the disable lapic code too
>
> with this commit, kdump is not working anymore on my setups with
> Nehalem, Westmere, sandbridge.
> these setup all have VT-d enabled.
>
>
> After reverting this commit, kdump is working again.
>
> So assume you need to drop this patch.

It sounds like there is a bug in ioapic initialization in the context of
VT-d.  Where do you fail?

It would be much better to debug this than to blindly revert this patch,
as this change has the potential to significantly increase the
reliability of the kdump path.

Eric

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Yinghai Lu]

On Sat, Feb 11, 2012 at 7:13 PM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> Yinghai Lu <yinghai@kernel.org> writes:
>> After reverting this commit, kdump is working again.
>>
>> So assume you need to drop this patch.
>
> It sounds like there is a bug in ioapic initialization in the context of
> VT-d.  Where do you fail?
>
before get debug print out from second kernel, the system get reset.

Yinghai

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Eric W. Biederman]

Yinghai Lu <yinghai@kernel.org> writes:

> On Sat, Feb 11, 2012 at 7:13 PM, Eric W. Biederman
> <ebiederm@xmission.com> wrote:
>> Yinghai Lu <yinghai@kernel.org> writes:
>>> After reverting this commit, kdump is working again.
>>>
>>> So assume you need to drop this patch.
>>
>> It sounds like there is a bug in ioapic initialization in the context of
>> VT-d.  Where do you fail?
>>
> before get debug print out from second kernel, the system get reset.

Ouch.

Don can you work with with Yinghai to figure out what is different
between your test configuration and his?

YH did you have early printk enabled?

YH Did I understand you properly that things work if you don't enable
VT-d?   By VT-d are you referring to interrupt remapping?

For anyone watching.  The premise of this patch was that we can boot
the kernel without resorting to legacy tricks that require us to
put the interrupt controllers in PIT mode.

Apparently there is some weird corner case that YH can reproduce that
Don did not have in his test set that YH does that causes things to
fail.

I really don't see any likely candidates when looking at the code.  The
most I can see is some code that we don't run when interrupt remapping
is enabled in disable_IO_APIC.

So I suspect we have a bug in our apic initialization somewhere, but
apic initialization should happen after printk are enabled.  Or at least
after early printks so the reset YH is reporting doesn't make much sense.

Eric

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Yinghai Lu]

On Mon, Feb 13, 2012 at 4:52 AM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> Yinghai Lu <yinghai@kernel.org> writes:
>
>> On Sat, Feb 11, 2012 at 7:13 PM, Eric W. Biederman
>> <ebiederm@xmission.com> wrote:
>>> Yinghai Lu <yinghai@kernel.org> writes:
>>>> After reverting this commit, kdump is working again.
>>>>
>>>> So assume you need to drop this patch.
>>>
>>> It sounds like there is a bug in ioapic initialization in the context of
>>> VT-d.  Where do you fail?
>>>
>> before get debug print out from second kernel, the system get reset.
>
> Ouch.
>
> Don can you work with with Yinghai to figure out what is different
> between your test configuration and his?
>
> YH did you have early printk enabled?

yes.

I always have "console=uart8250,io,0x3f8,115200n8" appended.

also i have local patches that print info from
arch/x86/boot/compressed/misc.c::decompress_kernel()
and
arch/x86/kernel/head64.c::x86_64_start_kernel()

>
> YH Did I understand you properly that things work if you don't enable
> VT-d?

that is default setting for BIOS and OS.

will check if disable that will help.

> By VT-d are you referring to interrupt remapping?

yes, include interrupt rempapping and dma remapping.

>
> For anyone watching.  The premise of this patch was that we can boot
> the kernel without resorting to legacy tricks that require us to
> put the interrupt controllers in PIT mode.
>
> Apparently there is some weird corner case that YH can reproduce that
> Don did not have in his test set that YH does that causes things to
> fail.
>
> I really don't see any likely candidates when looking at the code.  The
> most I can see is some code that we don't run when interrupt remapping
> is enabled in disable_IO_APIC.
>
> So I suspect we have a bug in our apic initialization somewhere, but
> apic initialization should happen after printk are enabled.  Or at least
> after early printks so the reset YH is reporting doesn't make much sense.

will try Don's first version patch that only removing disable_IO_APIC.

Yinghai

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Yinghai Lu]

On Mon, Feb 13, 2012 at 8:51 AM, Yinghai Lu <yinghai@kernel.org> wrote:
>> So I suspect we have a bug in our apic initialization somewhere, but
>> apic initialization should happen after printk are enabled.  Or at least
>> after early printks so the reset YH is reporting doesn't make much sense.
>
> will try Don's first version patch that only removing disable_IO_APIC.

first version patch (only removing disable_IO_APIC) is working.

Thanks

Yinghai

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Mon, Feb 13, 2012 at 10:16:00AM -0800, Yinghai Lu wrote:
> On Mon, Feb 13, 2012 at 8:51 AM, Yinghai Lu <yinghai@kernel.org> wrote:
> >> So I suspect we have a bug in our apic initialization somewhere, but
> >> apic initialization should happen after printk are enabled.  Or at least
> >> after early printks so the reset YH is reporting doesn't make much sense.
> >
> > will try Don's first version patch that only removing disable_IO_APIC.
> 
> first version patch (only removing disable_IO_APIC) is working.

So I think I figured it out.  I went through and commented out code in
disable_local_APIC until I narrowed it down to the piece of code that
needs to be disabled for it to work.

Surprise, surprise... its LVTPC or perf! :-)  Actually it is the
nmi_watchdog which uses perf.  My theory is NMIs are not disabled and one
is generated by the local apic during decompression (just bad timing) and
*splat*.

Yinghai, you can probably prove this by

echo 0 > /proc/sys/kernel/nmi_watchdog

then do your kdump crash test.

At least that test worked for me.

So either we explicitly shutdown perf or just mask off LVTPC in a modified
disable_local_APIC?

Eric, thoughts, preferences?

Cheers,
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Yinghai Lu]

On Thu, Feb 16, 2012 at 9:27 AM, Don Zickus <dzickus@redhat.com> wrote:

> So I think I figured it out.  I went through and commented out code in
> disable_local_APIC until I narrowed it down to the piece of code that
> needs to be disabled for it to work.
>
> Surprise, surprise... its LVTPC or perf! :-)  Actually it is the
> nmi_watchdog which uses perf.  My theory is NMIs are not disabled and one
> is generated by the local apic during decompression (just bad timing) and
> *splat*.
>
> Yinghai, you can probably prove this by
>
> echo 0 > /proc/sys/kernel/nmi_watchdog
>
> then do your kdump crash test.

yes.  that will make kdump crash working.

Yinghai

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Thu, Feb 16, 2012 at 01:53:29PM -0800, Yinghai Lu wrote:
> On Thu, Feb 16, 2012 at 9:27 AM, Don Zickus <dzickus@redhat.com> wrote:
> 
> > So I think I figured it out.  I went through and commented out code in
> > disable_local_APIC until I narrowed it down to the piece of code that
> > needs to be disabled for it to work.
> >
> > Surprise, surprise... its LVTPC or perf! :-)  Actually it is the
> > nmi_watchdog which uses perf.  My theory is NMIs are not disabled and one
> > is generated by the local apic during decompression (just bad timing) and
> > *splat*.
> >
> > Yinghai, you can probably prove this by
> >
> > echo 0 > /proc/sys/kernel/nmi_watchdog
> >
> > then do your kdump crash test.
> 
> yes.  that will make kdump crash working.

Cool.  Thanks.

Eric,

Just let me know how you want to handle disabling NMIs in the kexec in
panic shutdown case.

Cheers,
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Eric W. Biederman]

Don Zickus <dzickus@redhat.com> writes:

> On Thu, Feb 16, 2012 at 01:53:29PM -0800, Yinghai Lu wrote:
>> On Thu, Feb 16, 2012 at 9:27 AM, Don Zickus <dzickus@redhat.com> wrote:
>> 
>> > So I think I figured it out.  I went through and commented out code in
>> > disable_local_APIC until I narrowed it down to the piece of code that
>> > needs to be disabled for it to work.
>> >
>> > Surprise, surprise... its LVTPC or perf! :-)  Actually it is the
>> > nmi_watchdog which uses perf.  My theory is NMIs are not disabled and one
>> > is generated by the local apic during decompression (just bad timing) and
>> > *splat*.
>> >
>> > Yinghai, you can probably prove this by
>> >
>> > echo 0 > /proc/sys/kernel/nmi_watchdog
>> >
>> > then do your kdump crash test.
>> 
>> yes.  that will make kdump crash working.
>
> Cool.  Thanks.
>
> Eric,
>
> Just let me know how you want to handle disabling NMIs in the kexec in
> panic shutdown case.

Interesting.  Apparently we have been avoiding this problem by accident.

Thanks for hunting this down.

The options I can see are:
- Ensure we can handle and ignore exceptions like this.
- Always shutoff the lapic and ioapic entries that can generate this.

The good news is that both solutions should be lock free.

The current kernel boot code relies on the assumption that all
interrupts can be disabled.  In this case with nmi's that is clearly not
the case.

The most robust solution and what we want to do long term is to
install an idt that will simply ignore all interrupts until the
idt is replaced.  Since really all we need to deal with is the NMI
vector, which is vector #2, we can have a very small interrupt
descriptor table.

Unfortunately we go through some cpu mode switches in /sbin/kexec,
allowing us to enter the kernels 32bit entry point before we
run the decompresser, so at first glance both /sbin/kexec and the
kernel need to be fixed in a coordinated fashion.

There are two was I can see of removing the need for an exactly
coordinated release.
- Document that an old /sbin/kexec userspace requires you not to
  use the nmi watchdog with modern kernels.
- For a short while simply retain code that stomps the nmi watchdog.
  (But still leaves us open to other kinds of nmi's).

Grr.  Looking a little more closely, all throughout the linux kernel's
boot there is the assumption that any interrupt during boot is a failure
of some kind, and except for an errant nmi watchdog that is a true
assumption.

Don I guess I really have to recommend disabling the nmi watchdog in the
kexec on panic path if we can do so at all reasonably. 

I like the idea of ignoring nmis during boot but that seems to be a
slightly larger project and with little practical improvement in kexec
on panic quality.  Other than getting what should be one or two
i/o writes out of the kexec on panic path.

Eric

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Eric W. Biederman]

ebiederm@xmission.com (Eric W. Biederman) writes:

> Don Zickus <dzickus@redhat.com> writes:
>
>> On Thu, Feb 16, 2012 at 01:53:29PM -0800, Yinghai Lu wrote:
>>> On Thu, Feb 16, 2012 at 9:27 AM, Don Zickus <dzickus@redhat.com> wrote:
>>> 
>>> > So I think I figured it out.  I went through and commented out code in
>>> > disable_local_APIC until I narrowed it down to the piece of code that
>>> > needs to be disabled for it to work.
>>> >
>>> > Surprise, surprise... its LVTPC or perf! :-)  Actually it is the
>>> > nmi_watchdog which uses perf.  My theory is NMIs are not disabled and one
>>> > is generated by the local apic during decompression (just bad timing) and
>>> > *splat*.
>>> >
>>> > Yinghai, you can probably prove this by
>>> >
>>> > echo 0 > /proc/sys/kernel/nmi_watchdog
>>> >
>>> > then do your kdump crash test.
>>> 
>>> yes.  that will make kdump crash working.
>>
>> Cool.  Thanks.
>>
>> Eric,
>>
>> Just let me know how you want to handle disabling NMIs in the kexec in
>> panic shutdown case.
>
> Interesting.  Apparently we have been avoiding this problem by accident.
>
> Thanks for hunting this down.
>
> The options I can see are:
> - Ensure we can handle and ignore exceptions like this.
> - Always shutoff the lapic and ioapic entries that can generate this.
>
> The good news is that both solutions should be lock free.
>
> The current kernel boot code relies on the assumption that all
> interrupts can be disabled.  In this case with nmi's that is clearly not
> the case.
>
> The most robust solution and what we want to do long term is to
> install an idt that will simply ignore all interrupts until the
> idt is replaced.  Since really all we need to deal with is the NMI
> vector, which is vector #2, we can have a very small interrupt
> descriptor table.
>
> Unfortunately we go through some cpu mode switches in /sbin/kexec,
> allowing us to enter the kernels 32bit entry point before we
> run the decompresser, so at first glance both /sbin/kexec and the
> kernel need to be fixed in a coordinated fashion.
>
> There are two was I can see of removing the need for an exactly
> coordinated release.
> - Document that an old /sbin/kexec userspace requires you not to
>   use the nmi watchdog with modern kernels.
> - For a short while simply retain code that stomps the nmi watchdog.
>   (But still leaves us open to other kinds of nmi's).
>
> Grr.  Looking a little more closely, all throughout the linux kernel's
> boot there is the assumption that any interrupt during boot is a failure
> of some kind, and except for an errant nmi watchdog that is a true
> assumption.
>
> Don I guess I really have to recommend disabling the nmi watchdog in the
> kexec on panic path if we can do so at all reasonably. 
>
> I like the idea of ignoring nmis during boot but that seems to be a
> slightly larger project and with little practical improvement in kexec
> on panic quality.  Other than getting what should be one or two
> i/o writes out of the kexec on panic path.

Hmm.

Thinking about it a little more.  The kernel's boot path is inconsistent
with the rest of the kernel's nmi handling.  For anything exception
except an nmi stopping and giving up is fine.  For an nmi it is very
rare for an NMI to signal a truly nasty failure (usually it just means
someone saw a bitflip somewhere), and we can almost always continue
without problem.

I think in practice we really should make our boot path consistent with
the rest of the kernel and ignore/log/report nmis but not fail on them.
Triple faulting (trigger a cpu reset) as we do today just seems like a
recipe for deep and confusing mystery, and not being helpful to the
user.

My preferred fix would be to fix the boot path and /sbin/kexec to ignore
and report nmis as we boot, as that is really what we want long term and
it gives us the most robust solution.

The fix with a guarantee of no more scope creep is to just disable the
nmi watchdog on the kexec on panic path.

Don if you have time please figure out is needed to ignore nmi's and
possible record and/or report them while we boot, otherwise please cook
up a patch that just disables the nmi watchdog wherever we are sending
it from (the local apic or the ioapic).

Eric

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[HATAYAMA Daisuke]

2012/2/17 Eric W. Biederman <ebiederm@xmission.com>:
> ebiederm@xmission.com (Eric W. Biederman) writes:
>
>> Don Zickus <dzickus@redhat.com> writes:
>>
>>> On Thu, Feb 16, 2012 at 01:53:29PM -0800, Yinghai Lu wrote:
>>>> On Thu, Feb 16, 2012 at 9:27 AM, Don Zickus <dzickus@redhat.com> wrote:
>>>>
>>>> > So I think I figured it out.  I went through and commented out code in
>>>> > disable_local_APIC until I narrowed it down to the piece of code that
>>>> > needs to be disabled for it to work.
>>>> >
>>>> > Surprise, surprise... its LVTPC or perf! :-)  Actually it is the
>>>> > nmi_watchdog which uses perf.  My theory is NMIs are not disabled and one
>>>> > is generated by the local apic during decompression (just bad timing) and
>>>> > *splat*.
>>>> >
>>>> > Yinghai, you can probably prove this by
>>>> >
>>>> > echo 0 > /proc/sys/kernel/nmi_watchdog
>>>> >
>>>> > then do your kdump crash test.
>>>>
>>>> yes.  that will make kdump crash working.
>>>
>>> Cool.  Thanks.
>>>
>>> Eric,
>>>
>>> Just let me know how you want to handle disabling NMIs in the kexec in
>>> panic shutdown case.
>>
>> Interesting.  Apparently we have been avoiding this problem by accident.
>>
>> Thanks for hunting this down.
>>
>> The options I can see are:
>> - Ensure we can handle and ignore exceptions like this.
>> - Always shutoff the lapic and ioapic entries that can generate this.
>>
>> The good news is that both solutions should be lock free.
>>
>> The current kernel boot code relies on the assumption that all
>> interrupts can be disabled.  In this case with nmi's that is clearly not
>> the case.
>>
>> The most robust solution and what we want to do long term is to
>> install an idt that will simply ignore all interrupts until the
>> idt is replaced.  Since really all we need to deal with is the NMI
>> vector, which is vector #2, we can have a very small interrupt
>> descriptor table.
>>
>> Unfortunately we go through some cpu mode switches in /sbin/kexec,
>> allowing us to enter the kernels 32bit entry point before we
>> run the decompresser, so at first glance both /sbin/kexec and the
>> kernel need to be fixed in a coordinated fashion.
>>
>> There are two was I can see of removing the need for an exactly
>> coordinated release.
>> - Document that an old /sbin/kexec userspace requires you not to
>>   use the nmi watchdog with modern kernels.
>> - For a short while simply retain code that stomps the nmi watchdog.
>>   (But still leaves us open to other kinds of nmi's).
>>
>> Grr.  Looking a little more closely, all throughout the linux kernel's
>> boot there is the assumption that any interrupt during boot is a failure
>> of some kind, and except for an errant nmi watchdog that is a true
>> assumption.
>>
>> Don I guess I really have to recommend disabling the nmi watchdog in the
>> kexec on panic path if we can do so at all reasonably.
>>
>> I like the idea of ignoring nmis during boot but that seems to be a
>> slightly larger project and with little practical improvement in kexec
>> on panic quality.  Other than getting what should be one or two
>> i/o writes out of the kexec on panic path.
>
> Hmm.
>
> Thinking about it a little more.  The kernel's boot path is inconsistent
> with the rest of the kernel's nmi handling.  For anything exception
> except an nmi stopping and giving up is fine.  For an nmi it is very
> rare for an NMI to signal a truly nasty failure (usually it just means
> someone saw a bitflip somewhere), and we can almost always continue
> without problem.
>
> I think in practice we really should make our boot path consistent with
> the rest of the kernel and ignore/log/report nmis but not fail on them.
> Triple faulting (trigger a cpu reset) as we do today just seems like a
> recipe for deep and confusing mystery, and not being helpful to the
> user.
>
> My preferred fix would be to fix the boot path and /sbin/kexec to ignore
> and report nmis as we boot, as that is really what we want long term and
> it gives us the most robust solution.
>
> The fix with a guarantee of no more scope creep is to just disable the
> nmi watchdog on the kexec on panic path.
>
> Don if you have time please figure out is needed to ignore nmi's and
> possible record and/or report them while we boot, otherwise please cook
> up a patch that just disables the nmi watchdog wherever we are sending
> it from (the local apic or the ioapic).
>
> Eric

A few days ago I investigted the case where system is reseted due to
triple fault caused by the NMI after idt is disabled in
machine_kexec. I didn't see the reset when trigering the kdump with
NMI since the NMI is masked until next iret instruction executed as
described in 6.7.2. Handling Multiple NMIs of Intel Manual Vol.3A.
The NMI mask remains untill the first iret execution on the 2nd
kernel: just the return path of the first kernel_thread invocation for
init process. The exact path is:

  switch_to
  -> ret_from_fork
     -> int_ret_from_sys_call
        -> retint_restore_args
           -> irq_return

At that phase idt is already set up and kdump works.

>From the discussion I interpret kdump doesn't assume this behaviour,
right?

BTW, does anyone know the detail of the NMI mask? I couldn't figure
out about it from the Intel spec more than ``certain hardware
conditions''... I expect those who look at here are x86 NMI experts.

Thanks.
HATAYAMA, Daisuke

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Sat, Feb 18, 2012 at 12:49:16AM +0900, HATAYAMA Daisuke wrote:
> A few days ago I investigted the case where system is reseted due to
> triple fault caused by the NMI after idt is disabled in
> machine_kexec. I didn't see the reset when trigering the kdump with
> NMI since the NMI is masked until next iret instruction executed as
> described in 6.7.2. Handling Multiple NMIs of Intel Manual Vol.3A.
> The NMI mask remains untill the first iret execution on the 2nd
> kernel: just the return path of the first kernel_thread invocation for
> init process. The exact path is:

hmm.  So even though the local apic was disabled you still got an NMI?
That could have been from an external NMI.  I forget how that is wired up,
if it goes through the IOAPIC to the Local APIC or directly to the NMI pin
on the cpu.

> 
>   switch_to
>   -> ret_from_fork
>      -> int_ret_from_sys_call
>         -> retint_restore_args
>            -> irq_return
> 
> At that phase idt is already set up and kdump works.
> 
> From the discussion I interpret kdump doesn't assume this behaviour,
> right?

probably not.

> 
> BTW, does anyone know the detail of the NMI mask? I couldn't figure
> out about it from the Intel spec more than ``certain hardware
> conditions''... I expect those who look at here are x86 NMI experts.

I don't understand the question.

Cheers,
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[HATAYAMA Daisuke]

From: Don Zickus <dzickus@redhat.com>
Subject: Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path
Date: Fri, 17 Feb 2012 15:18:42 -0500

> On Sat, Feb 18, 2012 at 12:49:16AM +0900, HATAYAMA Daisuke wrote:
>> A few days ago I investigted the case where system is reseted due to
>> triple fault caused by the NMI after idt is disabled in
>> machine_kexec. I didn't see the reset when trigering the kdump with
>> NMI since the NMI is masked until next iret instruction executed as
>> described in 6.7.2. Handling Multiple NMIs of Intel Manual Vol.3A.
>> The NMI mask remains untill the first iret execution on the 2nd
>> kernel: just the return path of the first kernel_thread invocation for
>> init process. The exact path is:
> 
> hmm.  So even though the local apic was disabled you still got an NMI?
> That could have been from an external NMI.  I forget how that is wired up,
> if it goes through the IOAPIC to the Local APIC or directly to the NMI pin
> on the cpu.
> 

Please don't confused. I used RHEL kernels based on 2.6.18 and
2.6.32. I didn't use the patch disabling local apic.

>> 
>>   switch_to
>>   -> ret_from_fork
>>      -> int_ret_from_sys_call
>>         -> retint_restore_args
>>            -> irq_return
>> 
>> At that phase idt is already set up and kdump works.
>> 
>> From the discussion I interpret kdump doesn't assume this behaviour,
>> right?
> 
> probably not.
> 

Thanks.

>> 
>> BTW, does anyone know the detail of the NMI mask? I couldn't figure
>> out about it from the Intel spec more than ``certain hardware
>> conditions''... I expect those who look at here are x86 NMI experts.
> 
> I don't understand the question.
> 
> Cheers,
> Don
> 

Fig 10-4 explaining Local APIC Structure says INIT/NMI/SMI are
directly sent to CPU Core, but the later part of this route is not
explained formally anyware. Only the explanation is the sentence in
6.7 Nonmaskable Interrupt (NMI):

  The processor also invokes certain hardware conditions to insure
  that no other interrupts, including NMI interrupts, are received
  until the NMI handler has completed executing.

I'm just wondering if this is explained more formally anyware.

Thanks.
HATAYAMA, Daisuke

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Mon, Feb 20, 2012 at 02:17:33PM +0900, HATAYAMA Daisuke wrote:
> From: Don Zickus <dzickus@redhat.com>
> Subject: Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path
> Date: Fri, 17 Feb 2012 15:18:42 -0500
> 
> > On Sat, Feb 18, 2012 at 12:49:16AM +0900, HATAYAMA Daisuke wrote:
> >> A few days ago I investigted the case where system is reseted due to
> >> triple fault caused by the NMI after idt is disabled in
> >> machine_kexec. I didn't see the reset when trigering the kdump with
> >> NMI since the NMI is masked until next iret instruction executed as
> >> described in 6.7.2. Handling Multiple NMIs of Intel Manual Vol.3A.
> >> The NMI mask remains untill the first iret execution on the 2nd
> >> kernel: just the return path of the first kernel_thread invocation for
> >> init process. The exact path is:
> > 
> > hmm.  So even though the local apic was disabled you still got an NMI?
> > That could have been from an external NMI.  I forget how that is wired up,
> > if it goes through the IOAPIC to the Local APIC or directly to the NMI pin
> > on the cpu.
> > 
> 
> Please don't confused. I used RHEL kernels based on 2.6.18 and
> 2.6.32. I didn't use the patch disabling local apic.

Sure.  Those kernels should be using the 'disable_local_APIC' code.  My
patch just removed that call, IOW it stops disabling local apic or a
simpler way is to say it keeps the local apic enabled.

My question stills stands then, you might have experienced an external
NMI, but I am not entirely sure.


> 
> >> 
> >>   switch_to
> >>   -> ret_from_fork
> >>      -> int_ret_from_sys_call
> >>         -> retint_restore_args
> >>            -> irq_return
> >> 
> >> At that phase idt is already set up and kdump works.
> >> 
> >> From the discussion I interpret kdump doesn't assume this behaviour,
> >> right?
> > 
> > probably not.
> > 
> 
> Thanks.
> 
> >> 
> >> BTW, does anyone know the detail of the NMI mask? I couldn't figure
> >> out about it from the Intel spec more than ``certain hardware
> >> conditions''... I expect those who look at here are x86 NMI experts.
> > 
> > I don't understand the question.
> > 
> > Cheers,
> > Don
> > 
> 
> Fig 10-4 explaining Local APIC Structure says INIT/NMI/SMI are
> directly sent to CPU Core, but the later part of this route is not
> explained formally anyware. Only the explanation is the sentence in
> 6.7 Nonmaskable Interrupt (NMI):
> 
>   The processor also invokes certain hardware conditions to insure
>   that no other interrupts, including NMI interrupts, are received
>   until the NMI handler has completed executing.
> 
> I'm just wondering if this is explained more formally anyware.

It might be I just don't know where.  I just view the NMI as an exception.
Each cpu exception has a priority.  NMI has a higher priority than
interrupts but a lower priority that say INIT.  Therefore when the cpu
gets an exception it classifies it based on priority.  Higher priorities
will interrupt the current exception, such as NMI, while lower priorities
will wait until the current exception is finished.

To me those would be the hardware conditions, but that is my
interpretation.

Cheers,
Don

> 
> Thanks.
> HATAYAMA, Daisuke
> 

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Fri, Feb 17, 2012 at 04:41:01AM -0800, Eric W. Biederman wrote:
> 
> The fix with a guarantee of no more scope creep is to just disable the
> nmi watchdog on the kexec on panic path.
> 
> Don if you have time please figure out is needed to ignore nmi's and
> possible record and/or report them while we boot, otherwise please cook
> up a patch that just disables the nmi watchdog wherever we are sending
> it from (the local apic or the ioapic).

Can I keep things even simpler?  The original problem was the deadlock
with the ioapic lock.  We fixed that by removing the call to
disable_IO_APIC().  Can we just leave the disable_local_APIC calls in
there for now?  Is there any real harm?

All this rewrite is going to take time which will delay fixing a current
problem with kexec on panic, the ioapic deadlock.

Thoughts?

Cheers,
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Eric W. Biederman]

Don Zickus <dzickus@redhat.com> writes:

> On Fri, Feb 17, 2012 at 04:41:01AM -0800, Eric W. Biederman wrote:
>> 
>> The fix with a guarantee of no more scope creep is to just disable the
>> nmi watchdog on the kexec on panic path.
>> 
>> Don if you have time please figure out is needed to ignore nmi's and
>> possible record and/or report them while we boot, otherwise please cook
>> up a patch that just disables the nmi watchdog wherever we are sending
>> it from (the local apic or the ioapic).
>
> Can I keep things even simpler?  The original problem was the deadlock
> with the ioapic lock.  We fixed that by removing the call to
> disable_IO_APIC().  Can we just leave the disable_local_APIC calls in
> there for now?  Is there any real harm?

> All this rewrite is going to take time which will delay fixing a current
> problem with kexec on panic, the ioapic deadlock.

Hmm.

My apologies I just realized that we can not disable the nmi watchdog
safely in all cases.  To avoid the deadlock we fundamentally can not
write to the io_apic, because the locks are the io_apic write path.
The nmi watchdog can be sourced from either the local apics or the
io_apics.  To disable the nmi_watchdog we need at least potentially
to write io_apic.

So it appears to me that the only reasonable and robust thing we can
do is to ignore nmis in the kexec on panic path.

So it looks to me that the only path forward at this point is to fix
the other bug where an unexpected nmi will kill the kexec on panic boot.

I just took a look at the code in /sbin/kexec and that code does not in
fact change the idt except when we switch to 16bit mode, which we
definitely do not do in the kexec on panic case.  So it appears that we
don't need to coordinate an /sbin/kexec release with a kernel release to
ignore nmis.

In fact it looks like we only need to fix the interrupt descriptors
loaded in machine_kexec_64.c and head64.c to ignore nmis.

At which point we will have fixed two bugs and have a much more reliable
kexec on panic implementation.

Eric

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Fri, Feb 17, 2012 at 07:21:52PM -0800, Eric W. Biederman wrote:
> Don Zickus <dzickus@redhat.com> writes:
> 
> > On Fri, Feb 17, 2012 at 04:41:01AM -0800, Eric W. Biederman wrote:
> >> 
> >> The fix with a guarantee of no more scope creep is to just disable the
> >> nmi watchdog on the kexec on panic path.
> >> 
> >> Don if you have time please figure out is needed to ignore nmi's and
> >> possible record and/or report them while we boot, otherwise please cook
> >> up a patch that just disables the nmi watchdog wherever we are sending
> >> it from (the local apic or the ioapic).
> >
> > Can I keep things even simpler?  The original problem was the deadlock
> > with the ioapic lock.  We fixed that by removing the call to
> > disable_IO_APIC().  Can we just leave the disable_local_APIC calls in
> > there for now?  Is there any real harm?
> 
> > All this rewrite is going to take time which will delay fixing a current
> > problem with kexec on panic, the ioapic deadlock.
> 
> Hmm.
> 
> My apologies I just realized that we can not disable the nmi watchdog
> safely in all cases.  To avoid the deadlock we fundamentally can not
> write to the io_apic, because the locks are the io_apic write path.
> The nmi watchdog can be sourced from either the local apics or the
> io_apics.  To disable the nmi_watchdog we need at least potentially
> to write io_apic.

I am curious where you see the nmi watchdog being sourced from the ioapic?
I thought I removed that code 3 or 4 releases ago.

> 
> So it appears to me that the only reasonable and robust thing we can
> do is to ignore nmis in the kexec on panic path.
> 
> So it looks to me that the only path forward at this point is to fix
> the other bug where an unexpected nmi will kill the kexec on panic boot.
> 
> I just took a look at the code in /sbin/kexec and that code does not in
> fact change the idt except when we switch to 16bit mode, which we
> definitely do not do in the kexec on panic case.  So it appears that we
> don't need to coordinate an /sbin/kexec release with a kernel release to
> ignore nmis.
> 
> In fact it looks like we only need to fix the interrupt descriptors
> loaded in machine_kexec_64.c and head64.c to ignore nmis.
> 
> At which point we will have fixed two bugs and have a much more reliable
> kexec on panic implementation.

Ok.  I'll talk with Vivek about how the can be implemented.

Cheers.
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Eric W. Biederman]

Don Zickus <dzickus@redhat.com> writes:

2> On Fri, Feb 17, 2012 at 07:21:52PM -0800, Eric W. Biederman wrote:
>> Don Zickus <dzickus@redhat.com> writes:
>> 
>> > On Fri, Feb 17, 2012 at 04:41:01AM -0800, Eric W. Biederman wrote:
>> >> 
>> >> The fix with a guarantee of no more scope creep is to just disable the
>> >> nmi watchdog on the kexec on panic path.
>> >> 
>> >> Don if you have time please figure out is needed to ignore nmi's and
>> >> possible record and/or report them while we boot, otherwise please cook
>> >> up a patch that just disables the nmi watchdog wherever we are sending
>> >> it from (the local apic or the ioapic).
>> >
>> > Can I keep things even simpler?  The original problem was the deadlock
>> > with the ioapic lock.  We fixed that by removing the call to
>> > disable_IO_APIC().  Can we just leave the disable_local_APIC calls in
>> > there for now?  Is there any real harm?
>> 
>> > All this rewrite is going to take time which will delay fixing a current
>> > problem with kexec on panic, the ioapic deadlock.
>> 
>> Hmm.
>> 
>> My apologies I just realized that we can not disable the nmi watchdog
>> safely in all cases.  To avoid the deadlock we fundamentally can not
>> write to the io_apic, because the locks are the io_apic write path.
>> The nmi watchdog can be sourced from either the local apics or the
>> io_apics.  To disable the nmi_watchdog we need at least potentially
>> to write io_apic.
>
> I am curious where you see the nmi watchdog being sourced from the ioapic?
> I thought I removed that code 3 or 4 releases ago.

In my memory, and in references to the code in comments in various apic
related code.  I couldn't figure out what the current code was doing and
assumed the implementation was equivalent.  It does look like you
removed the code that used the io_apic.  I still haven't figured
out just how the new implementation works yet.

So maybe in the short term we can safely just stomp the timer that
triggers the nmi watchdog in the local apic.  Over the long term that
feels like it is just asking for trouble.

I wonder if the reason that we have an hpet stomp in that code is
for a similar reason.  Did we ever source nmi's from the hpet timer?

>> So it appears to me that the only reasonable and robust thing we can
>> do is to ignore nmis in the kexec on panic path.
>> 
>> So it looks to me that the only path forward at this point is to fix
>> the other bug where an unexpected nmi will kill the kexec on panic boot.
>> 
>> I just took a look at the code in /sbin/kexec and that code does not in
>> fact change the idt except when we switch to 16bit mode, which we
>> definitely do not do in the kexec on panic case.  So it appears that we
>> don't need to coordinate an /sbin/kexec release with a kernel release to
>> ignore nmis.
>> 
>> In fact it looks like we only need to fix the interrupt descriptors
>> loaded in machine_kexec_64.c and head64.c to ignore nmis.
>> 
>> At which point we will have fixed two bugs and have a much more reliable
>> kexec on panic implementation.
>
> Ok.  I'll talk with Vivek about how the can be implemented.

Thanks.  It really doesn't look very hard.  Just a tiny idt with
an nmi entry that says iret.

Eric

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Tue, Feb 21, 2012 at 12:01:07AM -0800, Eric W. Biederman wrote:
> Don Zickus <dzickus@redhat.com> writes:
> 
> 2> On Fri, Feb 17, 2012 at 07:21:52PM -0800, Eric W. Biederman wrote:
> >> Don Zickus <dzickus@redhat.com> writes:
> >> 
> >> > On Fri, Feb 17, 2012 at 04:41:01AM -0800, Eric W. Biederman wrote:
> >> >> 
> >> >> The fix with a guarantee of no more scope creep is to just disable the
> >> >> nmi watchdog on the kexec on panic path.
> >> >> 
> >> >> Don if you have time please figure out is needed to ignore nmi's and
> >> >> possible record and/or report them while we boot, otherwise please cook
> >> >> up a patch that just disables the nmi watchdog wherever we are sending
> >> >> it from (the local apic or the ioapic).
> >> >
> >> > Can I keep things even simpler?  The original problem was the deadlock
> >> > with the ioapic lock.  We fixed that by removing the call to
> >> > disable_IO_APIC().  Can we just leave the disable_local_APIC calls in
> >> > there for now?  Is there any real harm?
> >> 
> >> > All this rewrite is going to take time which will delay fixing a current
> >> > problem with kexec on panic, the ioapic deadlock.
> >> 
> >> Hmm.
> >> 
> >> My apologies I just realized that we can not disable the nmi watchdog
> >> safely in all cases.  To avoid the deadlock we fundamentally can not
> >> write to the io_apic, because the locks are the io_apic write path.
> >> The nmi watchdog can be sourced from either the local apics or the
> >> io_apics.  To disable the nmi_watchdog we need at least potentially
> >> to write io_apic.
> >
> > I am curious where you see the nmi watchdog being sourced from the ioapic?
> > I thought I removed that code 3 or 4 releases ago.
> 
> In my memory, and in references to the code in comments in various apic
> related code.  I couldn't figure out what the current code was doing and
> assumed the implementation was equivalent.  It does look like you
> removed the code that used the io_apic.  I still haven't figured
> out just how the new implementation works yet.

It is just a client of perf now.  Perf controls the local apic accesses.
A lot simpler now.

> 
> So maybe in the short term we can safely just stomp the timer that
> triggers the nmi watchdog in the local apic.  Over the long term that
> feels like it is just asking for trouble.

I can understand that.

> 
> I wonder if the reason that we have an hpet stomp in that code is
> for a similar reason.  Did we ever source nmi's from the hpet timer?
> 
> >> So it appears to me that the only reasonable and robust thing we can
> >> do is to ignore nmis in the kexec on panic path.
> >> 
> >> So it looks to me that the only path forward at this point is to fix
> >> the other bug where an unexpected nmi will kill the kexec on panic boot.
> >> 
> >> I just took a look at the code in /sbin/kexec and that code does not in
> >> fact change the idt except when we switch to 16bit mode, which we
> >> definitely do not do in the kexec on panic case.  So it appears that we
> >> don't need to coordinate an /sbin/kexec release with a kernel release to
> >> ignore nmis.
> >> 
> >> In fact it looks like we only need to fix the interrupt descriptors
> >> loaded in machine_kexec_64.c and head64.c to ignore nmis.
> >> 
> >> At which point we will have fixed two bugs and have a much more reliable
> >> kexec on panic implementation.
> >
> > Ok.  I'll talk with Vivek about how the can be implemented.
> 
> Thanks.  It really doesn't look very hard.  Just a tiny idt with
> an nmi entry that says iret.

It probably is, except I never hacked on idt code before and my assembly
isn't that good.  I have been trying to find examples to copy from to give
it a try.  So far I was using early_idt_handlers with early_printk to see
if I could capture some printk messages while jumping from the first
kernel to the second kernel (when the other early_idt_handlers would kick
in for the second kernel).

Tips?  Better examples?

Cheers,
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Eric W. Biederman]

Don Zickus <dzickus@redhat.com> writes:

> On Tue, Feb 21, 2012 at 12:01:07AM -0800, Eric W. Biederman wrote:
>> Don Zickus <dzickus@redhat.com> writes:
>> 
>> 2> On Fri, Feb 17, 2012 at 07:21:52PM -0800, Eric W. Biederman wrote:
>> >> Don Zickus <dzickus@redhat.com> writes:
>> >> 
>> >> > On Fri, Feb 17, 2012 at 04:41:01AM -0800, Eric W. Biederman wrote:
>> >> >> 
>> >> >> The fix with a guarantee of no more scope creep is to just disable the
>> >> >> nmi watchdog on the kexec on panic path.
>> >> >> 
>> >> >> Don if you have time please figure out is needed to ignore nmi's and
>> >> >> possible record and/or report them while we boot, otherwise please cook
>> >> >> up a patch that just disables the nmi watchdog wherever we are sending
>> >> >> it from (the local apic or the ioapic).
>> >> >
>> >> > Can I keep things even simpler?  The original problem was the deadlock
>> >> > with the ioapic lock.  We fixed that by removing the call to
>> >> > disable_IO_APIC().  Can we just leave the disable_local_APIC calls in
>> >> > there for now?  Is there any real harm?
>> >> 
>> >> > All this rewrite is going to take time which will delay fixing a current
>> >> > problem with kexec on panic, the ioapic deadlock.
>> >> 
>> >> Hmm.
>> >> 
>> >> My apologies I just realized that we can not disable the nmi watchdog
>> >> safely in all cases.  To avoid the deadlock we fundamentally can not
>> >> write to the io_apic, because the locks are the io_apic write path.
>> >> The nmi watchdog can be sourced from either the local apics or the
>> >> io_apics.  To disable the nmi_watchdog we need at least potentially
>> >> to write io_apic.
>> >
>> > I am curious where you see the nmi watchdog being sourced from the ioapic?
>> > I thought I removed that code 3 or 4 releases ago.
>> 
>> In my memory, and in references to the code in comments in various apic
>> related code.  I couldn't figure out what the current code was doing and
>> assumed the implementation was equivalent.  It does look like you
>> removed the code that used the io_apic.  I still haven't figured
>> out just how the new implementation works yet.
>
> It is just a client of perf now.  Perf controls the local apic accesses.
> A lot simpler now.
>
>> 
>> So maybe in the short term we can safely just stomp the timer that
>> triggers the nmi watchdog in the local apic.  Over the long term that
>> feels like it is just asking for trouble.
>
> I can understand that.
>
>> 
>> I wonder if the reason that we have an hpet stomp in that code is
>> for a similar reason.  Did we ever source nmi's from the hpet timer?
>> 
>> >> So it appears to me that the only reasonable and robust thing we can
>> >> do is to ignore nmis in the kexec on panic path.
>> >> 
>> >> So it looks to me that the only path forward at this point is to fix
>> >> the other bug where an unexpected nmi will kill the kexec on panic boot.
>> >> 
>> >> I just took a look at the code in /sbin/kexec and that code does not in
>> >> fact change the idt except when we switch to 16bit mode, which we
>> >> definitely do not do in the kexec on panic case.  So it appears that we
>> >> don't need to coordinate an /sbin/kexec release with a kernel release to
>> >> ignore nmis.
>> >> 
>> >> In fact it looks like we only need to fix the interrupt descriptors
>> >> loaded in machine_kexec_64.c and head64.c to ignore nmis.
>> >> 
>> >> At which point we will have fixed two bugs and have a much more reliable
>> >> kexec on panic implementation.
>> >
>> > Ok.  I'll talk with Vivek about how the can be implemented.
>> 
>> Thanks.  It really doesn't look very hard.  Just a tiny idt with
>> an nmi entry that says iret.
>
> It probably is, except I never hacked on idt code before and my assembly
> isn't that good.  I have been trying to find examples to copy from to give
> it a try.  So far I was using early_idt_handlers with early_printk to see
> if I could capture some printk messages while jumping from the first
> kernel to the second kernel (when the other early_idt_handlers would kick
> in for the second kernel).
>
> Tips?  Better examples?

That is a particularly good example.  When I took a quick look earlier
that is the first place we reload the idt in the kernel boot so that is
one of two places that needs to be modified.

The other place in the reboot code page (which will be a little tricky
because of it's relocatability) needs to have code that roughly says:

kexec_nmi_handler:
        ; Does this vector get any other kind of parameters and saved
        ; registers that we need to restore?
        ; Different types of exceptions handlers have different rules
        ; that I don't remember off the top of my head. call gates, trap
 	; gates, etc.
	iret

kexec_idt:
	.long 0, 0				; IDT entry 0
	.long 0, 0				; IDT entry 1
	.long something, kexec_nmi_handler 	; IDT entry 2 NMI's

You can write to the kexec_idt from C code and for the nmi handler you
just need to make certain all is good.

The code has to be tested and the proper Intel CPU manual should
probably be consulted to make certain you dot your eyes and cross your
t's but we really don't need much code to change on this path.

Eric

p.s. Apologies for the delay I overlooked this message earlier.

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Fernando Luis Vázquez Cao]

On 03/01/2012 08:19 AM, Eric W. Biederman wrote:

> Don Zickus<dzickus@redhat.com>  writes:
>> It probably is, except I never hacked on idt code before and my assembly
>> isn't that good.  I have been trying to find examples to copy from to give
>> it a try.  So far I was using early_idt_handlers with early_printk to see
>> if I could capture some printk messages while jumping from the first
>> kernel to the second kernel (when the other early_idt_handlers would kick
>> in for the second kernel).
>>
>> Tips?  Better examples?
> That is a particularly good example.  When I took a quick look earlier
> that is the first place we reload the idt in the kernel boot so that is
> one of two places that needs to be modified.

Hi Eric, Don

Sorry for chiming in so late.

We run into the same NMI problems and wrote some patches that tackle
the kernel boot side of things. They have been extensively tested using
qemu-kvm and things seem to be working as expected (after receiving an
early NMI the kernel continues without problem; after the iret there is no
stack corruption or register corruption).

I will be replying to this email with the patches.

Thanks,
Fernando

[PATCH 1/2] boot: ignore early NMIs

[Fernando Luis Vázquez Cao]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: ignore-early-nmis.patch --]
[-- Type: text/x-patch, Size: 3292 bytes --]

Subject: [PATCH] boot: ignore early NMIs

From: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>

NMIs very early in the boot process are rarely critical (usually
it just means that there was a spurious bit flip somewhere in the
hardware, or that this is a kdump kernel and we received an NMI
generated in the previous context), so the current behavior of
halting the system when one occurs is probably a bit over the top.

This patch changes the early IDT so that NMIs are ignored and the
kernel can, hopefully, continue executing other code. Harsher
measures (panic, etc) are defered to the final NMI handler, which
can actually make an informed decision.

This issue presented itself in our environment as seemingly
random hangs in kdump.

Signed-off-by: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>
---

diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head64.c linux-3.3-rc6/arch/x86/kernel/head64.c
--- linux-3.3-rc6-orig/arch/x86/kernel/head64.c	2012-03-07 15:49:01.834241787 +0900
+++ linux-3.3-rc6/arch/x86/kernel/head64.c	2012-03-07 18:39:03.173732875 +0900
@@ -71,7 +71,7 @@ void __init x86_64_start_kernel(char * r
 				(__START_KERNEL & PGDIR_MASK)));
 	BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END);
 
-	/* clear bss before set_intr_gate with early_idt_handler */
+	/* clear bss before set_intr_gate with early_idt_handlers */
 	clear_bss();
 
 	/* Make NULL pointers segfault */
@@ -79,13 +79,8 @@ void __init x86_64_start_kernel(char * r
 
 	max_pfn_mapped = KERNEL_IMAGE_SIZE >> PAGE_SHIFT;
 
-	for (i = 0; i < NUM_EXCEPTION_VECTORS; i++) {
-#ifdef CONFIG_EARLY_PRINTK
+	for (i = 0; i < NUM_EXCEPTION_VECTORS; i++)
 		set_intr_gate(i, &early_idt_handlers[i]);
-#else
-		set_intr_gate(i, early_idt_handler);
-#endif
-	}
 	load_idt((const struct desc_ptr *)&idt_descr);
 
 	if (console_loglevel == 10)
diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head_64.S linux-3.3-rc6/arch/x86/kernel/head_64.S
--- linux-3.3-rc6-orig/arch/x86/kernel/head_64.S	2012-03-07 15:49:01.838241839 +0900
+++ linux-3.3-rc6/arch/x86/kernel/head_64.S	2012-03-07 18:41:21.811516876 +0900
@@ -270,18 +270,29 @@ bad_address:
 	jmp bad_address
 
 	.section ".init.text","ax"
-#ifdef CONFIG_EARLY_PRINTK
 	.globl early_idt_handlers
 early_idt_handlers:
-	i = 0
+	vector = 0
 	.rept NUM_EXCEPTION_VECTORS
-	movl $i, %esi
-	jmp early_idt_handler
-	i = i + 1
+	/*
+	 * NMIs (vector 2) this early in the boot process are rarely critical
+	 * (usually it just means that there was a spurious bit flip somewhere
+	 * in the hardware, or that this is a kdump kernel and we received an
+	 * NMI generated in the previous context), so we ignore them here and
+	 * try to continue (see early_nmi_handler implementation below).
+	 * Harsher measures (panic, etc) are defered to the final NMI handler,
+	 * which can actually make an informed decision.
+	 */
+	.if vector == 2
+	jmp early_nmi_handler
+	.else
+	movl $vector, %esi
+	jmp early_exception_handler
+	.endif
+	vector = vector + 1
 	.endr
-#endif
 
-ENTRY(early_idt_handler)
+early_exception_handler:
 #ifdef CONFIG_EARLY_PRINTK
 	cmpl $2,early_recursion_flag(%rip)
 	jz  1f
@@ -315,6 +326,9 @@ ENTRY(early_idt_handler)
 1:	hlt
 	jmp 1b
 
+early_nmi_handler:
+	iretq
+
 #ifdef CONFIG_EARLY_PRINTK
 early_recursion_flag:
 	.long 0

[PATCH 2/2] boot: add early NMI counter

[Fernando Luis Vázquez Cao]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: add-early-nmi-counter.patch --]
[-- Type: text/x-patch, Size: 2106 bytes --]

Subject: [PATCH] boot: add early NMI counter

From: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>

We currently ignore early NMIs but it would be nice to be able to
know whether they actually occurred.

This patch adds an early NMI counter and exports it trough
/proc/interrupts with the name of ENM (Early NMI).

Signed-off-by: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>
---

diff -urNp linux-3.3-rc6-orig/arch/x86/include/asm/processor.h linux-3.3-rc6/arch/x86/include/asm/processor.h
--- linux-3.3-rc6-orig/arch/x86/include/asm/processor.h	2012-03-07 15:49:01.826241682 +0900
+++ linux-3.3-rc6/arch/x86/include/asm/processor.h	2012-03-07 18:52:27.668054122 +0900
@@ -402,6 +402,7 @@ DECLARE_PER_CPU(char *, irq_stack_ptr);
 DECLARE_PER_CPU(unsigned int, irq_count);
 extern unsigned long kernel_eflags;
 extern asmlinkage void ignore_sysret(void);
+extern unsigned int early_nmi_count;
 #else	/* X86_64 */
 #ifdef CONFIG_CC_STACKPROTECTOR
 /*
diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head_64.S linux-3.3-rc6/arch/x86/kernel/head_64.S
--- linux-3.3-rc6-orig/arch/x86/kernel/head_64.S	2012-03-07 18:51:17.583160114 +0900
+++ linux-3.3-rc6/arch/x86/kernel/head_64.S	2012-03-07 18:54:37.205699422 +0900
@@ -327,6 +327,7 @@ early_exception_handler:
 	jmp 1b
 
 early_nmi_handler:
+	incl early_nmi_count(%rip) # exported as ENM in /proc/interrupts
 	iretq
 
 #ifdef CONFIG_EARLY_PRINTK
@@ -340,6 +341,10 @@ early_idt_ripmsg:
 #endif /* CONFIG_EARLY_PRINTK */
 	.previous
 
+	.globl early_nmi_count
+early_nmi_count:
+	.long 0
+
 #define NEXT_PAGE(name) \
 	.balign	PAGE_SIZE; \
 ENTRY(name)
diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/irq.c linux-3.3-rc6/arch/x86/kernel/irq.c
--- linux-3.3-rc6-orig/arch/x86/kernel/irq.c	2012-03-07 15:49:01.838241839 +0900
+++ linux-3.3-rc6/arch/x86/kernel/irq.c	2012-03-07 18:55:14.078179448 +0900
@@ -125,6 +125,7 @@ int arch_show_interrupts(struct seq_file
 #if defined(CONFIG_X86_IO_APIC)
 	seq_printf(p, "%*s: %10u\n", prec, "MIS", atomic_read(&irq_mis_count));
 #endif
+	seq_printf(p, "%*s: %10u\n", prec, "ENM", early_nmi_count);
 	return 0;
 }
 

Re: [PATCH 2/2] boot: add early NMI counter

[Eric W. Biederman]

Fernando Luis Vázquez Cao <fernando@oss.ntt.co.jp> writes:

> Subject: [PATCH] boot: add early NMI counter
>
> From: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>
>
> We currently ignore early NMIs but it would be nice to be able to
> know whether they actually occurred.
>
> This patch adds an early NMI counter and exports it trough
> /proc/interrupts with the name of ENM (Early NMI).

For a userspace interface I suspect we just want to dump the count
into the NMI counter.

It probably makes sense to have a separate variable in early boot like
you do, but then about the time we setup the normal NMI handler move the
NMI count over and possibly do something like reboot if that is our
policy.

I don't see it making sense to educate userspace for an edge case like
this.

Eric


> Signed-off-by: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>
> ---
>
> diff -urNp linux-3.3-rc6-orig/arch/x86/include/asm/processor.h linux-3.3-rc6/arch/x86/include/asm/processor.h
> --- linux-3.3-rc6-orig/arch/x86/include/asm/processor.h	2012-03-07 15:49:01.826241682 +0900
> +++ linux-3.3-rc6/arch/x86/include/asm/processor.h	2012-03-07 18:52:27.668054122 +0900
> @@ -402,6 +402,7 @@ DECLARE_PER_CPU(char *, irq_stack_ptr);
>  DECLARE_PER_CPU(unsigned int, irq_count);
>  extern unsigned long kernel_eflags;
>  extern asmlinkage void ignore_sysret(void);
> +extern unsigned int early_nmi_count;
>  #else	/* X86_64 */
>  #ifdef CONFIG_CC_STACKPROTECTOR
>  /*
> diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head_64.S linux-3.3-rc6/arch/x86/kernel/head_64.S
> --- linux-3.3-rc6-orig/arch/x86/kernel/head_64.S	2012-03-07 18:51:17.583160114 +0900
> +++ linux-3.3-rc6/arch/x86/kernel/head_64.S	2012-03-07 18:54:37.205699422 +0900
> @@ -327,6 +327,7 @@ early_exception_handler:
>  	jmp 1b
>  
>  early_nmi_handler:
> +	incl early_nmi_count(%rip) # exported as ENM in /proc/interrupts
>  	iretq
>  
>  #ifdef CONFIG_EARLY_PRINTK
> @@ -340,6 +341,10 @@ early_idt_ripmsg:
>  #endif /* CONFIG_EARLY_PRINTK */
>  	.previous
>  
> +	.globl early_nmi_count
> +early_nmi_count:
> +	.long 0
> +
>  #define NEXT_PAGE(name) \
>  	.balign	PAGE_SIZE; \
>  ENTRY(name)
> diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/irq.c linux-3.3-rc6/arch/x86/kernel/irq.c
> --- linux-3.3-rc6-orig/arch/x86/kernel/irq.c	2012-03-07 15:49:01.838241839 +0900
> +++ linux-3.3-rc6/arch/x86/kernel/irq.c	2012-03-07 18:55:14.078179448 +0900
> @@ -125,6 +125,7 @@ int arch_show_interrupts(struct seq_file
>  #if defined(CONFIG_X86_IO_APIC)
>  	seq_printf(p, "%*s: %10u\n", prec, "MIS", atomic_read(&irq_mis_count));
>  #endif
> +	seq_printf(p, "%*s: %10u\n", prec, "ENM", early_nmi_count);
>  	return 0;
>  }
>  

Re: [PATCH 2/2] boot: add early NMI counter

[Fernando Luis Vázquez Cao]

On 03/08/2012 01:50 PM, Eric W. Biederman wrote:
> Fernando Luis Vázquez Cao<fernando@oss.ntt.co.jp>  writes:
>
>> Subject: [PATCH] boot: add early NMI counter
>>
>> From: Fernando Luis Vazquez Cao<fernando@oss.ntt.co.jp>
>>
>> We currently ignore early NMIs but it would be nice to be able to
>> know whether they actually occurred.
>>
>> This patch adds an early NMI counter and exports it trough
>> /proc/interrupts with the name of ENM (Early NMI).
> For a userspace interface I suspect we just want to dump the count
> into the NMI counter.
>
> It probably makes sense to have a separate variable in early boot like
> you do, but then about the time we setup the normal NMI handler move the
> NMI count over and possibly do something like reboot if that is our
> policy.

I am fine with either approach. I guess that is the x86 maintainers'
call. Ingo, Thomas, Peter, what do you think? Should I add a new
entry to /proc/interrupts or move over the early NMI count to the
existing per-cpu NMI counter?

Thanks,
Fernando

Re: [PATCH 1/2] boot: ignore early NMIs

[Eric W. Biederman]

Fernando Luis Vázquez Cao <fernando@oss.ntt.co.jp> writes:

> Subject: [PATCH] boot: ignore early NMIs
>
> From: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>
>
> NMIs very early in the boot process are rarely critical (usually
> it just means that there was a spurious bit flip somewhere in the
> hardware, or that this is a kdump kernel and we received an NMI
> generated in the previous context), so the current behavior of
> halting the system when one occurs is probably a bit over the top.
>
> This patch changes the early IDT so that NMIs are ignored and the
> kernel can, hopefully, continue executing other code. Harsher
> measures (panic, etc) are defered to the final NMI handler, which
> can actually make an informed decision.
>
> This issue presented itself in our environment as seemingly
> random hangs in kdump.
>
> Signed-off-by: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>
> ---
>
> diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head64.c linux-3.3-rc6/arch/x86/kernel/head64.c
> --- linux-3.3-rc6-orig/arch/x86/kernel/head64.c	2012-03-07 15:49:01.834241787 +0900
> +++ linux-3.3-rc6/arch/x86/kernel/head64.c	2012-03-07 18:39:03.173732875 +0900
> @@ -71,7 +71,7 @@ void __init x86_64_start_kernel(char * r
>  				(__START_KERNEL & PGDIR_MASK)));
>  	BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END);
>  
> -	/* clear bss before set_intr_gate with early_idt_handler */
> +	/* clear bss before set_intr_gate with early_idt_handlers */
>  	clear_bss();
>  
>  	/* Make NULL pointers segfault */
> @@ -79,13 +79,8 @@ void __init x86_64_start_kernel(char * r
>  
>  	max_pfn_mapped = KERNEL_IMAGE_SIZE >> PAGE_SHIFT;
>  
> -	for (i = 0; i < NUM_EXCEPTION_VECTORS; i++) {
> -#ifdef CONFIG_EARLY_PRINTK
> +	for (i = 0; i < NUM_EXCEPTION_VECTORS; i++)
>  		set_intr_gate(i, &early_idt_handlers[i]);
> -#else
> -		set_intr_gate(i, early_idt_handler);
> -#endif
> -	}
>  	load_idt((const struct desc_ptr *)&idt_descr);
>  
>  	if (console_loglevel == 10)
> diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head_64.S linux-3.3-rc6/arch/x86/kernel/head_64.S
> --- linux-3.3-rc6-orig/arch/x86/kernel/head_64.S	2012-03-07 15:49:01.838241839 +0900
> +++ linux-3.3-rc6/arch/x86/kernel/head_64.S	2012-03-07 18:41:21.811516876 +0900
> @@ -270,18 +270,29 @@ bad_address:
>  	jmp bad_address
>  
>  	.section ".init.text","ax"
> -#ifdef CONFIG_EARLY_PRINTK
>  	.globl early_idt_handlers
>  early_idt_handlers:
> -	i = 0
> +	vector = 0
>  	.rept NUM_EXCEPTION_VECTORS
> -	movl $i, %esi
> -	jmp early_idt_handler
> -	i = i + 1
> +	/*
> +	 * NMIs (vector 2) this early in the boot process are rarely critical
> +	 * (usually it just means that there was a spurious bit flip somewhere
> +	 * in the hardware, or that this is a kdump kernel and we received an
> +	 * NMI generated in the previous context), so we ignore them here and
> +	 * try to continue (see early_nmi_handler implementation below).
> +	 * Harsher measures (panic, etc) are defered to the final NMI handler,
> +	 * which can actually make an informed decision.
> +	 */
> +	.if vector == 2
> +	jmp early_nmi_handler

Is just a jump and not a move followed by a jump still 10 bytes?
I hate to say it but I think this fails miserably for any exception
after a nmi.

I expect the simplest solution is to modify early_idt_handler to test
for vector == 2.

Doing something less brittle than:
> extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][10];
in segment.h might be a good idea as well.

Eric

> +	.else
> +	movl $vector, %esi
> +	jmp early_exception_handler
> +	.endif
> +	vector = vector + 1
>  	.endr
> -#endif
>  
> -ENTRY(early_idt_handler)
> +early_exception_handler:
>  #ifdef CONFIG_EARLY_PRINTK
>  	cmpl $2,early_recursion_flag(%rip)
>  	jz  1f
> @@ -315,6 +326,9 @@ ENTRY(early_idt_handler)
>  1:	hlt
>  	jmp 1b
>  
> +early_nmi_handler:
> +	iretq
> +
>  #ifdef CONFIG_EARLY_PRINTK
>  early_recursion_flag:
>  	.long 0

Re: [PATCH 1/2] boot: ignore early NMIs

[Fernando Luis Vázquez Cao]

On 03/08/2012 01:41 PM, Eric W. Biederman wrote:
> Fernando Luis Vázquez Cao<fernando@oss.ntt.co.jp>  writes:
>
>> Subject: [PATCH] boot: ignore early NMIs
>>
>> From: Fernando Luis Vazquez Cao<fernando@oss.ntt.co.jp>
>>
>> NMIs very early in the boot process are rarely critical (usually
>> it just means that there was a spurious bit flip somewhere in the
>> hardware, or that this is a kdump kernel and we received an NMI
>> generated in the previous context), so the current behavior of
>> halting the system when one occurs is probably a bit over the top.
>>
>> This patch changes the early IDT so that NMIs are ignored and the
>> kernel can, hopefully, continue executing other code. Harsher
>> measures (panic, etc) are defered to the final NMI handler, which
>> can actually make an informed decision.
>>
>> This issue presented itself in our environment as seemingly
>> random hangs in kdump.
>>
>> Signed-off-by: Fernando Luis Vazquez Cao<fernando@oss.ntt.co.jp>
>> ---
>>
>> diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head64.c linux-3.3-rc6/arch/x86/kernel/head64.c
>> --- linux-3.3-rc6-orig/arch/x86/kernel/head64.c	2012-03-07 15:49:01.834241787 +0900
>> +++ linux-3.3-rc6/arch/x86/kernel/head64.c	2012-03-07 18:39:03.173732875 +0900
>> @@ -71,7 +71,7 @@ void __init x86_64_start_kernel(char * r
>>   				(__START_KERNEL&  PGDIR_MASK)));
>>   	BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses)<= MODULES_END);
>>
>> -	/* clear bss before set_intr_gate with early_idt_handler */
>> +	/* clear bss before set_intr_gate with early_idt_handlers */
>>   	clear_bss();
>>
>>   	/* Make NULL pointers segfault */
>> @@ -79,13 +79,8 @@ void __init x86_64_start_kernel(char * r
>>
>>   	max_pfn_mapped = KERNEL_IMAGE_SIZE>>  PAGE_SHIFT;
>>
>> -	for (i = 0; i<  NUM_EXCEPTION_VECTORS; i++) {
>> -#ifdef CONFIG_EARLY_PRINTK
>> +	for (i = 0; i<  NUM_EXCEPTION_VECTORS; i++)
>>   		set_intr_gate(i,&early_idt_handlers[i]);
>> -#else
>> -		set_intr_gate(i, early_idt_handler);
>> -#endif
>> -	}
>>   	load_idt((const struct desc_ptr *)&idt_descr);
>>
>>   	if (console_loglevel == 10)
>> diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head_64.S linux-3.3-rc6/arch/x86/kernel/head_64.S
>> --- linux-3.3-rc6-orig/arch/x86/kernel/head_64.S	2012-03-07 15:49:01.838241839 +0900
>> +++ linux-3.3-rc6/arch/x86/kernel/head_64.S	2012-03-07 18:41:21.811516876 +0900
>> @@ -270,18 +270,29 @@ bad_address:
>>   	jmp bad_address
>>
>>   	.section ".init.text","ax"
>> -#ifdef CONFIG_EARLY_PRINTK
>>   	.globl early_idt_handlers
>>   early_idt_handlers:
>> -	i = 0
>> +	vector = 0
>>   	.rept NUM_EXCEPTION_VECTORS
>> -	movl $i, %esi
>> -	jmp early_idt_handler
>> -	i = i + 1
>> +	/*
>> +	 * NMIs (vector 2) this early in the boot process are rarely critical
>> +	 * (usually it just means that there was a spurious bit flip somewhere
>> +	 * in the hardware, or that this is a kdump kernel and we received an
>> +	 * NMI generated in the previous context), so we ignore them here and
>> +	 * try to continue (see early_nmi_handler implementation below).
>> +	 * Harsher measures (panic, etc) are defered to the final NMI handler,
>> +	 * which can actually make an informed decision.
>> +	 */
>> +	.if vector == 2
>> +	jmp early_nmi_handler
> Is just a jump and not a move followed by a jump still 10 bytes?
> I hate to say it but I think this fails miserably for any exception
> after a nmi.

Thank you for the heads up! Actually, it was working for the
exceptions after the nmi but with a corrupted esi (vector
number). My original intention was to fill the empty space
with nops but forgot to actually implement it... Sorry about
that. Will fix for the next iteration.

> I expect the simplest solution is to modify early_idt_handler to test
> for vector == 2.

That is precisely what I did on a previous version but that would
involve using registers which need to be saved and restored and
I wanted to avoid using the stack in the NMI path. We would also
need to add a "pushq rsi " in early_idt_handlers which implies
modifying "early_idt_handlers" definition in "segment.h".

If you are OK with it I would like to go with the approach in
the two patches I sent.


> Doing something less brittle than:
>> extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][10];
> in segment.h might be a good idea as well.

Yes, I agree. I will give it some thought.

Re: [PATCH 1/2] boot: ignore early NMIs

[Eric W. Biederman]

Fernando Luis Vázquez Cao <fernando@oss.ntt.co.jp> writes:

>> Is just a jump and not a move followed by a jump still 10 bytes?
>> I hate to say it but I think this fails miserably for any exception
>> after a nmi.
>
> Thank you for the heads up! Actually, it was working for the
> exceptions after the nmi but with a corrupted esi (vector
> number). My original intention was to fill the empty space
> with nops but forgot to actually implement it... Sorry about
> that. Will fix for the next iteration.

Sound good, and thank you very much for tackling this.

>> I expect the simplest solution is to modify early_idt_handler to test
>> for vector == 2.
>
> That is precisely what I did on a previous version but that would
> involve using registers which need to be saved and restored and
> I wanted to avoid using the stack in the NMI path. We would also
> need to add a "pushq rsi " in early_idt_handlers which implies
> modifying "early_idt_handlers" definition in "segment.h".
>
> If you are OK with it I would like to go with the approach in
> the two patches I sent.

I am fine with your approach.  I suggest a big fat comment mentioning
the 10 byte requirement and the register requirement.

Neither one is locally obvious which makes it easy to goof when
modifying the code.

>> Doing something less brittle than:
>>> extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][10];
>> in segment.h might be a good idea as well.
>
> Yes, I agree. I will give it some thought.

Eric

Re: [PATCH 1/2] boot: ignore early NMIs

[Fernando Luis Vázquez Cao]

On 03/09/2012 01:35 AM, Eric W. Biederman wrote:

> I am fine with your approach.  I suggest a big fat comment mentioning
> the 10 byte requirement and the register requirement.
>
> Neither one is locally obvious which makes it easy to goof when
> modifying the code.
Done.



>>> Doing something less brittle than:
>>>> extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][10];
>>> in segment.h might be a good idea as well.
Tackled that one too.

I will be replying to this email with the updated patches.

Thanks,
Fernando

[PATCH 1/3] boot: fortify early_idt_handlers definition

[Fernando Luis Vázquez Cao]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: fortify-early_idt_handlers-definition.patch --]
[-- Type: text/x-patch, Size: 1809 bytes --]

Subject: [PATCH 1/3] boot: fortify early_idt_handlers definition

From: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>

The current definition is too brittle which makes it easy to screw things up
when modifying the code. Add some comments while at it.

Signed-off-by: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>
---

diff -urNp linux-3.3-rc6-orig/arch/x86/include/asm/segment.h linux-3.3-rc6/arch/x86/include/asm/segment.h
--- linux-3.3-rc6-orig/arch/x86/include/asm/segment.h	2012-01-05 08:55:44.000000000 +0900
+++ linux-3.3-rc6/arch/x86/include/asm/segment.h	2012-03-09 17:50:40.965438325 +0900
@@ -210,8 +210,10 @@
 #define TLS_SIZE (GDT_ENTRY_TLS_ENTRIES * 8)
 
 #ifdef __KERNEL__
+#define EARLY_IDT_HANDLER_SIZE 16
 #ifndef __ASSEMBLY__
-extern const char early_idt_handlers[NUM_EXCEPTION_VECTORS][10];
+extern const char
+	early_idt_handlers[NUM_EXCEPTION_VECTORS][EARLY_IDT_HANDLER_SIZE];
 #endif
 #endif
 
diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head_64.S linux-3.3-rc6/arch/x86/kernel/head_64.S
--- linux-3.3-rc6-orig/arch/x86/kernel/head_64.S	2012-03-09 17:48:04.563492864 +0900
+++ linux-3.3-rc6/arch/x86/kernel/head_64.S	2012-03-09 18:06:13.329009305 +0900
@@ -272,11 +272,20 @@ bad_address:
 	.section ".init.text","ax"
 #ifdef CONFIG_EARLY_PRINTK
 	.globl early_idt_handlers
+	.align EARLY_IDT_HANDLER_SIZE
 early_idt_handlers:
 	i = 0
 	.rept NUM_EXCEPTION_VECTORS
 	movl $i, %esi
 	jmp early_idt_handler
+	/*
+	 * early_idt_handlers is treated as a
+	 * [NUM_EXCEPTION_VECTORS][EARLY_IDT_HANDLER_SIZE] array from C code
+	 * so we have to make sure that each handler fits in its
+	 * EARLY_IDT_HANDLER_SIZE bytes long slot. Handlers shorter than that
+	 * will be taken care of by the align directive below.
+	 */
+	.align EARLY_IDT_HANDLER_SIZE
 	i = i + 1
 	.endr
 #endif

[PATCH 2/3] boot: ignore early NMIs

[Fernando Luis Vázquez Cao]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: ignore-early-nmis.patch --]
[-- Type: text/x-patch, Size: 3691 bytes --]

Subject: [PATCH 2/3] boot: ignore early NMIs

From: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>

NMIs very early in the boot process are rarely critical (usually
it just means that there was a spurious bit flip somewhere in the
hardware, or that this is a kdump kernel and we received an NMI
generated in the previous context), so the current behavior of
halting the system when one occurs is probably a bit over the top.

This patch changes the early IDT so that NMIs are ignored and the
kernel can, hopefully, continue executing other code. Harsher
measures (panic, etc) are defered to the final NMI handler, which
can actually make an informed decision.

This issue presented itself in our environment as seemingly
random hangs in kdump.

Signed-off-by: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>
---

diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head64.c linux-3.3-rc6/arch/x86/kernel/head64.c
--- linux-3.3-rc6-orig/arch/x86/kernel/head64.c	2012-03-09 17:48:04.563492864 +0900
+++ linux-3.3-rc6/arch/x86/kernel/head64.c	2012-03-09 18:18:41.890237360 +0900
@@ -71,7 +71,7 @@ void __init x86_64_start_kernel(char * r
 				(__START_KERNEL & PGDIR_MASK)));
 	BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END);
 
-	/* clear bss before set_intr_gate with early_idt_handler */
+	/* clear bss before set_intr_gate with early_idt_handlers */
 	clear_bss();
 
 	/* Make NULL pointers segfault */
@@ -79,13 +79,8 @@ void __init x86_64_start_kernel(char * r
 
 	max_pfn_mapped = KERNEL_IMAGE_SIZE >> PAGE_SHIFT;
 
-	for (i = 0; i < NUM_EXCEPTION_VECTORS; i++) {
-#ifdef CONFIG_EARLY_PRINTK
+	for (i = 0; i < NUM_EXCEPTION_VECTORS; i++)
 		set_intr_gate(i, &early_idt_handlers[i]);
-#else
-		set_intr_gate(i, early_idt_handler);
-#endif
-	}
 	load_idt((const struct desc_ptr *)&idt_descr);
 
 	if (console_loglevel == 10)
diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head_64.S linux-3.3-rc6/arch/x86/kernel/head_64.S
--- linux-3.3-rc6-orig/arch/x86/kernel/head_64.S	2012-03-09 18:15:59.656235587 +0900
+++ linux-3.3-rc6/arch/x86/kernel/head_64.S	2012-03-09 18:28:21.377388280 +0900
@@ -270,14 +270,30 @@ bad_address:
 	jmp bad_address
 
 	.section ".init.text","ax"
-#ifdef CONFIG_EARLY_PRINTK
 	.globl early_idt_handlers
 	.align EARLY_IDT_HANDLER_SIZE
 early_idt_handlers:
-	i = 0
+	vector = 0
 	.rept NUM_EXCEPTION_VECTORS
-	movl $i, %esi
-	jmp early_idt_handler
+	/*
+	 * NMIs (vector 2) this early in the boot process are rarely critical
+	 * (usually it just means that there was a spurious bit flip somewhere
+	 * in the hardware, or that this is a kdump kernel and we received an
+	 * NMI generated in the previous context), so we ignore them here and
+	 * try to continue (see early_nmi_handler implementation below).
+	 * Harsher measures (panic, etc) are defered to the final NMI handler,
+	 * which can actually make an informed decision.
+	 */
+	.if vector == 2
+	/*
+	 * We are careful not to use any register to avoid having to use the
+	 * stack in the NMI path.
+	 */
+	jmp early_nmi_handler
+	.else
+	movl $vector, %esi
+	jmp early_exception_handler
+	.endif
 	/*
 	 * early_idt_handlers is treated as a
 	 * [NUM_EXCEPTION_VECTORS][EARLY_IDT_HANDLER_SIZE] array from C code
@@ -286,11 +302,10 @@ early_idt_handlers:
 	 * will be taken care of by the align directive below.
 	 */
 	.align EARLY_IDT_HANDLER_SIZE
-	i = i + 1
+	vector = vector + 1
 	.endr
-#endif
 
-ENTRY(early_idt_handler)
+early_exception_handler:
 #ifdef CONFIG_EARLY_PRINTK
 	cmpl $2,early_recursion_flag(%rip)
 	jz  1f
@@ -324,6 +339,9 @@ ENTRY(early_idt_handler)
 1:	hlt
 	jmp 1b
 
+early_nmi_handler:
+	iretq
+
 #ifdef CONFIG_EARLY_PRINTK
 early_recursion_flag:
 	.long 0

[PATCH 3/3] boot: add early NMI counter

[Fernando Luis Vázquez Cao]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: add-early-nmi-counter.patch --]
[-- Type: text/x-patch, Size: 2253 bytes --]

Subject: [PATCH 3/3] boot: add early NMI counter

From: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>

We currently ignore early NMIs but it would be nice to be able to
know whether they actually occurred.

This patch adds an early NMI counter, whose count is moved over to
the per-cpu NMI counter before the kernel frees the init sections.

Signed-off-by: Fernando Luis Vazquez Cao <fernando@oss.ntt.co.jp>
---

diff -urNp linux-3.3-rc6-orig/arch/x86/include/asm/processor.h linux-3.3-rc6/arch/x86/include/asm/processor.h
--- linux-3.3-rc6-orig/arch/x86/include/asm/processor.h	2012-03-09 17:48:04.547492663 +0900
+++ linux-3.3-rc6/arch/x86/include/asm/processor.h	2012-03-09 18:37:32.060194673 +0900
@@ -402,6 +402,11 @@ DECLARE_PER_CPU(char *, irq_stack_ptr);
 DECLARE_PER_CPU(unsigned int, irq_count);
 extern unsigned long kernel_eflags;
 extern asmlinkage void ignore_sysret(void);
+/*
+ * early_nmi_count is in the .init.text ELF section so it can only be accessed
+ * before the init sections are freed by the kernel.
+ */
+extern unsigned int early_nmi_count;
 #else	/* X86_64 */
 #ifdef CONFIG_CC_STACKPROTECTOR
 /*
diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/head_64.S linux-3.3-rc6/arch/x86/kernel/head_64.S
--- linux-3.3-rc6-orig/arch/x86/kernel/head_64.S	2012-03-09 18:36:18.051281524 +0900
+++ linux-3.3-rc6/arch/x86/kernel/head_64.S	2012-03-09 18:38:47.673129036 +0900
@@ -340,6 +340,7 @@ early_exception_handler:
 	jmp 1b
 
 early_nmi_handler:
+	incl early_nmi_count(%rip)
 	iretq
 
 #ifdef CONFIG_EARLY_PRINTK
@@ -351,6 +352,9 @@ early_idt_msg:
 early_idt_ripmsg:
 	.asciz "RIP %s\n"
 #endif /* CONFIG_EARLY_PRINTK */
+	.globl early_nmi_count
+early_nmi_count:
+	.long 0
 	.previous
 
 #define NEXT_PAGE(name) \
diff -urNp linux-3.3-rc6-orig/arch/x86/kernel/traps.c linux-3.3-rc6/arch/x86/kernel/traps.c
--- linux-3.3-rc6-orig/arch/x86/kernel/traps.c	2012-03-09 17:48:04.567492912 +0900
+++ linux-3.3-rc6/arch/x86/kernel/traps.c	2012-03-09 18:39:23.665574123 +0900
@@ -719,6 +719,7 @@ void __init trap_init(void)
 	x86_init.irqs.trap_init();
 
 #ifdef CONFIG_X86_64
+	percpu_add(irq_stat.__nmi_count, early_nmi_count);
 	memcpy(&nmi_idt_table, &idt_table, IDT_ENTRIES * 16);
 	set_nmi_gate(1, &debug);
 	set_nmi_gate(3, &int3);

Re: [PATCH 1/2] boot: ignore early NMIs

[H. Peter Anvin]

Okay, I need to ask this...

Is there a reason to not just simply block these NMIs during the kexec
sequence?

	-hpa


-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: [PATCH 1/2] boot: ignore early NMIs

[Fernando Luis Vázquez Cao]

On 03/10/2012 05:52 AM, H. Peter Anvin wrote:

> Is there a reason to not just simply block these NMIs during the kexec
> sequence?
Ok, some background:

In the reboot path to the kdump kernel we disable local interrupts
and the APICs in native_machine_crash_shutdown() and reset the IDT
in machine_kexec(), which leaves an in valid IDT installed.

However, disabling the I/O APIC involves taking a lock, which in
the event of a crash can is racy and can lead to a deadlock. To
solve this issue Don wrote a patch that left the I/O APICs and
the LAPIC of the crashing CPU untouched in the kdump reboot path,
but this seemed to cause mysterious reboots in some systems.
It turned out that an NMI coming from the perf based hardlockup
detector was causing the system to triple fault. If a NMI happens
to arrive in the window between the invalidation of the IDT in
machine_kexec() and the configuration of the final IDT we will be
in big trouble. In particular, the system will either triple fault
or halt, depending on whether the NMI arrived before or after
installing the early IDT.

To tackle this issue we can either stop the hardlockup detector
or disable the LAPIC (the NMIs needed by x86's hardlockup detector
are generated using performance counters in the LAPIC), leaving
the I/O APICs untouched. The second is simpler and I think it
is the approach Don took to fix this issue in RHEL kernels.

Unfortunately, this is not enough, we are still exposed to external
NMIs not routed through the LAPIC. In other words, we have to make
sure that we always have and IDT that is able to handle NMIs without
seemingly random reboots and lockups. To achieve this goal we need
to fix machine_kexec() and the early IDT handlers. The current patch
set takes care of the latter.

- Fernando

Re: [PATCH 1/2] boot: ignore early NMIs

[H. Peter Anvin]

On 03/11/2012 10:43 PM, Fernando Luis Vázquez Cao wrote:

> 
> To tackle this issue we can either stop the hardlockup detector
> or disable the LAPIC (the NMIs needed by x86's hardlockup detector
> are generated using performance counters in the LAPIC), leaving
> the I/O APICs untouched. The second is simpler and I think it
> is the approach Don took to fix this issue in RHEL kernels.
> 
> Unfortunately, this is not enough, we are still exposed to external
> NMIs not routed through the LAPIC. In other words, we have to make
> sure that we always have and IDT that is able to handle NMIs without
> seemingly random reboots and lockups. To achieve this goal we need
> to fix machine_kexec() and the early IDT handlers. The current patch
> set takes care of the latter.
> 

The only source of NMIs other than the LAPIC should be the system error
which can be disabled through the RTC port, so I think your second
paragraph here is way more mechanism than you need for very little gain.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: [PATCH 1/2] boot: ignore early NMIs

[Fernando Luis Vázquez Cao]

On 03/12/2012 02:49 PM, H. Peter Anvin wrote:

> On 03/11/2012 10:43 PM, Fernando Luis Vázquez Cao wrote:
>> To tackle this issue we can either stop the hardlockup detector
>> or disable the LAPIC (the NMIs needed by x86's hardlockup detector
>> are generated using performance counters in the LAPIC), leaving
>> the I/O APICs untouched. The second is simpler and I think it
>> is the approach Don took to fix this issue in RHEL kernels.
>>
>> Unfortunately, this is not enough, we are still exposed to external
>> NMIs not routed through the LAPIC. In other words, we have to make
>> sure that we always have and IDT that is able to handle NMIs without
>> seemingly random reboots and lockups. To achieve this goal we need
>> to fix machine_kexec() and the early IDT handlers. The current patch
>> set takes care of the latter.
> The only source of NMIs other than the LAPIC should be the system error
> which can be disabled through the RTC port, so I think your second
> paragraph here is way more mechanism than you need for very little gain.

The thing is that we want to avoid playing with hardware in the kdump
reboot patch when we can avoid it, the premise being that it cannot
be accessed without risking a lockup or worse (as the deadlock accessing
the I/O APIC showed). The kernel is crashing after all. What is more,
I forgot to mention that the long term goal is to leave the LAPIC
untouched too (we really want to keep the number of things we do in the
context of the crashing kernel to the bare minimum), so we would still
need to fix the early IDT.

My patch set just installs a special handler for the NMI case so I think
it is pretty simple and self contained.

Another reason to apply these patches is to be consistent with the rest
of the kernel. Spurious NMIs that would have been ignored after installing
the final IDT would cause the system to halt if they happen
to arrive while the early IDT is in place.

- Fernando

Re: [PATCH 1/2] boot: ignore early NMIs

[Vivek Goyal]

On Mon, Mar 12, 2012 at 03:14:20PM +0900, Fernando Luis Vázquez Cao wrote:

[..]
> The thing is that we want to avoid playing with hardware in the kdump
> reboot patch when we can avoid it, the premise being that it cannot
> be accessed without risking a lockup or worse (as the deadlock accessing
> the I/O APIC showed).

I think there needs to be a limit to being paranoid. On one hand people
want to run panic notifiers, all the kmsg_dump() hooks in panic path, and
on the other hand we are afraid of even disabling LAPIC.

I personally think that disabling LAPIC is reasonably practical solution
to the problem until and unless somebody shows that it deadlocks easily.

Thanks
Vivek

Re: [PATCH 1/2] boot: ignore early NMIs

[Eric W. Biederman]

Vivek Goyal <vgoyal@redhat.com> writes:

> On Mon, Mar 12, 2012 at 03:14:20PM +0900, Fernando Luis Vázquez Cao wrote:
>
> [..]
>> The thing is that we want to avoid playing with hardware in the kdump
>> reboot patch when we can avoid it, the premise being that it cannot
>> be accessed without risking a lockup or worse (as the deadlock accessing
>> the I/O APIC showed).
>
> I think there needs to be a limit to being paranoid. On one hand people
> want to run panic notifiers, all the kmsg_dump() hooks in panic path, and
> on the other hand we are afraid of even disabling LAPIC.

And the kmsg_dump code and the panic notifiers aren't being run.  Having
seen some of their failure modes being patched up recently (Adding and
removing sysfs files!!!!) I'm very comfortable with the level of
paranoia.

It has been proven time and time again that the more you do in the
failing kernel that the greater your likely-hood of not getting your
failure information out.

> I personally think that disabling LAPIC is reasonably practical solution
> to the problem until and unless somebody shows that it deadlocks
> easily.

Disabling NMI generation in the LAPIC is fine, and for the short term
I don't even have a problem with disabling the entire LAPIC as all of
our platforms seem to have code for completely reprogramming it.

At the same time there have been cases like the i8259 routed through
the ExtInt pin of the lapci that we haven't been given programming
information about and that if we want to work we should avoid touching.

Furthermore we have two reported cases of people experiencing real NMIs
on the kdump path.  So we have to assume the presence of the CMOS nmi
disable as well if we are going to unequivocally disable NMIs.

Given the variety of x86 hardware today and the growing variety of x86
hardware tomorrow we are going to be fixing this until we can actually
handle the NMIs.  Hardware designers are unfortunately creative enough
that we aren't going to think of everything.  Given that it is has taken
us almost a decade to realize that there actually is a real world
problem  I'm not too keen on a solution that is just good enough to
fix a small problem.

I would love it if x86 had an architectural NMI off switch but with
Intel pushing EFI and the removal of the cmos clock x86 no longer
has an always available NMI off switch.

Furthermore handling of NMI is not hard it is just a little tricky,
to test.

Eric

Re: [PATCH 1/2] boot: ignore early NMIs

[Vivek Goyal]

On Mon, Mar 12, 2012 at 12:02:06PM -0700, Eric W. Biederman wrote:
[..]
> 
> > I personally think that disabling LAPIC is reasonably practical solution
> > to the problem until and unless somebody shows that it deadlocks
> > easily.
> 
> Disabling NMI generation in the LAPIC is fine, and for the short term
> I don't even have a problem with disabling the entire LAPIC as all of
> our platforms seem to have code for completely reprogramming it.
> 
> At the same time there have been cases like the i8259 routed through
> the ExtInt pin of the lapci that we haven't been given programming
> information about and that if we want to work we should avoid touching.
> 
> Furthermore we have two reported cases of people experiencing real NMIs
> on the kdump path.  So we have to assume the presence of the CMOS nmi
> disable as well if we are going to unequivocally disable NMIs.

So Don reported NMI problem only after he stopped disable_local_APIC().
His NMIs were being generated from Local APIC. The only question left
is what kind of NMIs Fernando was seeing.

> 
> Given the variety of x86 hardware today and the growing variety of x86
> hardware tomorrow we are going to be fixing this until we can actually
> handle the NMIs.  Hardware designers are unfortunately creative enough
> that we aren't going to think of everything.  Given that it is has taken
> us almost a decade to realize that there actually is a real world
> problem  I'm not too keen on a solution that is just good enough to
> fix a small problem.

Problem might be small but it has been noticed and fix is simple. That's
a different thing that it might not cover all other cases you mention.

And to begin with problem was with ioapic lock and we are blocking the
that fix because we want NMI handling to be cleaned up too.

So atleast Don's ioapic locking fix should be allowed to go in and then
debate can continue that what's the best fix to handle NMIs from all
sort of situations.

Thanks
Vivek

Re: [PATCH 1/2] boot: ignore early NMIs

[H. Peter Anvin]

On 03/12/2012 12:02 PM, Eric W. Biederman wrote:
> 
> Disabling NMI generation in the LAPIC is fine, and for the short term
> I don't even have a problem with disabling the entire LAPIC as all of
> our platforms seem to have code for completely reprogramming it.
> 
> At the same time there have been cases like the i8259 routed through
> the ExtInt pin of the lapci that we haven't been given programming
> information about and that if we want to work we should avoid touching.
> 

I'm pretty sure that information is in the SDM (I'm looking at it now.)

You can also block NMI though this route, since external NMI really is
LINT1 and hence is routed through the LAPIC; you can mask it by clearing
bit 16 in LVT LINT1 as described in the SDM.

	-hpa

Re: [PATCH 1/2] boot: ignore early NMIs

[H. Peter Anvin]

On 03/11/2012 11:14 PM, Fernando Luis Vázquez Cao wrote:
> 
> The thing is that we want to avoid playing with hardware in the kdump
> reboot patch when we can avoid it, the premise being that it cannot
> be accessed without risking a lockup or worse (as the deadlock accessing
> the I/O APIC showed). The kernel is crashing after all. What is more,
> I forgot to mention that the long term goal is to leave the LAPIC
> untouched too (we really want to keep the number of things we do in the
> context of the crashing kernel to the bare minimum), so we would still
> need to fix the early IDT.
> 
> My patch set just installs a special handler for the NMI case so I think
> it is pretty simple and self contained.
> 
> Another reason to apply these patches is to be consistent with the rest
> of the kernel. Spurious NMIs that would have been ignored after installing
> the final IDT would cause the system to halt if they happen
> to arrive while the early IDT is in place.
> 

I'm concerned that you're adding failure modes because you don't want to
solve the real problem which is you need to block this at the source.
It is way more than the IDT that has to work (at the very least, you
need the GDT and a working stack) at all times in order for NMIs to be
receivable.  That doesn't address what happens if you're getting an NMI
storm either.

	-hpa

Re: [PATCH 1/2] boot: ignore early NMIs

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> On 03/11/2012 11:14 PM, Fernando Luis Vázquez Cao wrote:
>> 
>> The thing is that we want to avoid playing with hardware in the kdump
>> reboot patch when we can avoid it, the premise being that it cannot
>> be accessed without risking a lockup or worse (as the deadlock accessing
>> the I/O APIC showed). The kernel is crashing after all. What is more,
>> I forgot to mention that the long term goal is to leave the LAPIC
>> untouched too (we really want to keep the number of things we do in the
>> context of the crashing kernel to the bare minimum), so we would still
>> need to fix the early IDT.
>> 
>> My patch set just installs a special handler for the NMI case so I think
>> it is pretty simple and self contained.
>> 
>> Another reason to apply these patches is to be consistent with the rest
>> of the kernel. Spurious NMIs that would have been ignored after installing
>> the final IDT would cause the system to halt if they happen
>> to arrive while the early IDT is in place.
>> 
>
> I'm concerned that you're adding failure modes because you don't want to
> solve the real problem which is you need to block this at the source.
> It is way more than the IDT that has to work (at the very least, you
> need the GDT and a working stack) at all times in order for NMIs to be
> receivable.  That doesn't address what happens if you're getting an NMI
> storm either.

Good criticism.

The basic problem is what do we do when we receive NMIs during the
kernel boot.  Dying mysteriously certainly isn't a good solution.

In the kexec on panic code path we already have a stack and as long
as we can fit the GDT and the LDT on that same page we can have all of
the rest during the entire transition.

After that is basically the kernel's boot code.

The basic problem is which source do we block this at?  How many
sources are their?  And architecturally last I looked x86 no longer
has a NMI disable EFI and similar systems want to get away without
a CMOS legacy clock because designers so often get them wrong.

Eric

Re: [PATCH 1/2] boot: ignore early NMIs

[H. Peter Anvin]

On 03/12/2012 01:01 PM, Eric W. Biederman wrote:
> 
> The basic problem is which source do we block this at?  How many
> sources are their?  And architecturally last I looked x86 no longer
> has a NMI disable EFI and similar systems want to get away without
> a CMOS legacy clock because designers so often get them wrong.
> 

On all processors which have an LAPIC you can block all NMI sources at
the LAPIC.  I think it's safe to assume that if you don't have an LAPIC
-- an ancient system by now -- you have port 70h.

	-hpa

Re: [PATCH 1/2] boot: ignore early NMIs

[H. Peter Anvin]

On 03/12/2012 01:04 PM, H. Peter Anvin wrote:
> On 03/12/2012 01:01 PM, Eric W. Biederman wrote:
>>
>> The basic problem is which source do we block this at?  How many
>> sources are their?  And architecturally last I looked x86 no longer
>> has a NMI disable EFI and similar systems want to get away without
>> a CMOS legacy clock because designers so often get them wrong.
>>
> 
> On all processors which have an LAPIC you can block all NMI sources at
> the LAPIC.  I think it's safe to assume that if you don't have an LAPIC
> -- an ancient system by now -- you have port 70h.
> 

One thing: *disabling* the LAPIC will allow external NMIs coming in on
LINT1 through, since the LAPIC in the disabled state tries to mimic the
no-LAPIC configuration.  So I don't think you want to disable LAPIC as
much as disable the interrupt vectors within.

	-hpa

Re: [PATCH 1/2] boot: ignore early NMIs

[Fernando Luis Vázquez Cao]

On 03/13/2012 05:16 AM, H. Peter Anvin wrote:
> On 03/12/2012 01:04 PM, H. Peter Anvin wrote:
>> On 03/12/2012 01:01 PM, Eric W. Biederman wrote:
>>> The basic problem is which source do we block this at?  How many
>>> sources are their?  And architecturally last I looked x86 no longer
>>> has a NMI disable EFI and similar systems want to get away without
>>> a CMOS legacy clock because designers so often get them wrong.
>>>
>> On all processors which have an LAPIC you can block all NMI sources at
>> the LAPIC.  I think it's safe to assume that if you don't have an LAPIC
>> -- an ancient system by now -- you have port 70h.
>>
> One thing: *disabling* the LAPIC will allow external NMIs coming in on
> LINT1 through, since the LAPIC in the disabled state tries to mimic the
> no-LAPIC configuration.  So I don't think you want to disable LAPIC as
> much as disable the interrupt vectors within.

Does this sound like a plan to get the ball rolling?:

1.- Merge Don's patch to disable the LAPIC in kdump reboot path (this
   fixes a real issue seen in the field, is a net win and certainly not a
   regression - indeed it makes the code simpler because the I/O
   APICs are left untouched).

2.- Merge my patch set to ignore early NMIs (this brings the behavior
   of the boot code in line with what we do in the rest of the kernel
   a we can avoid situations were a spurious NMI causes the kernel
   to halt). The early NMI handler is temporary and the final NMI
   handler installed shortly afterwards will take care of subsequent
   NMIs.

3.- Make sure that spurious NMIs (i.e. NMIs that for whatever reason
   could not be stopped at the source) received during the reboot
   path to the kdump kernel do not cause a triple fault or a system
   lockup. This is under testing.

4.- Identify all the NMI sources and keep them from reaching the CPU
   when it can be done in a race-free way.

Can we get 1 and 2 merged while we work on further improvements
(3 and 4)?

Thanks,
Fernando

Re: [PATCH 1/2] boot: ignore early NMIs

[Don Zickus]

On Tue, Mar 13, 2012 at 11:11:49AM +0900, Fernando Luis Vázquez Cao wrote:
> On 03/13/2012 05:16 AM, H. Peter Anvin wrote:
> >On 03/12/2012 01:04 PM, H. Peter Anvin wrote:
> >>On 03/12/2012 01:01 PM, Eric W. Biederman wrote:
> >>>The basic problem is which source do we block this at?  How many
> >>>sources are their?  And architecturally last I looked x86 no longer
> >>>has a NMI disable EFI and similar systems want to get away without
> >>>a CMOS legacy clock because designers so often get them wrong.
> >>>
> >>On all processors which have an LAPIC you can block all NMI sources at
> >>the LAPIC.  I think it's safe to assume that if you don't have an LAPIC
> >>-- an ancient system by now -- you have port 70h.
> >>
> >One thing: *disabling* the LAPIC will allow external NMIs coming in on
> >LINT1 through, since the LAPIC in the disabled state tries to mimic the
> >no-LAPIC configuration.  So I don't think you want to disable LAPIC as
> >much as disable the interrupt vectors within.
> 
> Does this sound like a plan to get the ball rolling?:
> 
> 1.- Merge Don's patch to disable the LAPIC in kdump reboot path (this
>   fixes a real issue seen in the field, is a net win and certainly not a
>   regression - indeed it makes the code simpler because the I/O
>   APICs are left untouched).

I think you mean my patch to stop disabling the I/O APIC.  That patch
hasn't seen any new issues.  It was the piece that stopped disabling the
LAPIC that opened the doors for NMIs to fault the system.

> 
> 2.- Merge my patch set to ignore early NMIs (this brings the behavior
>   of the boot code in line with what we do in the rest of the kernel
>   a we can avoid situations were a spurious NMI causes the kernel
>   to halt). The early NMI handler is temporary and the final NMI
>   handler installed shortly afterwards will take care of subsequent
>   NMIs.
> 
> 3.- Make sure that spurious NMIs (i.e. NMIs that for whatever reason
>   could not be stopped at the source) received during the reboot
>   path to the kdump kernel do not cause a triple fault or a system
>   lockup. This is under testing.

This will require changes in kexec-tools as the purgatory code zaps the
GDT I believe.  This is going to make a 'complete solution' dependent on
a version of kexec-tools.  Not sure what we want to do there.

> 
> 4.- Identify all the NMI sources and keep them from reaching the CPU
>   when it can be done in a race-free way.

Cheers,
Don

Re: [PATCH 1/2] boot: ignore early NMIs

[Simon Horman]

On Tue, Mar 13, 2012 at 09:33:50AM -0400, Don Zickus wrote:
> On Tue, Mar 13, 2012 at 11:11:49AM +0900, Fernando Luis Vázquez Cao wrote:
> > On 03/13/2012 05:16 AM, H. Peter Anvin wrote:
> > >On 03/12/2012 01:04 PM, H. Peter Anvin wrote:
> > >>On 03/12/2012 01:01 PM, Eric W. Biederman wrote:
> > >>>The basic problem is which source do we block this at?  How many
> > >>>sources are their?  And architecturally last I looked x86 no longer
> > >>>has a NMI disable EFI and similar systems want to get away without
> > >>>a CMOS legacy clock because designers so often get them wrong.
> > >>>
> > >>On all processors which have an LAPIC you can block all NMI sources at
> > >>the LAPIC.  I think it's safe to assume that if you don't have an LAPIC
> > >>-- an ancient system by now -- you have port 70h.
> > >>
> > >One thing: *disabling* the LAPIC will allow external NMIs coming in on
> > >LINT1 through, since the LAPIC in the disabled state tries to mimic the
> > >no-LAPIC configuration.  So I don't think you want to disable LAPIC as
> > >much as disable the interrupt vectors within.
> > 
> > Does this sound like a plan to get the ball rolling?:
> > 
> > 1.- Merge Don's patch to disable the LAPIC in kdump reboot path (this
> >   fixes a real issue seen in the field, is a net win and certainly not a
> >   regression - indeed it makes the code simpler because the I/O
> >   APICs are left untouched).
> 
> I think you mean my patch to stop disabling the I/O APIC.  That patch
> hasn't seen any new issues.  It was the piece that stopped disabling the
> LAPIC that opened the doors for NMIs to fault the system.
> 
> > 
> > 2.- Merge my patch set to ignore early NMIs (this brings the behavior
> >   of the boot code in line with what we do in the rest of the kernel
> >   a we can avoid situations were a spurious NMI causes the kernel
> >   to halt). The early NMI handler is temporary and the final NMI
> >   handler installed shortly afterwards will take care of subsequent
> >   NMIs.
> > 
> > 3.- Make sure that spurious NMIs (i.e. NMIs that for whatever reason
> >   could not be stopped at the source) received during the reboot
> >   path to the kdump kernel do not cause a triple fault or a system
> >   lockup. This is under testing.
> 
> This will require changes in kexec-tools as the purgatory code zaps the
> GDT I believe.  This is going to make a 'complete solution' dependent on
> a version of kexec-tools.  Not sure what we want to do there.

Ouch. I guess that in the event that purgatory needs to be modified
some backwards-compatibility will need to be provided, possibly
by allowing purgatory to switch between two behaviours based on
e.g. a command line parameter.

> > 4.- Identify all the NMI sources and keep them from reaching the CPU
> >   when it can be done in a race-free way.

Re: [PATCH 1/2] boot: ignore early NMIs

[Fernando Luis Vázquez Cao]

On 03/13/2012 03:40 AM, H. Peter Anvin wrote:

> On 03/11/2012 11:14 PM, Fernando Luis Vázquez Cao wrote:
>> The thing is that we want to avoid playing with hardware in the kdump
>> reboot patch when we can avoid it, the premise being that it cannot
>> be accessed without risking a lockup or worse (as the deadlock accessing
>> the I/O APIC showed). The kernel is crashing after all. What is more,
>> I forgot to mention that the long term goal is to leave the LAPIC
>> untouched too (we really want to keep the number of things we do in the
>> context of the crashing kernel to the bare minimum), so we would still
>> need to fix the early IDT.
>>
>> My patch set just installs a special handler for the NMI case so I think
>> it is pretty simple and self contained.
>>
>> Another reason to apply these patches is to be consistent with the rest
>> of the kernel. Spurious NMIs that would have been ignored after installing
>> the final IDT would cause the system to halt if they happen
>> to arrive while the early IDT is in place.
> I'm concerned that you're adding failure modes


This patch set just brings the early IDT in line with what we do after
switching to the final IDT, i.e. we ignore NMIs. The only difference is
that we do not honor panic*nmi and unknown_nmi_panic. As things
stand now the kernel will sometimes mysteriously hang. I really
think that independently of the kdump problem it would be nice
to be consistent in this regard.

> because you don't want to
> solve the real problem which is you need to block this at the source.

Indeed, I want to do both. Try to block NMIs at the source when
possible and install an IDT that ignores NMIs to have our backs
covered (avoid triple faults and lockups). As Eric mentioned
it is not clear that we can always identify and stop all the sources.


> It is way more than the IDT that has to work (at the very least, you
> need the GDT and a working stack) at all times in order for NMIs to be
> receivable.

Of course, and that is what the follow-up patch set does. It just needs
some more testing. The patches I sent make use of the early GDT and
it works as expected.

> That doesn't address what happens if you're getting an NMI
> storm either.

Well, the same applies to the final IDT. As I mentioned before
I think we should also try to stop things at the source when
it is safe (of course, first we need to identify all the sources).

- Fernando

Re: [PATCH 1/2] boot: ignore early NMIs

[Don Zickus]

On Sun, Mar 11, 2012 at 10:49:23PM -0700, H. Peter Anvin wrote:
> On 03/11/2012 10:43 PM, Fernando Luis Vázquez Cao wrote:
> 
> > 
> > To tackle this issue we can either stop the hardlockup detector
> > or disable the LAPIC (the NMIs needed by x86's hardlockup detector
> > are generated using performance counters in the LAPIC), leaving
> > the I/O APICs untouched. The second is simpler and I think it
> > is the approach Don took to fix this issue in RHEL kernels.
> > 
> > Unfortunately, this is not enough, we are still exposed to external
> > NMIs not routed through the LAPIC. In other words, we have to make
> > sure that we always have and IDT that is able to handle NMIs without
> > seemingly random reboots and lockups. To achieve this goal we need
> > to fix machine_kexec() and the early IDT handlers. The current patch
> > set takes care of the latter.
> > 
> 
> The only source of NMIs other than the LAPIC should be the system error
> which can be disabled through the RTC port, so I think your second
> paragraph here is way more mechanism than you need for very little gain.

I forgot about the RTC port.  I can't seem to find the documentation for
it, but I believe it was port 0x70?  That would cover external NMIs I
believe.  Leaving the disable_lapic would cover internal NMIs.

I don't know how far do we want to go with installing stub idt handlers
and such.  Honestly, I just wanted to i/o apic race condition fixed.

http://lkml.indiana.edu/hypermail/linux/kernel/1202.3/02533.html

Cheers,
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Vivek Goyal]

On Wed, Mar 07, 2012 at 07:53:16PM +0900, Fernando Luis Vázquez Cao wrote:
> On 03/01/2012 08:19 AM, Eric W. Biederman wrote:
> 
> >Don Zickus<dzickus@redhat.com>  writes:
> >>It probably is, except I never hacked on idt code before and my assembly
> >>isn't that good.  I have been trying to find examples to copy from to give
> >>it a try.  So far I was using early_idt_handlers with early_printk to see
> >>if I could capture some printk messages while jumping from the first
> >>kernel to the second kernel (when the other early_idt_handlers would kick
> >>in for the second kernel).
> >>
> >>Tips?  Better examples?
> >That is a particularly good example.  When I took a quick look earlier
> >that is the first place we reload the idt in the kernel boot so that is
> >one of two places that needs to be modified.
> 
> Hi Eric, Don
> 
> Sorry for chiming in so late.
> 
> We run into the same NMI problems and wrote some patches that tackle
> the kernel boot side of things. They have been extensively tested using
> qemu-kvm and things seem to be working as expected (after receiving an
> early NMI the kernel continues without problem; after the iret there is no
> stack corruption or register corruption).

What happens if NMI happens while we are still in purgatory code?

Thanks
Vivek

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Yinghai Lu]

2012/3/7 Vivek Goyal <vgoyal@redhat.com>:
> On Wed, Mar 07, 2012 at 07:53:16PM +0900, Fernando Luis Vázquez Cao wrote:
>> On 03/01/2012 08:19 AM, Eric W. Biederman wrote:
>>
>> >Don Zickus<dzickus@redhat.com>  writes:
>> >>It probably is, except I never hacked on idt code before and my assembly
>> >>isn't that good.  I have been trying to find examples to copy from to give
>> >>it a try.  So far I was using early_idt_handlers with early_printk to see
>> >>if I could capture some printk messages while jumping from the first
>> >>kernel to the second kernel (when the other early_idt_handlers would kick
>> >>in for the second kernel).
>> >>
>> >>Tips?  Better examples?
>> >That is a particularly good example.  When I took a quick look earlier
>> >that is the first place we reload the idt in the kernel boot so that is
>> >one of two places that needs to be modified.
>>
>> Hi Eric, Don
>>
>> Sorry for chiming in so late.
>>
>> We run into the same NMI problems and wrote some patches that tackle
>> the kernel boot side of things. They have been extensively tested using
>> qemu-kvm and things seem to be working as expected (after receiving an
>> early NMI the kernel continues without problem; after the iret there is no
>> stack corruption or register corruption).
>
> What happens if NMI happens while we are still in purgatory code?
>
yes, you are right. that is too late to set that in arch/x86/kernel/head64.c

i even not get print out "early console in decompress_kernel"
from arch/x86/boot/compressed/misc.c::decompress_kernel()

Yinghai

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Fernando Luis Vázquez Cao]

On 03/08/2012 03:27 AM, Yinghai Lu wrote:

> 2012/3/7 Vivek Goyal<vgoyal@redhat.com>:
>> On Wed, Mar 07, 2012 at 07:53:16PM +0900, Fernando Luis Vázquez Cao wrote:
>>> We run into the same NMI problems and wrote some patches that tackle
>>> the kernel boot side of things. They have been extensively tested using
>>> qemu-kvm and things seem to be working as expected (after receiving an
>>> early NMI the kernel continues without problem; after the iret there is no
>>> stack corruption or register corruption).
>> What happens if NMI happens while we are still in purgatory code?
The system triple faults. As Eric said we need to fix both the reboot path
to kdump and kdump kernel's boot code. The two patches I sent address the
later. I sent these first because I had time to test them properly, fix a
real issue and can be applied independently.



> yes, you are right. that is too late to set that in arch/x86/kernel/head64.c
Of course, the reboot path needs to be fixed too. Now that I solved some
problems I was having with my qemu-kvm test environment (NMIs injected
through the qemu monitor only get devivered to the BSP, etc), I will make
sure that the code I wrote for it works as expected.

In the meantime, I would appreciate it if you could review/ack yesterday's
patches.

Thanks,
Fernando

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[HATAYAMA Daisuke]

From: Fernando Luis Vázquez Cao <fernando@oss.ntt.co.jp>
Subject: Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path
Date: Wed, 07 Mar 2012 19:53:16 +0900

> On 03/01/2012 08:19 AM, Eric W. Biederman wrote:
> 
>> Don Zickus<dzickus@redhat.com>  writes:
>>> It probably is, except I never hacked on idt code before and my
>>> assembly
>>> isn't that good.  I have been trying to find examples to copy from to
>>> give
>>> it a try.  So far I was using early_idt_handlers with early_printk to
>>> see
>>> if I could capture some printk messages while jumping from the first
>>> kernel to the second kernel (when the other early_idt_handlers would
>>> kick
>>> in for the second kernel).
>>>
>>> Tips?  Better examples?
>> That is a particularly good example.  When I took a quick look earlier
>> that is the first place we reload the idt in the kernel boot so that
>> is
>> one of two places that needs to be modified.
> 
> Hi Eric, Don
> 
> Sorry for chiming in so late.
> 
> We run into the same NMI problems and wrote some patches that tackle
> the kernel boot side of things. They have been extensively tested
> using
> qemu-kvm and things seem to be working as expected (after receiving an
> early NMI the kernel continues without problem; after the iret there
> is no
> stack corruption or register corruption).
> 
> I will be replying to this email with the patches.
> 

Should the following also be fixed?

/*
 * Do not allocate memory (or fail in any way) in machine_kexec().
 * We are past the point of no return, committed to rebooting now.
 */
void machine_kexec(struct kimage *image)
{
        unsigned long page_list[PAGES_NR];
        void *control_page;
        int save_ftrace_enabled;
<cut>
        /*
         * The gdt & idt are now invalid.
         * If you want to load them you must set up your own idt & gdt.
         */
        set_gdt(phys_to_virt(0), 0);
        set_idt(phys_to_virt(0), 0);    <--- This

Thanks.
HATAYAMA, Daisuke

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Eric W. Biederman]

HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com> writes:

> From: Fernando Luis Vázquez Cao <fernando@oss.ntt.co.jp>
> Subject: Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path
> Date: Wed, 07 Mar 2012 19:53:16 +0900
>
>> On 03/01/2012 08:19 AM, Eric W. Biederman wrote:
>> 
>>> Don Zickus<dzickus@redhat.com>  writes:
>>>> It probably is, except I never hacked on idt code before and my
>>>> assembly
>>>> isn't that good.  I have been trying to find examples to copy from to
>>>> give
>>>> it a try.  So far I was using early_idt_handlers with early_printk to
>>>> see
>>>> if I could capture some printk messages while jumping from the first
>>>> kernel to the second kernel (when the other early_idt_handlers would
>>>> kick
>>>> in for the second kernel).
>>>>
>>>> Tips?  Better examples?
>>> That is a particularly good example.  When I took a quick look earlier
>>> that is the first place we reload the idt in the kernel boot so that
>>> is
>>> one of two places that needs to be modified.
>> 
>> Hi Eric, Don
>> 
>> Sorry for chiming in so late.
>> 
>> We run into the same NMI problems and wrote some patches that tackle
>> the kernel boot side of things. They have been extensively tested
>> using
>> qemu-kvm and things seem to be working as expected (after receiving an
>> early NMI the kernel continues without problem; after the iret there
>> is no
>> stack corruption or register corruption).
>> 
>> I will be replying to this email with the patches.
>> 
>
> Should the following also be fixed?

Most definitely.  I think Fernando said he was going to tackle that
next.


Eric


> /*
>  * Do not allocate memory (or fail in any way) in machine_kexec().
>  * We are past the point of no return, committed to rebooting now.
>  */
> void machine_kexec(struct kimage *image)
> {
>         unsigned long page_list[PAGES_NR];
>         void *control_page;
>         int save_ftrace_enabled;
> <cut>
>         /*
>          * The gdt & idt are now invalid.
>          * If you want to load them you must set up your own idt & gdt.
>          */
>         set_gdt(phys_to_virt(0), 0);
>         set_idt(phys_to_virt(0), 0);    <--- This
>
> Thanks.
> HATAYAMA, Daisuke

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Ingo Molnar]

* Yinghai Lu <yinghai@kernel.org> wrote:

> On Sat, Feb 11, 2012 at 3:09 PM, tip-bot for Don Zickus
> <dzickus@redhat.com> wrote:
> > Commit-ID:  d9bc9be89629445758670220787683e37c93f6c1
> > Gitweb:     http://git.kernel.org/tip/d9bc9be89629445758670220787683e37c93f6c1
> > Author:     Don Zickus <dzickus@redhat.com>
> > AuthorDate: Thu, 9 Feb 2012 16:53:41 -0500
> > Committer:  Ingo Molnar <mingo@elte.hu>
> > CommitDate: Sat, 11 Feb 2012 15:38:53 +0100
> >
> > x86/kdump: No need to disable ioapic/lapic in crash path
> >
> > A customer of ours noticed when their machine crashed, kdump did
> > not work but hung instead.  Using their firmware dumping
> > solution they grabbed a vmcore and decoded the stacks on the
> > cpus.  What they noticed seemed to be a rare deadlock with the
> > ioapic_lock.
> >
> >  CPU4:
> >  machine_crash_shutdown
> >  -> machine_ops.crash_shutdown
> >    -> native_machine_crash_shutdown
> >       -> kdump_nmi_shootdown_cpus ------> Send NMI to other CPUs
> >       -> disable_IO_APIC
> >          -> clear_IO_APIC
> >             -> clear_IO_APIC_pin
> >                -> ioapic_read_entry
> >                   -> spin_lock_irqsave(&ioapic_lock, flags)
> >                   ---Infinite loop here---
> >
> >  CPU0:
> >  do_IRQ
> >  -> handle_irq
> >    -> handle_edge_irq
> >        -> ack_apic_edge
> >           -> move_native_irq
> >               -> mask_IO_APIC_irq
> >                  -> mask_IO_APIC_irq_desc
> >                     -> spin_lock_irqsave(&ioapic_lock, flags)
> >                     ---Receive NMI here after getting spinlock---
> >                        -> nmi
> >                           -> do_nmi
> >                              -> crash_nmi_callback
> >                              ---Infinite loop here---
> >
> > The problem is that although kdump tries to shutdown minimal
> > hardware, it still needs to disable the IO APIC.  This requires
> > spinlocks which may be held by another cpu.  This other cpu is
> > being held infinitely in an NMI context by kdump in order to
> > serialize the crashing path.  Instant deadlock.
> >
> > Eric brought up a point that because the boot code was
> > restructured we may not need to disable the io apic any more in
> > the crash path.  The original concern that led to the
> > development of disable_IO_APIC, was that the jiffies calibration
> > on boot up relied on the PIT timer for reference.  Access to the
> > PIT required 8259 interrupts to be working.  This wouldn't work
> > if the ioapic needed to be configured.  So on panic path, the
> > ioapic was reconfigured to use virtual wire mode to allow the 8259 to passthrough.
> >
> > Those concerns don't hold true now, thanks to the jiffies
> > calibration code not needing the PIT.  As a result, we can
> > remove this call and simplify the locking needed in the panic
> > path.
> >
> > The same work allowed us to remove the need to disable the local
> > apic on shutdown too.  This should allow us to jump to the
> > second a little faster.
> >
> > I tested kdump on an Ivy Bridge platform, a Pentium4 and an old
> > athlon that did not have an ioapic.  All three were successful.
> >
> > I also tested using lkdtm that would use jprobes to panic the
> > system when entering do_IRQ.  The idea was to see how the system
> > reacted with an interrupt pending in the second kernel.  My
> > core2 quad successfully kdump'd 3 times in a row with no issues.
> >
> > v2: removed the disable lapic code too
> 
> with this commit, kdump is not working anymore on my setups with
> Nehalem, Westmere, sandbridge.
> these setup all have VT-d enabled.
> 
> After reverting this commit, kdump is working again.
> 
> So assume you need to drop this patch.

Dropped the patch, thanks for reporting this.

Don, Eric?

	Ingo

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Sat, Feb 11, 2012 at 05:04:15PM -0800, Yinghai Lu wrote:
> >
> > v2: removed the disable lapic code too
> 
> with this commit, kdump is not working anymore on my setups with
> Nehalem, Westmere, sandbridge.
> these setup all have VT-d enabled.

Hi Yinghai,

Thanks for the report.  Can you attach the .config you were using and what
vendor you were using for Nehalem, Westmere and Sandybridge.  I'll try to
reproduce it in my lab.

Cheers,
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Yinghai Lu]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]

On Mon, Feb 13, 2012 at 7:28 AM, Don Zickus <dzickus@redhat.com> wrote:
> On Sat, Feb 11, 2012 at 05:04:15PM -0800, Yinghai Lu wrote:
>> >
>> > v2: removed the disable lapic code too
>>
>> with this commit, kdump is not working anymore on my setups with
>> Nehalem, Westmere, sandbridge.
>> these setup all have VT-d enabled.
>
> Hi Yinghai,
>
> Thanks for the report.  Can you attach the .config you were using and what
> vendor you were using for Nehalem, Westmere and Sandybridge.  I'll try to
> reproduce it in my lab.

Sure, please check attached one

Yinghai

[-- Attachment #2: config.2012_02_11 --]
[-- Type: application/octet-stream, Size: 91818 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86_64 3.3.0-rc3 Kernel Configuration
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_GPIO=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
# CONFIG_KTIME_SCALAR is not set
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION="-yh"
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
# CONFIG_KERNEL_GZIP is not set
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
CONFIG_KERNEL_XZ=y
# CONFIG_KERNEL_LZO is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
# CONFIG_FHANDLE is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
# CONFIG_TASK_XACCT is not set
# CONFIG_AUDIT is not set
CONFIG_HAVE_GENERIC_HARDIRQS=y

#
# IRQ subsystem
#
CONFIG_GENERIC_HARDIRQS=y
CONFIG_HAVE_SPARSE_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_RCU_FAST_NO_HZ is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=21
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
# CONFIG_CGROUP_FREEZER is not set
# CONFIG_CGROUP_DEVICE is not set
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
# CONFIG_CGROUP_MEM_RES_CTLR is not set
# CONFIG_CGROUP_PERF is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_CFS_BANDWIDTH is not set
CONFIG_RT_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
# CONFIG_CHECKPOINT_RESTORE is not set
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
# CONFIG_SCHED_AUTOGROUP is not set
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EXPERT is not set
CONFIG_UID16=y
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
CONFIG_PERF_COUNTERS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
CONFIG_COMPAT_BRK=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_PROFILING is not set
CONFIG_TRACEPOINTS=y
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
CONFIG_KPROBES=y
# CONFIG_JUMP_LABEL is not set
CONFIG_OPTPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_BSGLIB=y
CONFIG_BLK_DEV_INTEGRITY=y
# CONFIG_BLK_DEV_THROTTLING is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
# CONFIG_MINIX_SUBPARTITION is not set
CONFIG_SOLARIS_X86_PARTITION=y
# CONFIG_UNIXWARE_DISKLABEL is not set
CONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_CFQ_GROUP_IOSCHED is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_INLINE_SPIN_TRYLOCK is not set
# CONFIG_INLINE_SPIN_TRYLOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK is not set
# CONFIG_INLINE_SPIN_LOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK_IRQ is not set
# CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
# CONFIG_INLINE_SPIN_UNLOCK is not set
# CONFIG_INLINE_SPIN_UNLOCK_BH is not set
# CONFIG_INLINE_SPIN_UNLOCK_IRQ is not set
# CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_READ_TRYLOCK is not set
# CONFIG_INLINE_READ_LOCK is not set
# CONFIG_INLINE_READ_LOCK_BH is not set
# CONFIG_INLINE_READ_LOCK_IRQ is not set
# CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
# CONFIG_INLINE_READ_UNLOCK is not set
# CONFIG_INLINE_READ_UNLOCK_BH is not set
# CONFIG_INLINE_READ_UNLOCK_IRQ is not set
# CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_WRITE_TRYLOCK is not set
# CONFIG_INLINE_WRITE_LOCK is not set
# CONFIG_INLINE_WRITE_LOCK_BH is not set
# CONFIG_INLINE_WRITE_LOCK_IRQ is not set
# CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
# CONFIG_INLINE_WRITE_UNLOCK is not set
# CONFIG_INLINE_WRITE_UNLOCK_BH is not set
# CONFIG_INLINE_WRITE_UNLOCK_IRQ is not set
# CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
# CONFIG_MUTEX_SPIN_ON_OWNER is not set
CONFIG_FREEZER=y

#
# Processor type and features
#
CONFIG_ZONE_DMA=y
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_SMP=y
CONFIG_X86_X2APIC=y
CONFIG_X86_MPPARSE=y
# CONFIG_X86_EXTENDED_PLATFORM is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
# CONFIG_KVMTOOL_TEST_ENABLE is not set
# CONFIG_PARAVIRT_GUEST is not set
CONFIG_NO_BOOTMEM=y
CONFIG_MEMTEST=y
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=7
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=255
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
# CONFIG_IRQ_TIME_ACCOUNTING is not set
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_MCE_INJECT=y
CONFIG_X86_THERMAL_VECTOR=y
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=8
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=999999
# CONFIG_COMPACTION is not set
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_MEMORY_FAILURE is not set
# CONFIG_TRANSPARENT_HUGEPAGE is not set
# CONFIG_CLEANCACHE is not set
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
CONFIG_EFI=y
# CONFIG_EFI_STUB is not set
CONFIG_SECCOMP=y
CONFIG_CC_STACKPROTECTOR=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
# CONFIG_KEXEC_JUMP is not set
CONFIG_PHYSICAL_START=0x200000
CONFIG_RELOCATABLE=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_COMPAT_VDSO=y
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y

#
# Power management and ACPI options
#
CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_RUNTIME=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
# CONFIG_PM_ADVANCED_DEBUG is not set
# CONFIG_PM_TEST_SUSPEND is not set
CONFIG_CAN_PM_TRACE=y
# CONFIG_PM_TRACE_RTC is not set
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
# CONFIG_ACPI_PROCFS is not set
# CONFIG_ACPI_PROCFS_POWER is not set
# CONFIG_ACPI_EC_DEBUGFS is not set
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_FAN=y
# CONFIG_ACPI_DOCK is not set
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_IPMI=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_NUMA=y
CONFIG_ACPI_CUSTOM_DSDT_FILE=""
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_DEBUG=y
# CONFIG_ACPI_DEBUG_FUNC_TRACE is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_HOTPLUG_MEMORY=y
# CONFIG_ACPI_SBS is not set
CONFIG_ACPI_HED=y
# CONFIG_ACPI_CUSTOM_METHOD is not set
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=y
CONFIG_ACPI_APEI_PCIEAER=y
CONFIG_ACPI_APEI_EINJ=y
CONFIG_ACPI_APEI_ERST_DEBUG=y
# CONFIG_SFI is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

#
# x86 CPU frequency scaling drivers
#
CONFIG_X86_PCC_CPUFREQ=y
CONFIG_X86_ACPI_CPUFREQ=y
# CONFIG_X86_POWERNOW_K8 is not set
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_P4_CLOCKMOD is not set

#
# shared options
#
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_INTEL_IDLE is not set

#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCI_CNB20LE_QUIRK is not set
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_PCIEAER=y
CONFIG_PCIE_ECRC=y
CONFIG_PCIEAER_INJECT=y
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEBUG=y
CONFIG_PCIEASPM_DEFAULT=y
# CONFIG_PCIEASPM_POWERSAVE is not set
# CONFIG_PCIEASPM_PERFORMANCE is not set
CONFIG_PCIE_PME=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_DEBUG=y
CONFIG_PCI_REALLOC_ENABLE_DETECT=y
CONFIG_PCI_STUB=y
CONFIG_HT_IRQ=y
CONFIG_PCI_ATS=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
CONFIG_PCCARD=y
# CONFIG_PCMCIA is not set
CONFIG_CARDBUS=y

#
# PC-card bridges
#
CONFIG_YENTA=y
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_HOTPLUG_PCI=y
# CONFIG_HOTPLUG_PCI_FAKE is not set
# CONFIG_HOTPLUG_PCI_ACPI is not set
CONFIG_HOTPLUG_PCI_CPCI=y
# CONFIG_HOTPLUG_PCI_CPCI_ZT5550 is not set
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=y
CONFIG_HOTPLUG_PCI_SHPC=y
CONFIG_RAPIDIO=y
CONFIG_RAPIDIO_TSI721=y
CONFIG_RAPIDIO_DISC_TIMEOUT=30
CONFIG_RAPIDIO_ENABLE_RX_TX_PORTS=y
CONFIG_RAPIDIO_DEBUG=y
CONFIG_RAPIDIO_TSI57X=y
CONFIG_RAPIDIO_CPS_XX=y
CONFIG_RAPIDIO_TSI568=y
CONFIG_RAPIDIO_CPS_GEN2=y
CONFIG_RAPIDIO_TSI500=y

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
# CONFIG_HAVE_AOUT is not set
# CONFIG_BINFMT_MISC is not set
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_KEYS_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_PNP_RARP=y
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE_DEMUX is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_INET_UDP_DIAG is not set
CONFIG_TCP_CONG_ADVANCED=y
# CONFIG_TCP_CONG_BIC is not set
CONFIG_TCP_CONG_CUBIC=y
# CONFIG_TCP_CONG_WESTWOOD is not set
# CONFIG_TCP_CONG_HTCP is not set
CONFIG_TCP_CONG_HSTCP=y
# CONFIG_TCP_CONG_HYBLA is not set
# CONFIG_TCP_CONG_VEGAS is not set
CONFIG_TCP_CONG_SCALABLE=y
# CONFIG_TCP_CONG_LP is not set
# CONFIG_TCP_CONG_VENO is not set
# CONFIG_TCP_CONG_YEAH is not set
# CONFIG_TCP_CONG_ILLINOIS is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
# CONFIG_NETFILTER is not set
CONFIG_IP_DCCP=y
CONFIG_INET_DCCP_DIAG=y

#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_TFRC_LIB=y

#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
# CONFIG_NET_DCCPPROBE is not set
CONFIG_IP_SCTP=y
# CONFIG_NET_SCTPPROBE is not set
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
CONFIG_RDS=y
# CONFIG_RDS_RDMA is not set
CONFIG_RDS_TCP=y
# CONFIG_RDS_DEBUG is not set
CONFIG_TIPC=y
# CONFIG_TIPC_ADVANCED is not set
# CONFIG_TIPC_DEBUG is not set
# CONFIG_ATM is not set
# CONFIG_L2TP is not set
# CONFIG_BRIDGE is not set
# CONFIG_NET_DSA is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
# CONFIG_NET_SCHED is not set
# CONFIG_DCB is not set
CONFIG_DNS_RESOLVER=y
# CONFIG_BATMAN_ADV is not set
# CONFIG_OPENVSWITCH is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
# CONFIG_NETPRIO_CGROUP is not set
CONFIG_BQL=y
CONFIG_HAVE_BPF_JIT=y
# CONFIG_BPF_JIT is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_NET_TCPPROBE is not set
# CONFIG_NET_DROP_MONITOR is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
CONFIG_WIRELESS=y
# CONFIG_CFG80211 is not set
# CONFIG_LIB80211 is not set

#
# CFG80211 needs to be enabled for MAC80211
#
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
# CONFIG_CEPH_LIB is not set
# CONFIG_NFC is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
# CONFIG_DEVTMPFS is not set
# CONFIG_STANDALONE is not set
# CONFIG_PREVENT_FIRMWARE_BUILD is not set
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE="ql2100_fw.bin ql2300_fw.bin ql2400_fw.bin ql2200_fw.bin ql2322_fw.bin ql2500_fw.bin"
CONFIG_EXTRA_FIRMWARE_DIR="../firmware"
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_GENERIC_CPU_DEVICES is not set
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=y
CONFIG_REGMAP_SPI=y
# CONFIG_DMA_SHARED_BUFFER is not set
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_MTD is not set
# CONFIG_PARPORT is not set
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
CONFIG_BLK_DEV_PCIESSD_MTIP32XX=y
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_DRBD is not set
# CONFIG_BLK_DEV_NBD is not set
CONFIG_BLK_DEV_NVME=y
# CONFIG_BLK_DEV_OSD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=262144
# CONFIG_BLK_DEV_XIP is not set
# CONFIG_CDROM_PKTCDVD is not set
CONFIG_ATA_OVER_ETH=y
CONFIG_VIRTIO_BLK=y
# CONFIG_BLK_DEV_HD is not set
# CONFIG_BLK_DEV_RBD is not set

#
# Misc devices
#
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_AD525X_DPOT is not set
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_INTEL_MID_PTI is not set
# CONFIG_SGI_IOC4 is not set
CONFIG_TIFM_CORE=y
CONFIG_TIFM_7XX1=y
# CONFIG_ICS932S401 is not set
CONFIG_ENCLOSURE_SERVICES=y
# CONFIG_HP_ILO is not set
# CONFIG_APDS9802ALS is not set
# CONFIG_ISL29003 is not set
# CONFIG_ISL29020 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_SENSORS_BH1780 is not set
# CONFIG_SENSORS_BH1770 is not set
# CONFIG_SENSORS_APDS990X is not set
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
# CONFIG_TI_DAC7512 is not set
# CONFIG_VMWARE_BALLOON is not set
# CONFIG_BMP085 is not set
# CONFIG_PCH_PHUB is not set
# CONFIG_USB_SWITCH_FSA9480 is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
CONFIG_EEPROM_AT24=y
CONFIG_EEPROM_AT25=y
# CONFIG_EEPROM_LEGACY is not set
# CONFIG_EEPROM_MAX6875 is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_EEPROM_93XX46 is not set
CONFIG_CB710_CORE=y
# CONFIG_CB710_DEBUG is not set
CONFIG_CB710_DEBUG_ASSUMPTIONS=y
# CONFIG_IWMC3200TOP is not set

#
# Texas Instruments shared transport line discipline
#
# CONFIG_TI_ST is not set
# CONFIG_SENSORS_LIS3_I2C is not set

#
# Altera FPGA firmware download module
#
# CONFIG_ALTERA_STAPL is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=y
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=y
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=y
CONFIG_SCSI_ENCLOSURE=y
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=y
CONFIG_SCSI_FC_TGT_ATTRS=y
CONFIG_SCSI_ISCSI_ATTRS=y
CONFIG_SCSI_SAS_ATTRS=y
CONFIG_SCSI_SAS_LIBSAS=y
CONFIG_SCSI_SAS_ATA=y
CONFIG_SCSI_SAS_HOST_SMP=y
CONFIG_SCSI_SRP_ATTRS=y
CONFIG_SCSI_SRP_TGT_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=y
CONFIG_ISCSI_BOOT_SYSFS=y
CONFIG_SCSI_CXGB3_ISCSI=y
CONFIG_SCSI_CXGB4_ISCSI=y
CONFIG_SCSI_BNX2_ISCSI=y
CONFIG_SCSI_BNX2X_FCOE=y
CONFIG_BE2ISCSI=y
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_HPSA is not set
CONFIG_SCSI_3W_9XXX=y
# CONFIG_SCSI_3W_SAS is not set
# CONFIG_SCSI_ACARD is not set
CONFIG_SCSI_AACRAID=y
CONFIG_SCSI_AIC7XXX=y
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=5000
# CONFIG_AIC7XXX_BUILD_FIRMWARE is not set
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
# CONFIG_SCSI_AIC7XXX_OLD is not set
CONFIG_SCSI_AIC79XX=y
CONFIG_AIC79XX_CMDS_PER_DEVICE=32
CONFIG_AIC79XX_RESET_DELAY_MS=5000
# CONFIG_AIC79XX_BUILD_FIRMWARE is not set
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
CONFIG_AIC79XX_REG_PRETTY_PRINT=y
CONFIG_SCSI_AIC94XX=y
# CONFIG_AIC94XX_DEBUG is not set
CONFIG_SCSI_MVSAS=y
CONFIG_SCSI_MVSAS_DEBUG=y
# CONFIG_SCSI_MVSAS_TASKLET is not set
# CONFIG_SCSI_MVUMI is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
CONFIG_MEGARAID_NEWGEN=y
CONFIG_MEGARAID_MM=y
CONFIG_MEGARAID_MAILBOX=y
CONFIG_MEGARAID_LEGACY=y
CONFIG_MEGARAID_SAS=y
CONFIG_SCSI_MPT2SAS=y
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
CONFIG_SCSI_MPT2SAS_LOGGING=y
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_VMWARE_PVSCSI is not set
CONFIG_LIBFC=y
CONFIG_LIBFCOE=y
CONFIG_FCOE=y
CONFIG_FCOE_FNIC=y
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
CONFIG_SCSI_ISCI=y
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_STEX is not set
CONFIG_SCSI_SYM53C8XX_2=y
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_SYM53C8XX_MMIO=y
# CONFIG_SCSI_IPR is not set
CONFIG_SCSI_QLOGIC_1280=y
CONFIG_SCSI_QLA_FC=y
CONFIG_SCSI_QLA_ISCSI=y
CONFIG_SCSI_LPFC=y
CONFIG_SCSI_LPFC_DEBUG_FS=y
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
CONFIG_SCSI_DEBUG=y
CONFIG_SCSI_PMCRAID=y
# CONFIG_SCSI_PM8001 is not set
CONFIG_SCSI_SRP=y
CONFIG_SCSI_BFA_FC=y
CONFIG_SCSI_DH=y
CONFIG_SCSI_DH_RDAC=y
# CONFIG_SCSI_DH_HP_SW is not set
# CONFIG_SCSI_DH_EMC is not set
# CONFIG_SCSI_DH_ALUA is not set
CONFIG_SCSI_OSD_INITIATOR=y
CONFIG_SCSI_OSD_ULD=y
CONFIG_SCSI_OSD_DPRINT_SENSE=1
CONFIG_SCSI_OSD_DEBUG=y
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y

#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=y
# CONFIG_SATA_AHCI_PLATFORM is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_SATA_ACARD_AHCI is not set
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y

#
# SFF controllers with custom DMA interface
#
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_SX4 is not set
CONFIG_ATA_BMDMA=y

#
# SATA SFF controllers with BMDMA
#
CONFIG_ATA_PIIX=y
CONFIG_SATA_MV=y
CONFIG_SATA_NV=y
# CONFIG_SATA_PROMISE is not set
CONFIG_SATA_SIL=y
# CONFIG_SATA_SIS is not set
CONFIG_SATA_SVW=y
# CONFIG_SATA_ULI is not set
CONFIG_SATA_VIA=y
# CONFIG_SATA_VITESSE is not set

#
# PATA SFF controllers with BMDMA
#
# CONFIG_PATA_ALI is not set
CONFIG_PATA_AMD=y
CONFIG_PATA_ARASAN_CF=y
# CONFIG_PATA_ARTOP is not set
CONFIG_PATA_ATIIXP=y
# CONFIG_PATA_ATP867X is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CS5536 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87415 is not set
CONFIG_PATA_OLDPIIX=y
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RDC is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SCH is not set
CONFIG_PATA_SERVERWORKS=y
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_TOSHIBA is not set
# CONFIG_PATA_TRIFLEX is not set
CONFIG_PATA_VIA=y
# CONFIG_PATA_WINBOND is not set

#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
CONFIG_PATA_MPIIX=y
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_RZ1000 is not set

#
# Generic fallback / legacy drivers
#
# CONFIG_PATA_ACPI is not set
# CONFIG_ATA_GENERIC is not set
CONFIG_PATA_LEGACY=y
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_MD_RAID10=y
CONFIG_MD_RAID456=y
# CONFIG_MULTICORE_RAID456 is not set
CONFIG_MD_MULTIPATH=y
CONFIG_MD_FAULTY=y
CONFIG_BLK_DEV_DM=y
CONFIG_DM_DEBUG=y
CONFIG_DM_CRYPT=y
CONFIG_DM_SNAPSHOT=y
# CONFIG_DM_THIN_PROVISIONING is not set
CONFIG_DM_MIRROR=y
# CONFIG_DM_RAID is not set
# CONFIG_DM_LOG_USERSPACE is not set
CONFIG_DM_ZERO=y
CONFIG_DM_MULTIPATH=y
# CONFIG_DM_MULTIPATH_QL is not set
# CONFIG_DM_MULTIPATH_ST is not set
CONFIG_DM_DELAY=y
CONFIG_DM_UEVENT=y
# CONFIG_DM_FLAKEY is not set
# CONFIG_TARGET_CORE is not set
CONFIG_FUSION=y
CONFIG_FUSION_SPI=y
CONFIG_FUSION_FC=y
CONFIG_FUSION_SAS=y
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=y
CONFIG_FUSION_LAN=y
# CONFIG_FUSION_LOGGING is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
# CONFIG_FIREWIRE_NOSY is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
# CONFIG_BONDING is not set
# CONFIG_DUMMY is not set
# CONFIG_EQUALIZER is not set
CONFIG_NET_FC=y
CONFIG_MII=y
CONFIG_NET_TEAM=y
CONFIG_NET_TEAM_MODE_ROUNDROBIN=y
CONFIG_NET_TEAM_MODE_ACTIVEBACKUP=y
CONFIG_MACVLAN=y
CONFIG_MACVTAP=y
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
CONFIG_RIONET=y
CONFIG_RIONET_TX_SIZE=128
CONFIG_RIONET_RX_SIZE=128
CONFIG_TUN=y
CONFIG_VETH=y
CONFIG_VIRTIO_NET=y
# CONFIG_ARCNET is not set

#
# CAIF transport drivers
#
CONFIG_ETHERNET=y
CONFIG_MDIO=y
CONFIG_NET_VENDOR_3COM=y
# CONFIG_VORTEX is not set
# CONFIG_TYPHOON is not set
CONFIG_NET_VENDOR_ADAPTEC=y
# CONFIG_ADAPTEC_STARFIRE is not set
CONFIG_NET_VENDOR_ALTEON=y
# CONFIG_ACENIC is not set
CONFIG_NET_VENDOR_AMD=y
# CONFIG_AMD8111_ETH is not set
# CONFIG_PCNET32 is not set
CONFIG_NET_VENDOR_ATHEROS=y
# CONFIG_ATL2 is not set
# CONFIG_ATL1 is not set
# CONFIG_ATL1E is not set
# CONFIG_ATL1C is not set
CONFIG_NET_VENDOR_BROADCOM=y
# CONFIG_B44 is not set
CONFIG_BNX2=y
CONFIG_CNIC=y
CONFIG_TIGON3=y
CONFIG_BNX2X=y
CONFIG_NET_VENDOR_BROCADE=y
CONFIG_BNA=y
CONFIG_NET_CALXEDA_XGMAC=y
CONFIG_NET_VENDOR_CHELSIO=y
CONFIG_CHELSIO_T1=y
CONFIG_CHELSIO_T1_1G=y
CONFIG_CHELSIO_T3=y
CONFIG_CHELSIO_T4=y
CONFIG_CHELSIO_T4VF=y
CONFIG_NET_VENDOR_CISCO=y
CONFIG_ENIC=y
# CONFIG_DNET is not set
CONFIG_NET_VENDOR_DEC=y
# CONFIG_NET_TULIP is not set
CONFIG_NET_VENDOR_DLINK=y
# CONFIG_DL2K is not set
# CONFIG_SUNDANCE is not set
CONFIG_NET_VENDOR_EMULEX=y
CONFIG_BE2NET=y
CONFIG_NET_VENDOR_EXAR=y
# CONFIG_S2IO is not set
# CONFIG_VXGE is not set
CONFIG_NET_VENDOR_HP=y
# CONFIG_HP100 is not set
CONFIG_NET_VENDOR_INTEL=y
# CONFIG_E100 is not set
CONFIG_E1000=y
CONFIG_E1000E=y
CONFIG_IGB=y
CONFIG_IGB_DCA=y
CONFIG_IGBVF=y
CONFIG_IXGB=y
CONFIG_IXGBE=y
CONFIG_IXGBE_DCA=y
CONFIG_IXGBEVF=y
CONFIG_NET_VENDOR_I825XX=y
# CONFIG_ZNET is not set
# CONFIG_IP1000 is not set
# CONFIG_JME is not set
CONFIG_NET_VENDOR_MARVELL=y
CONFIG_SKGE=y
# CONFIG_SKGE_DEBUG is not set
# CONFIG_SKGE_GENESIS is not set
CONFIG_SKY2=y
# CONFIG_SKY2_DEBUG is not set
CONFIG_NET_VENDOR_MELLANOX=y
CONFIG_MLX4_EN=y
CONFIG_MLX4_CORE=y
CONFIG_MLX4_DEBUG=y
CONFIG_NET_VENDOR_MICREL=y
# CONFIG_KS8842 is not set
# CONFIG_KS8851 is not set
# CONFIG_KS8851_MLL is not set
# CONFIG_KSZ884X_PCI is not set
CONFIG_NET_VENDOR_MICROCHIP=y
# CONFIG_ENC28J60 is not set
CONFIG_NET_VENDOR_MYRI=y
CONFIG_MYRI10GE=y
CONFIG_MYRI10GE_DCA=y
# CONFIG_FEALNX is not set
CONFIG_NET_VENDOR_NATSEMI=y
# CONFIG_NATSEMI is not set
# CONFIG_NS83820 is not set
CONFIG_NET_VENDOR_8390=y
# CONFIG_NE2K_PCI is not set
CONFIG_NET_VENDOR_NVIDIA=y
CONFIG_FORCEDETH=y
CONFIG_NET_VENDOR_OKI=y
# CONFIG_PCH_GBE is not set
# CONFIG_ETHOC is not set
# CONFIG_NET_PACKET_ENGINE is not set
CONFIG_NET_VENDOR_QLOGIC=y
CONFIG_QLA3XXX=y
CONFIG_QLCNIC=y
CONFIG_QLGE=y
# CONFIG_NETXEN_NIC is not set
CONFIG_NET_VENDOR_REALTEK=y
CONFIG_8139CP=y
CONFIG_8139TOO=y
# CONFIG_8139TOO_PIO is not set
CONFIG_8139TOO_TUNE_TWISTER=y
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
CONFIG_R8169=y
# CONFIG_NET_VENDOR_RDC is not set
CONFIG_NET_VENDOR_SEEQ=y
# CONFIG_SEEQ8005 is not set
CONFIG_NET_VENDOR_SILAN=y
# CONFIG_SC92031 is not set
CONFIG_NET_VENDOR_SIS=y
# CONFIG_SIS900 is not set
# CONFIG_SIS190 is not set
CONFIG_SFC=y
CONFIG_SFC_MCDI_MON=y
CONFIG_NET_VENDOR_SMSC=y
# CONFIG_EPIC100 is not set
# CONFIG_SMSC9420 is not set
CONFIG_NET_VENDOR_STMICRO=y
# CONFIG_STMMAC_ETH is not set
CONFIG_NET_VENDOR_SUN=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
CONFIG_NIU=y
CONFIG_SXGE=y
CONFIG_SXGEVF=y
CONFIG_NET_VENDOR_TEHUTI=y
# CONFIG_TEHUTI is not set
CONFIG_NET_VENDOR_TI=y
# CONFIG_TLAN is not set
CONFIG_NET_VENDOR_VIA=y
# CONFIG_VIA_RHINE is not set
# CONFIG_VIA_VELOCITY is not set
CONFIG_FDDI=y
# CONFIG_DEFXX is not set
CONFIG_SKFP=y
# CONFIG_HIPPI is not set
# CONFIG_NET_SB1000 is not set
CONFIG_PHYLIB=y

#
# MII PHY device drivers
#
CONFIG_MARVELL_PHY=y
CONFIG_DAVICOM_PHY=y
CONFIG_QSEMI_PHY=y
CONFIG_LXT_PHY=y
CONFIG_CICADA_PHY=y
CONFIG_VITESSE_PHY=y
CONFIG_SMSC_PHY=y
CONFIG_BROADCOM_PHY=y
CONFIG_ICPLUS_PHY=y
CONFIG_REALTEK_PHY=y
CONFIG_NATIONAL_PHY=y
CONFIG_STE10XP=y
CONFIG_LSI_ET1011C_PHY=y
# CONFIG_MICREL_PHY is not set
CONFIG_FIXED_PHY=y
CONFIG_MDIO_BITBANG=y
# CONFIG_MDIO_GPIO is not set
# CONFIG_MICREL_KS8995MA is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_TR is not set

#
# USB Network Adapters
#
CONFIG_USB_CATC=y
CONFIG_USB_KAWETH=y
CONFIG_USB_PEGASUS=y
CONFIG_USB_RTL8150=y
CONFIG_USB_USBNET=y
CONFIG_USB_NET_AX8817X=y
CONFIG_USB_NET_CDCETHER=y
# CONFIG_USB_NET_CDC_EEM is not set
CONFIG_USB_NET_CDC_NCM=y
CONFIG_USB_NET_DM9601=y
# CONFIG_USB_NET_SMSC75XX is not set
CONFIG_USB_NET_SMSC95XX=y
CONFIG_USB_NET_GL620A=y
CONFIG_USB_NET_NET1080=y
CONFIG_USB_NET_PLUSB=y
CONFIG_USB_NET_MCS7830=y
CONFIG_USB_NET_RNDIS_HOST=y
CONFIG_USB_NET_CDC_SUBSET=y
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_KC2190=y
CONFIG_USB_NET_ZAURUS=y
# CONFIG_USB_NET_CX82310_ETH is not set
# CONFIG_USB_NET_KALMIA is not set
# CONFIG_USB_NET_QMI_WWAN is not set
# CONFIG_USB_NET_INT51X1 is not set
CONFIG_USB_IPHETH=y
CONFIG_USB_SIERRA_NET=y
# CONFIG_USB_VL600 is not set
CONFIG_WLAN=y
# CONFIG_AIRO is not set
# CONFIG_ATMEL is not set
# CONFIG_PRISM54 is not set
# CONFIG_USB_ZD1201 is not set
# CONFIG_HOSTAP is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
# CONFIG_WAN is not set
# CONFIG_VMXNET3 is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=y
# CONFIG_INPUT_POLLDEV is not set
# CONFIG_INPUT_SPARSEKMAP is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
# CONFIG_KEYBOARD_ADP5589 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1070 is not set
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_GPIO is not set
# CONFIG_KEYBOARD_GPIO_POLLED is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_TCA8418 is not set
# CONFIG_KEYBOARD_MATRIX is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_SENTELIC is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_SERIAL=y
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_MOUSE_GPIO is not set
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
# CONFIG_INPUT_BMA150 is not set
CONFIG_INPUT_PCSPKR=y
# CONFIG_INPUT_MMA8450 is not set
# CONFIG_INPUT_MPU3050 is not set
# CONFIG_INPUT_APANEL is not set
# CONFIG_INPUT_GP2A is not set
# CONFIG_INPUT_GPIO_TILT_POLLED is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_KXTJ9 is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_UINPUT is not set
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_GPIO_ROTARY_ENCODER is not set
# CONFIG_INPUT_ADXL34X is not set
# CONFIG_INPUT_CMA3000 is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
CONFIG_SERIO_PCIPS2=y
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=y
# CONFIG_SERIO_ALTERA_PS2 is not set
# CONFIG_SERIO_PS2MULT is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set
# CONFIG_N_GSM is not set
# CONFIG_TRACE_SINK is not set
CONFIG_DEVKMEM=y

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
# CONFIG_SERIAL_8250_MANY_PORTS is not set
# CONFIG_SERIAL_8250_SHARE_IRQ is not set
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
# CONFIG_SERIAL_8250_RSA is not set

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MAX3100 is not set
# CONFIG_SERIAL_MAX3107 is not set
# CONFIG_SERIAL_MFD_HSU is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=y
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_IFX6X60 is not set
# CONFIG_SERIAL_PCH_UART is not set
# CONFIG_SERIAL_XILINX_PS_UART is not set
# CONFIG_VIRTIO_CONSOLE is not set
CONFIG_IPMI_HANDLER=y
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=y
CONFIG_IPMI_SI=y
# CONFIG_IPMI_WATCHDOG is not set
# CONFIG_IPMI_POWEROFF is not set
CONFIG_HW_RANDOM=y
# CONFIG_HW_RANDOM_TIMERIOMEM is not set
CONFIG_HW_RANDOM_INTEL=y
CONFIG_HW_RANDOM_AMD=y
CONFIG_HW_RANDOM_VIA=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_NVRAM=y
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
CONFIG_RAW_DRIVER=y
CONFIG_MAX_RAW_DEVS=256
# CONFIG_HPET is not set
# CONFIG_HANGCHECK_TIMER is not set
CONFIG_TCG_TPM=y
CONFIG_TCG_TIS=y
CONFIG_TCG_NSC=y
CONFIG_TCG_ATMEL=y
# CONFIG_TCG_INFINEON is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
# CONFIG_RAMOOPS is not set
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=y
# CONFIG_I2C_MUX is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=y

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=y
# CONFIG_I2C_AMD756_S4882 is not set
CONFIG_I2C_AMD8111=y
CONFIG_I2C_I801=y
CONFIG_I2C_ISCH=y
CONFIG_I2C_PIIX4=y
CONFIG_I2C_NFORCE2=y
# CONFIG_I2C_NFORCE2_S4985 is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set

#
# ACPI drivers
#
# CONFIG_I2C_SCMI is not set

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_DESIGNWARE_PCI is not set
# CONFIG_I2C_GPIO is not set
# CONFIG_I2C_INTEL_MID is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_PXA_PCI is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_XILINX is not set
# CONFIG_I2C_EG20T is not set

#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_DIOLAN_U2C is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
CONFIG_SPI=y
# CONFIG_SPI_DEBUG is not set
CONFIG_SPI_MASTER=y

#
# SPI Master Controller Drivers
#
# CONFIG_SPI_ALTERA is not set
CONFIG_SPI_BITBANG=y
# CONFIG_SPI_GPIO is not set
# CONFIG_SPI_OC_TINY is not set
# CONFIG_SPI_PXA2XX_PCI is not set
# CONFIG_SPI_TOPCLIFF_PCH is not set
# CONFIG_SPI_XILINX is not set
# CONFIG_SPI_DESIGNWARE is not set

#
# SPI Protocol Masters
#
# CONFIG_SPI_SPIDEV is not set
# CONFIG_SPI_TLE62X0 is not set

#
# PPS support
#
# CONFIG_PPS is not set

#
# PPS generators support
#

#
# PTP clock support
#

#
# Enable Device Drivers -> PPS to see the PTP clock options.
#
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_GPIOLIB=y
# CONFIG_DEBUG_GPIO is not set
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_MAX730X=y

#
# Memory mapped GPIO drivers:
#
# CONFIG_GPIO_GENERIC_PLATFORM is not set
# CONFIG_GPIO_IT8761E is not set
CONFIG_GPIO_SCH=y
# CONFIG_GPIO_VX855 is not set

#
# I2C GPIO expanders:
#
# CONFIG_GPIO_MAX7300 is not set
CONFIG_GPIO_MAX732X=y
# CONFIG_GPIO_MAX732X_IRQ is not set
CONFIG_GPIO_PCA953X=y
# CONFIG_GPIO_PCA953X_IRQ is not set
CONFIG_GPIO_PCF857X=y
# CONFIG_GPIO_SX150X is not set
# CONFIG_GPIO_ADP5588 is not set

#
# PCI GPIO expanders:
#
CONFIG_GPIO_BT8XX=y
# CONFIG_GPIO_LANGWELL is not set
# CONFIG_GPIO_PCH is not set
# CONFIG_GPIO_ML_IOH is not set
# CONFIG_GPIO_RDC321X is not set

#
# SPI GPIO expanders:
#
CONFIG_GPIO_MAX7301=y
CONFIG_GPIO_MCP23S08=y
# CONFIG_GPIO_MC33880 is not set
# CONFIG_GPIO_74X164 is not set

#
# AC97 GPIO expanders:
#

#
# MODULbus GPIO expanders:
#
CONFIG_W1=y
CONFIG_W1_CON=y

#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=y
CONFIG_W1_MASTER_DS2490=y
CONFIG_W1_MASTER_DS2482=y
# CONFIG_W1_MASTER_DS1WM is not set
CONFIG_W1_MASTER_GPIO=y

#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=y
CONFIG_W1_SLAVE_SMEM=y
# CONFIG_W1_SLAVE_DS2408 is not set
# CONFIG_W1_SLAVE_DS2423 is not set
# CONFIG_W1_SLAVE_DS2431 is not set
CONFIG_W1_SLAVE_DS2433=y
# CONFIG_W1_SLAVE_DS2433_CRC is not set
CONFIG_W1_SLAVE_DS2760=y
# CONFIG_W1_SLAVE_DS2780 is not set
# CONFIG_W1_SLAVE_BQ27000 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_BATTERY_DS2780 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_SBS is not set
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_BATTERY_MAX17042 is not set
# CONFIG_CHARGER_MAX8903 is not set
# CONFIG_CHARGER_LP8727 is not set
# CONFIG_CHARGER_GPIO is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=y
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7314 is not set
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
# CONFIG_SENSORS_ADCXX is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
CONFIG_SENSORS_ADM1026=y
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7411 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7475 is not set
# CONFIG_SENSORS_ASC7621 is not set
CONFIG_SENSORS_K8TEMP=y
CONFIG_SENSORS_K10TEMP=y
CONFIG_SENSORS_FAM15H_POWER=y
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS620 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_I5K_AMB is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_G760A is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_GPIO_FAN is not set
CONFIG_SENSORS_CORETEMP=y
# CONFIG_SENSORS_IBMAEM is not set
# CONFIG_SENSORS_IBMPEX is not set
CONFIG_SENSORS_IT87=y
# CONFIG_SENSORS_JC42 is not set
# CONFIG_SENSORS_LINEAGE is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM70 is not set
# CONFIG_SENSORS_LM73 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
CONFIG_SENSORS_LM83=y
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LTC4151 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4245 is not set
# CONFIG_SENSORS_LTC4261 is not set
# CONFIG_SENSORS_LM95241 is not set
# CONFIG_SENSORS_LM95245 is not set
# CONFIG_SENSORS_MAX1111 is not set
# CONFIG_SENSORS_MAX16065 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX1668 is not set
# CONFIG_SENSORS_MAX6639 is not set
# CONFIG_SENSORS_MAX6642 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_NTC_THERMISTOR is not set
# CONFIG_SENSORS_PC87360 is not set
CONFIG_SENSORS_PC87427=y
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_PMBUS is not set
# CONFIG_SENSORS_SHT15 is not set
# CONFIG_SENSORS_SHT21 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_SMM665 is not set
CONFIG_SENSORS_DME1737=y
# CONFIG_SENSORS_EMC1403 is not set
# CONFIG_SENSORS_EMC2103 is not set
# CONFIG_SENSORS_EMC6W201 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
CONFIG_SENSORS_SMSC47B397=y
# CONFIG_SENSORS_SCH56XX_COMMON is not set
# CONFIG_SENSORS_SCH5627 is not set
# CONFIG_SENSORS_SCH5636 is not set
# CONFIG_SENSORS_ADS1015 is not set
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_ADS7871 is not set
# CONFIG_SENSORS_AMC6821 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_TMP102 is not set
# CONFIG_SENSORS_TMP401 is not set
# CONFIG_SENSORS_TMP421 is not set
# CONFIG_SENSORS_VIA_CPUTEMP is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83795 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83L786NG is not set
CONFIG_SENSORS_W83627HF=y
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_APPLESMC is not set

#
# ACPI drivers
#
# CONFIG_SENSORS_ACPI_POWER is not set
# CONFIG_SENSORS_ATK0110 is not set
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_CORE is not set
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
# CONFIG_SOFT_WATCHDOG is not set
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
# CONFIG_F71808E_WDT is not set
CONFIG_SP5100_TCO=y
# CONFIG_SC520_WDT is not set
# CONFIG_SBC_FITPC2_WATCHDOG is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
# CONFIG_IBMASR is not set
# CONFIG_WAFER_WDT is not set
# CONFIG_I6300ESB_WDT is not set
# CONFIG_ITCO_WDT is not set
# CONFIG_IT8712F_WDT is not set
# CONFIG_IT87_WDT is not set
# CONFIG_HP_WATCHDOG is not set
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
# CONFIG_NV_TCO is not set
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_SMSC_SCH311X_WDT is not set
# CONFIG_SMSC37B787_WDT is not set
# CONFIG_VIA_WDT is not set
# CONFIG_W83627HF_WDT is not set
# CONFIG_W83697HF_WDT is not set
# CONFIG_W83697UG_WDT is not set
# CONFIG_W83877F_WDT is not set
# CONFIG_W83977F_WDT is not set
# CONFIG_MACHZ_WDT is not set
# CONFIG_SBC_EPX_C3_WATCHDOG is not set

#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
# CONFIG_WDTPCI is not set

#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y

#
# Broadcom specific AMBA
#
# CONFIG_BCMA is not set

#
# Multifunction device drivers
#
CONFIG_MFD_CORE=y
# CONFIG_MFD_88PM860X is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_HTC_I2CPLD is not set
# CONFIG_UCB1400_CORE is not set
# CONFIG_TPS6105X is not set
# CONFIG_TPS65010 is not set
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TPS6586X is not set
# CONFIG_MFD_TPS65910 is not set
# CONFIG_MFD_TPS65912_I2C is not set
# CONFIG_MFD_TPS65912_SPI is not set
# CONFIG_MFD_STMPE is not set
# CONFIG_MFD_TC3589X is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_DA9052_SPI is not set
# CONFIG_MFD_DA9052_I2C is not set
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8997 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_S5M_CORE is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X_I2C is not set
# CONFIG_MFD_WM831X_SPI is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_WM8994 is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_MFD_MC13XXX is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_EZX_PCAP is not set
# CONFIG_MFD_CS5535 is not set
# CONFIG_MFD_TIMBERDALE is not set
CONFIG_LPC_SCH=y
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_MFD_VX855 is not set
# CONFIG_MFD_WL1273_CORE is not set
# CONFIG_MFD_AAT2870_CORE is not set
# CONFIG_REGULATOR is not set
CONFIG_MEDIA_SUPPORT=y

#
# Multimedia core support
#
# CONFIG_MEDIA_CONTROLLER is not set
CONFIG_VIDEO_DEV=y
CONFIG_VIDEO_V4L2_COMMON=y
CONFIG_DVB_CORE=y
CONFIG_DVB_NET=y
CONFIG_VIDEO_MEDIA=y

#
# Multimedia drivers
#
CONFIG_RC_CORE=y
CONFIG_LIRC=y
CONFIG_RC_MAP=y
CONFIG_IR_NEC_DECODER=y
CONFIG_IR_RC5_DECODER=y
CONFIG_IR_RC6_DECODER=y
CONFIG_IR_JVC_DECODER=y
CONFIG_IR_SONY_DECODER=y
CONFIG_IR_RC5_SZ_DECODER=y
CONFIG_IR_SANYO_DECODER=y
CONFIG_IR_MCE_KBD_DECODER=y
CONFIG_IR_LIRC_CODEC=y
# CONFIG_RC_ATI_REMOTE is not set
# CONFIG_IR_ENE is not set
# CONFIG_IR_IMON is not set
# CONFIG_IR_MCEUSB is not set
# CONFIG_IR_ITE_CIR is not set
# CONFIG_IR_FINTEK is not set
# CONFIG_IR_NUVOTON is not set
# CONFIG_IR_REDRAT3 is not set
# CONFIG_IR_STREAMZAP is not set
# CONFIG_IR_WINBOND_CIR is not set
# CONFIG_RC_LOOPBACK is not set
# CONFIG_MEDIA_ATTACH is not set
CONFIG_MEDIA_TUNER=y
# CONFIG_MEDIA_TUNER_CUSTOMISE is not set
CONFIG_MEDIA_TUNER_SIMPLE=y
CONFIG_MEDIA_TUNER_TDA8290=y
CONFIG_MEDIA_TUNER_TDA827X=y
CONFIG_MEDIA_TUNER_TDA18271=y
CONFIG_MEDIA_TUNER_TDA9887=y
CONFIG_MEDIA_TUNER_TEA5761=y
CONFIG_MEDIA_TUNER_TEA5767=y
CONFIG_MEDIA_TUNER_MT20XX=y
CONFIG_MEDIA_TUNER_XC2028=y
CONFIG_MEDIA_TUNER_XC5000=y
CONFIG_MEDIA_TUNER_XC4000=y
CONFIG_MEDIA_TUNER_MC44S803=y
CONFIG_VIDEO_V4L2=y
CONFIG_VIDEO_CAPTURE_DRIVERS=y
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_IR_I2C=y

#
# Audio decoders, processors and mixers
#

#
# RDS decoders
#

#
# Video decoders
#

#
# Video and audio decoders
#

#
# MPEG video encoders
#

#
# Video encoders
#

#
# Camera sensor devices
#

#
# Flash devices
#

#
# Video improvement chips
#

#
# Miscelaneous helper chips
#
# CONFIG_VIDEO_VIVI is not set
CONFIG_V4L_USB_DRIVERS=y
# CONFIG_USB_VIDEO_CLASS is not set
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_USB_GSPCA=y
# CONFIG_USB_M5602 is not set
# CONFIG_USB_STV06XX is not set
# CONFIG_USB_GL860 is not set
# CONFIG_USB_GSPCA_BENQ is not set
# CONFIG_USB_GSPCA_CONEX is not set
# CONFIG_USB_GSPCA_CPIA1 is not set
# CONFIG_USB_GSPCA_ETOMS is not set
# CONFIG_USB_GSPCA_FINEPIX is not set
# CONFIG_USB_GSPCA_JEILINJ is not set
# CONFIG_USB_GSPCA_JL2005BCD is not set
# CONFIG_USB_GSPCA_KINECT is not set
# CONFIG_USB_GSPCA_KONICA is not set
# CONFIG_USB_GSPCA_MARS is not set
# CONFIG_USB_GSPCA_MR97310A is not set
# CONFIG_USB_GSPCA_NW80X is not set
# CONFIG_USB_GSPCA_OV519 is not set
# CONFIG_USB_GSPCA_OV534 is not set
# CONFIG_USB_GSPCA_OV534_9 is not set
# CONFIG_USB_GSPCA_PAC207 is not set
# CONFIG_USB_GSPCA_PAC7302 is not set
# CONFIG_USB_GSPCA_PAC7311 is not set
# CONFIG_USB_GSPCA_SE401 is not set
# CONFIG_USB_GSPCA_SN9C2028 is not set
# CONFIG_USB_GSPCA_SN9C20X is not set
# CONFIG_USB_GSPCA_SONIXB is not set
# CONFIG_USB_GSPCA_SONIXJ is not set
# CONFIG_USB_GSPCA_SPCA500 is not set
# CONFIG_USB_GSPCA_SPCA501 is not set
CONFIG_USB_GSPCA_SPCA505=y
# CONFIG_USB_GSPCA_SPCA506 is not set
# CONFIG_USB_GSPCA_SPCA508 is not set
# CONFIG_USB_GSPCA_SPCA561 is not set
# CONFIG_USB_GSPCA_SPCA1528 is not set
# CONFIG_USB_GSPCA_SQ905 is not set
# CONFIG_USB_GSPCA_SQ905C is not set
# CONFIG_USB_GSPCA_SQ930X is not set
# CONFIG_USB_GSPCA_STK014 is not set
# CONFIG_USB_GSPCA_STV0680 is not set
# CONFIG_USB_GSPCA_SUNPLUS is not set
# CONFIG_USB_GSPCA_T613 is not set
# CONFIG_USB_GSPCA_TOPRO is not set
# CONFIG_USB_GSPCA_TV8532 is not set
# CONFIG_USB_GSPCA_VC032X is not set
# CONFIG_USB_GSPCA_VICAM is not set
# CONFIG_USB_GSPCA_XIRLINK_CIT is not set
# CONFIG_USB_GSPCA_ZC3XX is not set
# CONFIG_VIDEO_PVRUSB2 is not set
# CONFIG_VIDEO_HDPVR is not set
# CONFIG_VIDEO_EM28XX is not set
# CONFIG_VIDEO_TLG2300 is not set
# CONFIG_VIDEO_CX231XX is not set
# CONFIG_VIDEO_TM6000 is not set
# CONFIG_VIDEO_USBVISION is not set
# CONFIG_USB_ET61X251 is not set
# CONFIG_USB_SN9C102 is not set
# CONFIG_USB_PWC is not set
# CONFIG_VIDEO_CPIA2 is not set
# CONFIG_USB_ZR364XX is not set
# CONFIG_USB_STKWEBCAM is not set
# CONFIG_USB_S2255 is not set
CONFIG_V4L_PCI_DRIVERS=y
# CONFIG_VIDEO_AU0828 is not set
# CONFIG_VIDEO_BT848 is not set
# CONFIG_VIDEO_CX18 is not set
# CONFIG_VIDEO_CX23885 is not set
# CONFIG_VIDEO_CX25821 is not set
# CONFIG_VIDEO_CX88 is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_IVTV is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_SAA7134 is not set
# CONFIG_VIDEO_SAA7164 is not set
# CONFIG_VIDEO_ZORAN is not set
# CONFIG_V4L_PLATFORM_DRIVERS is not set
# CONFIG_V4L_MEM2MEM_DRIVERS is not set
CONFIG_RADIO_ADAPTERS=y
# CONFIG_RADIO_SI470X is not set
# CONFIG_USB_MR800 is not set
# CONFIG_USB_DSBR is not set
# CONFIG_RADIO_MAXIRADIO is not set
# CONFIG_I2C_SI4713 is not set
# CONFIG_RADIO_SI4713 is not set
# CONFIG_RADIO_TEA5764 is not set
# CONFIG_RADIO_SAA7706H is not set
# CONFIG_RADIO_TEF6862 is not set
# CONFIG_RADIO_WL1273 is not set

#
# Texas Instruments WL128x FM driver (ST based)
#
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
CONFIG_DVB_CAPTURE_DRIVERS=y

#
# Supported SAA7146 based PCI Adapters
#
# CONFIG_TTPCI_EEPROM is not set
# CONFIG_DVB_AV7110 is not set
# CONFIG_DVB_BUDGET_CORE is not set

#
# Supported USB Adapters
#
# CONFIG_DVB_USB is not set
# CONFIG_DVB_TTUSB_BUDGET is not set
# CONFIG_DVB_TTUSB_DEC is not set
# CONFIG_SMS_SIANO_MDTV is not set

#
# Supported FlexCopII (B2C2) Adapters
#
# CONFIG_DVB_B2C2_FLEXCOP is not set

#
# Supported BT878 Adapters
#

#
# Supported Pluto2 Adapters
#
# CONFIG_DVB_PLUTO2 is not set

#
# Supported SDMC DM1105 Adapters
#
# CONFIG_DVB_DM1105 is not set

#
# Supported Earthsoft PT1 Adapters
#
# CONFIG_DVB_PT1 is not set

#
# Supported Mantis Adapters
#
CONFIG_MANTIS_CORE=y
CONFIG_DVB_MANTIS=y
CONFIG_DVB_HOPPER=y

#
# Supported nGene Adapters
#
# CONFIG_DVB_NGENE is not set

#
# Supported ddbridge ('Octopus') Adapters
#
# CONFIG_DVB_DDBRIDGE is not set

#
# Supported DVB Frontends
#
# CONFIG_DVB_FE_CUSTOMISE is not set

#
# Multistandard (satellite) frontends
#
CONFIG_DVB_STB0899=y
CONFIG_DVB_STB6100=y

#
# Multistandard (cable + terrestrial) frontends
#

#
# DVB-S (satellite) frontends
#
CONFIG_DVB_STV0299=y
CONFIG_DVB_MB86A16=y

#
# DVB-T (terrestrial) frontends
#
CONFIG_DVB_ZL10353=y

#
# DVB-C (cable) frontends
#
CONFIG_DVB_TDA10021=y
CONFIG_DVB_TDA10023=y

#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#

#
# ISDB-T (terrestrial) frontends
#

#
# Digital terrestrial only tuners/PLL
#
CONFIG_DVB_PLL=y

#
# SEC control devices for DVB-S
#
CONFIG_DVB_LNBP21=y
CONFIG_DVB_TDA665x=y

#
# Tools to develop new frontends
#
# CONFIG_DVB_DUMMY_FE is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_VIA is not set
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
# CONFIG_VGA_SWITCHEROO is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set
# CONFIG_DRM_VMWGFX is not set
# CONFIG_DRM_GMA500 is not set
# CONFIG_STUB_POULSBO is not set
# CONFIG_VGASTATE is not set
# CONFIG_VIDEO_OUTPUT_CONTROL is not set
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
# CONFIG_FB_DDC is not set
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
# CONFIG_FB_WMT_GE_ROPS is not set
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
# CONFIG_FB_TILEBLITTING is not set

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
# CONFIG_FB_EFI is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_TMIO is not set
# CONFIG_FB_SMSCUFX is not set
# CONFIG_FB_UDL is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=y
# CONFIG_LCD_L4F00242T03 is not set
# CONFIG_LCD_LMS283GF05 is not set
# CONFIG_LCD_LTV350QV is not set
# CONFIG_LCD_TDO24M is not set
# CONFIG_LCD_VGG2432A4 is not set
# CONFIG_LCD_PLATFORM is not set
# CONFIG_LCD_S6E63M0 is not set
# CONFIG_LCD_LD9040 is not set
# CONFIG_LCD_AMS369FG06 is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=y
# CONFIG_BACKLIGHT_PROGEAR is not set
# CONFIG_BACKLIGHT_APPLE is not set
# CONFIG_BACKLIGHT_SAHARA is not set
# CONFIG_BACKLIGHT_ADP8860 is not set
# CONFIG_BACKLIGHT_ADP8870 is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
CONFIG_FONTS=y
# CONFIG_FONT_8x8 is not set
CONFIG_FONT_8x16=y
# CONFIG_FONT_6x11 is not set
# CONFIG_FONT_7x14 is not set
# CONFIG_FONT_PEARL_8x8 is not set
# CONFIG_FONT_ACORN_8x8 is not set
# CONFIG_FONT_MINI_4x6 is not set
# CONFIG_FONT_SUN8x16 is not set
# CONFIG_FONT_SUN12x22 is not set
# CONFIG_FONT_10x18 is not set
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_MONO=y
CONFIG_LOGO_LINUX_VGA16=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=y
# CONFIG_SOUND_OSS_CORE is not set
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_HWDEP=y
CONFIG_SND_RAWMIDI=y
CONFIG_SND_JACK=y
# CONFIG_SND_SEQUENCER is not set
# CONFIG_SND_MIXER_OSS is not set
# CONFIG_SND_PCM_OSS is not set
# CONFIG_SND_HRTIMER is not set
CONFIG_SND_DYNAMIC_MINORS=y
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_KCTL_JACK=y
CONFIG_SND_DMA_SGBUF=y
# CONFIG_SND_RAWMIDI_SEQ is not set
# CONFIG_SND_OPL3_LIB_SEQ is not set
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
# CONFIG_SND_EMU10K1_SEQ is not set
CONFIG_SND_MPU401_UART=y
CONFIG_SND_AC97_CODEC=y
CONFIG_SND_DRIVERS=y
# CONFIG_SND_PCSP is not set
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_ALOOP is not set
# CONFIG_SND_MTPAV is not set
CONFIG_SND_SERIAL_U16550=y
CONFIG_SND_MPU401=y
# CONFIG_SND_AC97_POWER_SAVE is not set
CONFIG_SND_PCI=y
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ASIHPI is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_CTXFI is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_INDIGOIOX is not set
# CONFIG_SND_INDIGODJX is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
CONFIG_SND_HDA_INTEL=y
CONFIG_SND_HDA_PREALLOC_SIZE=64
# CONFIG_SND_HDA_HWDEP is not set
# CONFIG_SND_HDA_INPUT_BEEP is not set
# CONFIG_SND_HDA_INPUT_JACK is not set
# CONFIG_SND_HDA_PATCH_LOADER is not set
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_ENABLE_REALTEK_QUIRKS=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CA0110=y
CONFIG_SND_HDA_CODEC_CA0132=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
# CONFIG_SND_HDA_POWER_SAVE is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
CONFIG_SND_INTEL8X0=y
CONFIG_SND_INTEL8X0M=y
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_LOLA is not set
# CONFIG_SND_LX6464ES is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
CONFIG_SND_SPI=y
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=y
# CONFIG_SND_USB_UA101 is not set
# CONFIG_SND_USB_USX2Y is not set
CONFIG_SND_USB_CAIAQ=y
CONFIG_SND_USB_CAIAQ_INPUT=y
# CONFIG_SND_USB_US122L is not set
# CONFIG_SND_USB_6FIRE is not set
CONFIG_SND_SOC=y
CONFIG_SND_SOC_I2C_AND_SPI=y
# CONFIG_SND_SOC_ALL_CODECS is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
CONFIG_HID_BATTERY_STRENGTH=y
# CONFIG_HIDRAW is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
# CONFIG_HID_PID is not set
CONFIG_USB_HIDDEV=y

#
# Special HID drivers
#
CONFIG_HID_A4TECH=y
# CONFIG_HID_ACRUX is not set
CONFIG_HID_APPLE=y
CONFIG_HID_BELKIN=y
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
# CONFIG_HID_PRODIKEYS is not set
CONFIG_HID_CYPRESS=y
CONFIG_HID_DRAGONRISE=y
# CONFIG_DRAGONRISE_FF is not set
# CONFIG_HID_EMS_FF is not set
CONFIG_HID_EZKEY=y
# CONFIG_HID_HOLTEK is not set
# CONFIG_HID_KEYTOUCH is not set
CONFIG_HID_KYE=y
# CONFIG_HID_UCLOGIC is not set
# CONFIG_HID_WALTOP is not set
CONFIG_HID_GYRATION=y
CONFIG_HID_TWINHAN=y
CONFIG_HID_KENSINGTON=y
# CONFIG_HID_LCPOWER is not set
CONFIG_HID_LOGITECH=y
CONFIG_HID_LOGITECH_DJ=y
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
# CONFIG_LOGIG940_FF is not set
# CONFIG_LOGIWHEELS_FF is not set
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
# CONFIG_HID_MULTITOUCH is not set
CONFIG_HID_NTRIG=y
CONFIG_HID_ORTEK=y
CONFIG_HID_PANTHERLORD=y
# CONFIG_PANTHERLORD_FF is not set
CONFIG_HID_PETALYNX=y
# CONFIG_HID_PICOLCD is not set
# CONFIG_HID_PRIMAX is not set
# CONFIG_HID_ROCCAT is not set
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SONY=y
# CONFIG_HID_SPEEDLINK is not set
CONFIG_HID_SUNPLUS=y
CONFIG_HID_GREENASIA=y
# CONFIG_GREENASIA_FF is not set
CONFIG_HID_SMARTJOYPLUS=y
# CONFIG_SMARTJOYPLUS_FF is not set
CONFIG_HID_TOPSEED=y
CONFIG_HID_THRUSTMASTER=y
# CONFIG_THRUSTMASTER_FF is not set
CONFIG_HID_ZEROPLUS=y
# CONFIG_ZEROPLUS_FF is not set
# CONFIG_HID_ZYDACRON is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB_ARCH_HAS_XHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

#
# Miscellaneous USB options
#
# CONFIG_USB_DEVICEFS is not set
# CONFIG_USB_DEVICE_CLASS is not set
CONFIG_USB_DYNAMIC_MINORS=y
CONFIG_USB_SUSPEND=y
# CONFIG_USB_OTG is not set
# CONFIG_USB_MON is not set
CONFIG_USB_WUSB=y
CONFIG_USB_WUSB_CBAF=y
# CONFIG_USB_WUSB_CBAF_DEBUG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_C67X00_HCD=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_HCD_DEBUGGING=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
CONFIG_USB_EHCI_MV=y
CONFIG_USB_OXU210HP_HCD=y
CONFIG_USB_ISP116X_HCD=y
CONFIG_USB_ISP1760_HCD=y
CONFIG_USB_ISP1362_HCD=y
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=y
# CONFIG_USB_U132_HCD is not set
CONFIG_USB_SL811_HCD=y
# CONFIG_USB_SL811_HCD_ISO is not set
CONFIG_USB_R8A66597_HCD=y
CONFIG_USB_WHCI_HCD=y
CONFIG_USB_HWA_HCD=y

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=y
CONFIG_USB_WDM=y
CONFIG_USB_TMC=y

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=y
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_REALTEK is not set
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_USBAT=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
CONFIG_USB_STORAGE_ALAUDA=y
CONFIG_USB_STORAGE_ONETOUCH=y
CONFIG_USB_STORAGE_KARMA=y
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
CONFIG_USB_STORAGE_ENE_UB6250=y
CONFIG_USB_UAS=y
CONFIG_USB_LIBUSUAL=y

#
# USB Imaging devices
#
CONFIG_USB_MDC800=y
CONFIG_USB_MICROTEK=y

#
# USB port drivers
#
CONFIG_USB_SERIAL=y
CONFIG_USB_SERIAL_CONSOLE=y
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_AIRCABLE is not set
# CONFIG_USB_SERIAL_ARK3116 is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_CH341 is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_CP210X is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
# CONFIG_USB_SERIAL_EMPEG is not set
CONFIG_USB_SERIAL_FTDI_SIO=y
# CONFIG_USB_SERIAL_FUNSOFT is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_GARMIN is not set
# CONFIG_USB_SERIAL_IPW is not set
# CONFIG_USB_SERIAL_IUU is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_MOS7720 is not set
# CONFIG_USB_SERIAL_MOS7840 is not set
# CONFIG_USB_SERIAL_MOTOROLA is not set
# CONFIG_USB_SERIAL_NAVMAN is not set
CONFIG_USB_SERIAL_PL2303=y
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_QCAUX is not set
# CONFIG_USB_SERIAL_QUALCOMM is not set
# CONFIG_USB_SERIAL_SPCP8X5 is not set
# CONFIG_USB_SERIAL_HP4X is not set
# CONFIG_USB_SERIAL_SAFE is not set
# CONFIG_USB_SERIAL_SIEMENS_MPI is not set
# CONFIG_USB_SERIAL_SIERRAWIRELESS is not set
# CONFIG_USB_SERIAL_SYMBOL is not set
# CONFIG_USB_SERIAL_TI is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
# CONFIG_USB_SERIAL_OPTION is not set
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_SERIAL_OPTICON is not set
# CONFIG_USB_SERIAL_VIVOPAY_SERIAL is not set
# CONFIG_USB_SERIAL_ZIO is not set
# CONFIG_USB_SERIAL_SSU100 is not set
CONFIG_USB_SERIAL_DEBUG=y

#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=y
CONFIG_USB_EMI26=y
CONFIG_USB_ADUTUX=y
CONFIG_USB_SEVSEG=y
CONFIG_USB_RIO500=y
CONFIG_USB_LEGOTOWER=y
CONFIG_USB_LCD=y
CONFIG_USB_LED=y
CONFIG_USB_CYPRESS_CY7C63=y
CONFIG_USB_CYTHERM=y
CONFIG_USB_IDMOUSE=y
CONFIG_USB_FTDI_ELAN=y
CONFIG_USB_APPLEDISPLAY=y
CONFIG_USB_SISUSBVGA=y
# CONFIG_USB_SISUSBVGA_CON is not set
CONFIG_USB_LD=y
# CONFIG_USB_TRANCEVIBRATOR is not set
CONFIG_USB_IOWARRIOR=y
# CONFIG_USB_TEST is not set
CONFIG_USB_ISIGHTFW=y
# CONFIG_USB_YUREX is not set
# CONFIG_USB_GADGET is not set

#
# OTG and related infrastructure
#
# CONFIG_USB_GPIO_VBUS is not set
# CONFIG_NOP_USB_XCEIV is not set
CONFIG_UWB=y
CONFIG_UWB_HWA=y
CONFIG_UWB_WHCI=y
CONFIG_UWB_I1480U=y
CONFIG_MMC=y
# CONFIG_MMC_DEBUG is not set
# CONFIG_MMC_UNSAFE_RESUME is not set
# CONFIG_MMC_CLKGATE is not set

#
# MMC/SD/SDIO Card Drivers
#
CONFIG_MMC_BLOCK=y
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_MMC_BLOCK_BOUNCE=y
CONFIG_SDIO_UART=y
# CONFIG_MMC_TEST is not set

#
# MMC/SD/SDIO Host Controller Drivers
#
CONFIG_MMC_SDHCI=y
CONFIG_MMC_SDHCI_PCI=y
# CONFIG_MMC_RICOH_MMC is not set
CONFIG_MMC_SDHCI_PLTFM=y
CONFIG_MMC_WBSD=y
CONFIG_MMC_TIFM_SD=y
CONFIG_MMC_SPI=y
CONFIG_MMC_CB710=y
CONFIG_MMC_VIA_SDMMC=y
CONFIG_MMC_VUB300=y
CONFIG_MMC_USHC=y
CONFIG_MEMSTICK=y
CONFIG_MEMSTICK_DEBUG=y

#
# MemoryStick drivers
#
# CONFIG_MEMSTICK_UNSAFE_RESUME is not set
CONFIG_MSPRO_BLOCK=y

#
# MemoryStick Host Controller Drivers
#
CONFIG_MEMSTICK_TIFM_MS=y
CONFIG_MEMSTICK_JMICRON_38X=y
# CONFIG_MEMSTICK_R592 is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

#
# LED drivers
#
# CONFIG_LEDS_LM3530 is not set
# CONFIG_LEDS_PCA9532 is not set
# CONFIG_LEDS_GPIO is not set
# CONFIG_LEDS_LP3944 is not set
# CONFIG_LEDS_LP5521 is not set
# CONFIG_LEDS_LP5523 is not set
# CONFIG_LEDS_CLEVO_MAIL is not set
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_DAC124S085 is not set
# CONFIG_LEDS_BD2802 is not set
# CONFIG_LEDS_INTEL_SS4200 is not set
# CONFIG_LEDS_LT3593 is not set
# CONFIG_LEDS_TCA6507 is not set
# CONFIG_LEDS_OT200 is not set
CONFIG_LEDS_TRIGGERS=y

#
# LED Triggers
#
CONFIG_LEDS_TRIGGER_TIMER=y
CONFIG_LEDS_TRIGGER_HEARTBEAT=y
# CONFIG_LEDS_TRIGGER_BACKLIGHT is not set
# CONFIG_LEDS_TRIGGER_GPIO is not set
# CONFIG_LEDS_TRIGGER_DEFAULT_ON is not set

#
# iptables trigger is under Netfilter config (LED target)
#
# CONFIG_ACCESSIBILITY is not set
CONFIG_INFINIBAND=y
CONFIG_INFINIBAND_USER_MAD=y
CONFIG_INFINIBAND_USER_ACCESS=y
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_MTHCA=y
CONFIG_INFINIBAND_MTHCA_DEBUG=y
CONFIG_INFINIBAND_IPATH=y
CONFIG_INFINIBAND_QIB=y
CONFIG_INFINIBAND_AMSO1100=y
# CONFIG_INFINIBAND_AMSO1100_DEBUG is not set
CONFIG_INFINIBAND_CXGB3=y
# CONFIG_INFINIBAND_CXGB3_DEBUG is not set
CONFIG_INFINIBAND_CXGB4=y
CONFIG_MLX4_INFINIBAND=y
CONFIG_INFINIBAND_NES=y
# CONFIG_INFINIBAND_NES_DEBUG is not set
CONFIG_INFINIBAND_IPOIB=y
CONFIG_INFINIBAND_IPOIB_CM=y
CONFIG_INFINIBAND_IPOIB_DEBUG=y
CONFIG_INFINIBAND_IPOIB_DEBUG_DATA=y
CONFIG_INFINIBAND_SRP=y
CONFIG_INFINIBAND_ISER=y
CONFIG_EDAC=y

#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
# CONFIG_EDAC_DECODE_MCE is not set
# CONFIG_EDAC_MM_EDAC is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_DS1307 is not set
# CONFIG_RTC_DRV_DS1374 is not set
# CONFIG_RTC_DRV_DS1672 is not set
# CONFIG_RTC_DRV_DS3232 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
# CONFIG_RTC_DRV_RS5C372 is not set
# CONFIG_RTC_DRV_ISL1208 is not set
# CONFIG_RTC_DRV_ISL12022 is not set
# CONFIG_RTC_DRV_X1205 is not set
# CONFIG_RTC_DRV_PCF8563 is not set
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_BQ32K is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
# CONFIG_RTC_DRV_RX8581 is not set
# CONFIG_RTC_DRV_RX8025 is not set
# CONFIG_RTC_DRV_EM3027 is not set
# CONFIG_RTC_DRV_RV3029C2 is not set

#
# SPI RTC drivers
#
# CONFIG_RTC_DRV_M41T93 is not set
# CONFIG_RTC_DRV_M41T94 is not set
# CONFIG_RTC_DRV_DS1305 is not set
# CONFIG_RTC_DRV_DS1390 is not set
# CONFIG_RTC_DRV_MAX6902 is not set
# CONFIG_RTC_DRV_R9701 is not set
# CONFIG_RTC_DRV_RS5C348 is not set
# CONFIG_RTC_DRV_DS3234 is not set
# CONFIG_RTC_DRV_PCF2123 is not set

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
# CONFIG_RTC_DRV_DS1553 is not set
# CONFIG_RTC_DRV_DS1742 is not set
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_MSM6242 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_RP5C01 is not set
# CONFIG_RTC_DRV_V3020 is not set

#
# on-CPU RTC drivers
#
CONFIG_DMADEVICES=y
CONFIG_DMADEVICES_DEBUG=y
CONFIG_DMADEVICES_VDEBUG=y

#
# DMA Devices
#
CONFIG_INTEL_MID_DMAC=y
CONFIG_INTEL_IOATDMA=y
CONFIG_TIMB_DMA=y
CONFIG_PCH_DMA=y
CONFIG_DMA_ENGINE=y

#
# DMA Clients
#
CONFIG_NET_DMA=y
CONFIG_ASYNC_TX_DMA=y
# CONFIG_DMATEST is not set
CONFIG_DCA=y
CONFIG_AUXDISPLAY=y
CONFIG_UIO=y
# CONFIG_UIO_CIF is not set
CONFIG_UIO_PDRV=y
CONFIG_UIO_PDRV_GENIRQ=y
# CONFIG_UIO_AEC is not set
# CONFIG_UIO_SERCOS3 is not set
# CONFIG_UIO_PCI_GENERIC is not set
# CONFIG_UIO_NETX is not set
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y

#
# Virtio drivers
#
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
# CONFIG_VIRTIO_MMIO is not set

#
# Microsoft Hyper-V guest support
#
# CONFIG_HYPERV is not set
# CONFIG_STAGING is not set
CONFIG_X86_PLATFORM_DEVICES=y
# CONFIG_ACERHDF is not set
# CONFIG_ASUS_LAPTOP is not set
# CONFIG_FUJITSU_LAPTOP is not set
# CONFIG_HP_ACCEL is not set
# CONFIG_PANASONIC_LAPTOP is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_INTEL_MENLOW is not set
# CONFIG_EEEPC_LAPTOP is not set
# CONFIG_ACPI_WMI is not set
# CONFIG_ACPI_ASUS is not set
# CONFIG_TOPSTAR_LAPTOP is not set
# CONFIG_ACPI_TOSHIBA is not set
# CONFIG_TOSHIBA_BT_RFKILL is not set
# CONFIG_ACPI_CMPC is not set
# CONFIG_INTEL_IPS is not set
# CONFIG_IBM_RTL is not set
# CONFIG_XO15_EBOOK is not set
# CONFIG_SAMSUNG_Q10 is not set

#
# Hardware Spinlock drivers
#
CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_AMD_IOMMU=y
# CONFIG_AMD_IOMMU_STATS is not set
CONFIG_DMAR_TABLE=y
CONFIG_INTEL_IOMMU=y
CONFIG_INTEL_IOMMU_DEFAULT_ON=y
CONFIG_INTEL_IOMMU_FLOPPY_WA=y
CONFIG_IRQ_REMAP=y
CONFIG_VIRT_DRIVERS=y
# CONFIG_PM_DEVFREQ is not set

#
# Firmware Drivers
#
CONFIG_EDD=y
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_EFI_VARS=y
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=y
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=y
# CONFIG_GOOGLE_FIRMWARE is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT2_FS_XIP=y
CONFIG_EXT3_FS=y
# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_FS_XIP=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
# CONFIG_REISERFS_CHECK is not set
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
CONFIG_GFS2_FS=y
# CONFIG_GFS2_FS_LOCKING_DLM is not set
# CONFIG_OCFS2_FS is not set
CONFIG_BTRFS_FS=y
# CONFIG_BTRFS_FS_POSIX_ACL is not set
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y
# CONFIG_NILFS2_FS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_FANOTIFY is not set
CONFIG_QUOTA=y
# CONFIG_QUOTA_NETLINK_INTERFACE is not set
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
# CONFIG_QFMT_V1 is not set
# CONFIG_QFMT_V2 is not set
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=y
# CONFIG_FUSE_FS is not set

#
# Caches
#
CONFIG_FSCACHE=y
CONFIG_FSCACHE_STATS=y
CONFIG_FSCACHE_HISTOGRAM=y
# CONFIG_FSCACHE_DEBUG is not set
# CONFIG_FSCACHE_OBJECT_LIST is not set
CONFIG_CACHEFILES=y
# CONFIG_CACHEFILES_DEBUG is not set
CONFIG_CACHEFILES_HISTOGRAM=y

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=y
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_TMPFS_XATTR is not set
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=y
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_LOGFS is not set
CONFIG_CRAMFS=y
CONFIG_SQUASHFS=y
# CONFIG_SQUASHFS_XATTR is not set
CONFIG_SQUASHFS_ZLIB=y
# CONFIG_SQUASHFS_LZO is not set
# CONFIG_SQUASHFS_XZ is not set
# CONFIG_SQUASHFS_4K_DEVBLK_SIZE is not set
CONFIG_SQUASHFS_EMBEDDED=y
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
CONFIG_VXFS_FS=y
CONFIG_MINIX_FS=y
CONFIG_OMFS_FS=y
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
CONFIG_ROMFS_FS=y
CONFIG_ROMFS_BACKED_BY_BLOCK=y
CONFIG_ROMFS_ON_BLOCK=y
CONFIG_PSTORE=y
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_EXOFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=y
# CONFIG_NFS_V4_1 is not set
CONFIG_ROOT_NFS=y
CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
# CONFIG_NFS_USE_NEW_IDMAPPER is not set
# CONFIG_NFSD is not set
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_SUNRPC_XPRT_RDMA=y
# CONFIG_CEPH_FS is not set
CONFIG_CIFS=y
CONFIG_CIFS_STATS=y
CONFIG_CIFS_STATS2=y
CONFIG_CIFS_WEAK_PW_HASH=y
# CONFIG_CIFS_UPCALL is not set
# CONFIG_CIFS_XATTR is not set
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DFS_UPCALL is not set
# CONFIG_CIFS_FSCACHE is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y
CONFIG_DLM=y
# CONFIG_DLM_DEBUG is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_HARDLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=400
# CONFIG_DEBUG_KMEMLEAK_TEST is not set
# CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF is not set
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_PI_LIST=y
# CONFIG_RT_MUTEX_TESTER is not set
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
# CONFIG_PROVE_RCU is not set
# CONFIG_SPARSE_RCU_POINTER is not set
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
CONFIG_DEBUG_LOCKDEP=y
CONFIG_TRACE_IRQFLAGS=y
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=y
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_LIST=y
# CONFIG_TEST_LIST_SORT is not set
CONFIG_DEBUG_SG=y
CONFIG_DEBUG_NOTIFIERS=y
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=300
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
CONFIG_LKDTM=m
# CONFIG_CPU_NOTIFIER_ERROR_INJECT is not set
CONFIG_FAULT_INJECTION=y
CONFIG_FAILSLAB=y
CONFIG_FAIL_PAGE_ALLOC=y
CONFIG_FAIL_MAKE_REQUEST=y
# CONFIG_FAIL_IO_TIMEOUT is not set
# CONFIG_FAIL_MMC_REQUEST is not set
CONFIG_FAULT_INJECTION_DEBUG_FS=y
CONFIG_LATENCYTOP=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FTRACE_NMI_ENTER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_RING_BUFFER=y
CONFIG_FTRACE_NMI_ENTER=y
CONFIG_EVENT_TRACING=y
# CONFIG_EVENT_POWER_TRACING_DEPRECATED is not set
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_IRQSOFF_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
CONFIG_MMIOTRACE=y
# CONFIG_MMIOTRACE_TEST is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DMA_API_DEBUG is not set
CONFIG_ATOMIC64_SELFTEST=y
# CONFIG_ASYNC_RAID6_TEST is not set
CONFIG_SAMPLES=y
# CONFIG_SAMPLE_TRACEPOINTS is not set
CONFIG_SAMPLE_TRACE_EVENTS=m
CONFIG_SAMPLE_KOBJECT=m
# CONFIG_SAMPLE_KPROBES is not set
# CONFIG_SAMPLE_HW_BREAKPOINT is not set
# CONFIG_SAMPLE_KFIFO is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_TEST_KSTRTOX is not set
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
CONFIG_X86_PTDUMP=y
# CONFIG_DEBUG_RODATA is not set
# CONFIG_DEBUG_SET_MODULE_RONX is not set
# CONFIG_DEBUG_NX_TEST is not set
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
# CONFIG_X86_DECODER_SELFTEST is not set
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
# CONFIG_CPA_DEBUG is not set
# CONFIG_OPTIMIZE_INLINING is not set
CONFIG_DEBUG_STRICT_USER_COPY_CHECKS=y
# CONFIG_DEBUG_NMI_SELFTEST is not set

#
# Security options
#
CONFIG_KEYS=y
# CONFIG_TRUSTED_KEYS is not set
# CONFIG_ENCRYPTED_KEYS is not set
# CONFIG_KEYS_DEBUG_PROC_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
CONFIG_SECURITYFS=y
# CONFIG_INTEL_TXT is not set
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_XOR_BLOCKS=y
CONFIG_ASYNC_CORE=y
CONFIG_ASYNC_MEMCPY=y
CONFIG_ASYNC_XOR=y
CONFIG_ASYNC_PQ=y
CONFIG_ASYNC_RAID6_RECOV=y
CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
# CONFIG_CRYPTO_USER is not set
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
# CONFIG_CRYPTO_GF128MUL is not set
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_PCRYPT is not set
CONFIG_CRYPTO_WORKQUEUE=y
# CONFIG_CRYPTO_CRYPTD is not set
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_TEST is not set

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=y
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set
# CONFIG_CRYPTO_VMAC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CRC32C_INTEL is not set
# CONFIG_CRYPTO_GHASH is not set
CONFIG_CRYPTO_MD4=y
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=y
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
# CONFIG_CRYPTO_SHA1_SSSE3 is not set
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set
# CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL is not set

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_X86_64 is not set
# CONFIG_CRYPTO_AES_NI_INTEL is not set
# CONFIG_CRYPTO_ANUBIS is not set
CONFIG_CRYPTO_ARC4=y
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_BLOWFISH_X86_64 is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
CONFIG_CRYPTO_DES=y
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_SERPENT_SSE2_X86_64 is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_TWOFISH_X86_64 is not set
# CONFIG_CRYPTO_TWOFISH_X86_64_3WAY is not set

#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_ZLIB is not set
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
CONFIG_CRYPTO_HW=y
# CONFIG_CRYPTO_DEV_PADLOCK is not set
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
CONFIG_KVM_INTEL=y
CONFIG_KVM_AMD=y
# CONFIG_KVM_MMU_AUDIT is not set
# CONFIG_VHOST_NET is not set
CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_RAID6_PQ=y
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=y
# CONFIG_CRC8 is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
# CONFIG_AVERAGE is not set
# CONFIG_CORDIC is not set

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Mon, Feb 13, 2012 at 08:52:19AM -0800, Yinghai Lu wrote:
> On Mon, Feb 13, 2012 at 7:28 AM, Don Zickus <dzickus@redhat.com> wrote:
> > On Sat, Feb 11, 2012 at 05:04:15PM -0800, Yinghai Lu wrote:
> >> >
> >> > v2: removed the disable lapic code too
> >>
> >> with this commit, kdump is not working anymore on my setups with
> >> Nehalem, Westmere, sandbridge.
> >> these setup all have VT-d enabled.
> >
> > Hi Yinghai,
> >
> > Thanks for the report.  Can you attach the .config you were using and what
> > vendor you were using for Nehalem, Westmere and Sandybridge.  I'll try to
> > reproduce it in my lab.
> 
> Sure, please check attached one

Looks like I can reproduce the hang with my original patch and things work
when we stick the disable_local_APIC() back in there.  I'll keep poking.

Cheers,
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Mon, Feb 13, 2012 at 05:12:58PM -0500, Don Zickus wrote:
> On Mon, Feb 13, 2012 at 08:52:19AM -0800, Yinghai Lu wrote:
> > On Mon, Feb 13, 2012 at 7:28 AM, Don Zickus <dzickus@redhat.com> wrote:
> > > On Sat, Feb 11, 2012 at 05:04:15PM -0800, Yinghai Lu wrote:
> > >> >
> > >> > v2: removed the disable lapic code too
> > >>
> > >> with this commit, kdump is not working anymore on my setups with
> > >> Nehalem, Westmere, sandbridge.
> > >> these setup all have VT-d enabled.
> > >
> > > Hi Yinghai,
> > >
> > > Thanks for the report.  Can you attach the .config you were using and what
> > > vendor you were using for Nehalem, Westmere and Sandybridge.  I'll try to
> > > reproduce it in my lab.
> > 
> > Sure, please check attached one
> 
> Looks like I can reproduce the hang with my original patch and things work
> when we stick the disable_local_APIC() back in there.  I'll keep poking.

Hmm, my config worked fine in both cases, hence why I didn't catch it (its
based on a bloated Fedora config).  I'll look at the differences to see if
something pops out.

Cheers,
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Mon, Feb 13, 2012 at 08:52:19AM -0800, Yinghai Lu wrote:
> On Mon, Feb 13, 2012 at 7:28 AM, Don Zickus <dzickus@redhat.com> wrote:
> > On Sat, Feb 11, 2012 at 05:04:15PM -0800, Yinghai Lu wrote:
> >> >
> >> > v2: removed the disable lapic code too
> >>
> >> with this commit, kdump is not working anymore on my setups with
> >> Nehalem, Westmere, sandbridge.
> >> these setup all have VT-d enabled.
> >
> > Hi Yinghai,
> >
> > Thanks for the report.  Can you attach the .config you were using and what
> > vendor you were using for Nehalem, Westmere and Sandybridge.  I'll try to
> > reproduce it in my lab.
> 
> Sure, please check attached one

For some reason using gzip instead of xz makes things work for me with
your config.

CONFIG_KERNEL_GZIP=y
CONFIG_KERNEL_XZ=n

Do you see the same thing?

Cheers,
Don

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Yinghai Lu]

On Wed, Feb 15, 2012 at 6:53 PM, Don Zickus <dzickus@redhat.com> wrote:
>
> For some reason using gzip instead of xz makes things work for me with
> your config.
>
> CONFIG_KERNEL_GZIP=y
> CONFIG_KERNEL_XZ=n
>
> Do you see the same thing?

no.

with GZIP still get instant reset.

Yinghai

Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[Don Zickus]

On Thu, Feb 16, 2012 at 10:43:06AM -0800, Yinghai Lu wrote:
> On Wed, Feb 15, 2012 at 6:53 PM, Don Zickus <dzickus@redhat.com> wrote:
> >
> > For some reason using gzip instead of xz makes things work for me with
> > your config.
> >
> > CONFIG_KERNEL_GZIP=y
> > CONFIG_KERNEL_XZ=n
> >
> > Do you see the same thing?
> 
> no.
> 
> with GZIP still get instant reset.

What about my other idea, 'echo 0 > /proc/sys/kernel/nmi_watchdog'?

Cheers,
Don