A question of perf NMI handler

A question of perf NMI handler

[Lin Ming]

With nmi_watchdog enabled, perf_event_nmi_handler always return
NOTIFY_STOP(active_events > 0), and the notifier call chain will not
call further.

If it was not perf NMI, does the perf nmi handler may stop the real NMI
handler get called because NOTIFY_STOP is returned??

static int __kprobes
perf_event_nmi_handler(struct notifier_block *self,
			 unsigned long cmd, void *__args)
{
	struct die_args *args = __args;
	struct pt_regs *regs;

	if (!atomic_read(&active_events)) ===> With nmi_watchdog enabled, active_events > 0
		return NOTIFY_DONE;

	switch (cmd) {
	case DIE_NMI:
	case DIE_NMI_IPI:
		break;

	default:
		return NOTIFY_DONE;
	}

	regs = args->regs;

	apic_write(APIC_LVTPC, APIC_DM_NMI);
	/*
	 * Can't rely on the handled return value to say it was our NMI, two
	 * events could trigger 'simultaneously' raising two back-to-back NMIs.
	 *
	 * If the first NMI handles both, the latter will be empty and daze
	 * the CPU.
	 */
	x86_pmu.handle_irq(regs);

	return NOTIFY_STOP;
}

Thanks,
Lin Ming

Re: A question of perf NMI handler

[Peter Zijlstra]

On Wed, 2010-08-04 at 17:21 +0800, Lin Ming wrote:
> With nmi_watchdog enabled, perf_event_nmi_handler always return
> NOTIFY_STOP(active_events > 0), and the notifier call chain will not
> call further.
> 
> If it was not perf NMI, does the perf nmi handler may stop the real NMI
> handler get called because NOTIFY_STOP is returned??
> 
> static int __kprobes
> perf_event_nmi_handler(struct notifier_block *self,
> 			 unsigned long cmd, void *__args)
> {
> 	struct die_args *args = __args;
> 	struct pt_regs *regs;
> 
> 	if (!atomic_read(&active_events)) ===> With nmi_watchdog enabled, active_events > 0
> 		return NOTIFY_DONE;
> 
> 	switch (cmd) {
> 	case DIE_NMI:
> 	case DIE_NMI_IPI:
> 		break;
> 
> 	default:
> 		return NOTIFY_DONE;
> 	}
> 
> 	regs = args->regs;
> 
> 	apic_write(APIC_LVTPC, APIC_DM_NMI);
> 	/*
> 	 * Can't rely on the handled return value to say it was our NMI, two
> 	 * events could trigger 'simultaneously' raising two back-to-back NMIs.
> 	 *
> 	 * If the first NMI handles both, the latter will be empty and daze
> 	 * the CPU.
> 	 */
> 	x86_pmu.handle_irq(regs);
> 
> 	return NOTIFY_STOP;
> }

Urgh,.. right, so what is the alternative? we don't seem to have a
reliable way of telling where the NMI originated from.

As that comment says, the PMU can raise the NMI and raise the pending
NMI latch for a second over-run, at which point the first NMI will
likely see the overflow status for both, clear both, and the second NMI
will see a 0 overflow status, return it wasn't the PMU, but since the
PMU did raise it, nobody else will claim it, and we get these silly
dazed and confused thingies.

What NMI source are you concerned about and can it reliably tell if it
raised the NMI or not?

Re: A question of perf NMI handler

[Robert Richter]

On 04.08.10 05:21:10, Lin Ming wrote:

> With nmi_watchdog enabled, perf_event_nmi_handler always return
> NOTIFY_STOP(active_events > 0), and the notifier call chain will not
> call further.
> 
> If it was not perf NMI, does the perf nmi handler may stop the real NMI
> handler get called because NOTIFY_STOP is returned??

There is no general mechanism for recording the NMI source (except if
it was external triggered, e.g. by the southbridge). Also, all nmis
are mapped to NMI vector 2 and therefore there is no way to find out
the reason by using apic mask registers.

Now, if multiple perfctrs trigger an nmi, it may happen that a handler
has nothing to do because the counter was already handled by the
previous one. Thus, it is valid to have unhandled nmis caused by
perfctrs.

So, with counters enabled we always have to return stop for *all* nmis
as we cannot detect that it was an perfctr nmi. Otherwise we could
trigger an unhandled nmi. To ensure that all other nmi handlers are
called, the perfctr's nmi handler must have the lowest priority. Then,
the handler will be the last in the chain.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: A question of perf NMI handler

[Peter Zijlstra]

On Wed, 2010-08-04 at 12:01 +0200, Robert Richter wrote:
> To ensure that all other nmi handlers are
> called, the perfctr's nmi handler must have the lowest priority. Then,
> the handler will be the last in the chain. 

Well, unless another NMI handler has the exact same issue and also
starts eating all NMIs, just in case.

Re: A question of perf NMI handler

[Robert Richter]

On 04.08.10 06:24:18, Peter Zijlstra wrote:
> On Wed, 2010-08-04 at 12:01 +0200, Robert Richter wrote:
> > To ensure that all other nmi handlers are
> > called, the perfctr's nmi handler must have the lowest priority. Then,
> > the handler will be the last in the chain. 
> 
> Well, unless another NMI handler has the exact same issue and also
> starts eating all NMIs, just in case.

In this case we will have to change the implementation for unhandled
nmis. But I don't know of other sources with this issue.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: A question of perf NMI handler

[Don Zickus]

On Wed, Aug 04, 2010 at 12:01:16PM +0200, Robert Richter wrote:
> There is no general mechanism for recording the NMI source (except if
> it was external triggered, e.g. by the southbridge). Also, all nmis
> are mapped to NMI vector 2 and therefore there is no way to find out
> the reason by using apic mask registers.

This is no different than a shared interrupt, no?  All the nmi handlers
need to check their own sources to see if they triggered it.  You can't
expect the generic nmi handler to determine this.

> 
> Now, if multiple perfctrs trigger an nmi, it may happen that a handler
> has nothing to do because the counter was already handled by the
> previous one. Thus, it is valid to have unhandled nmis caused by
> perfctrs.
> 
> So, with counters enabled we always have to return stop for *all* nmis
> as we cannot detect that it was an perfctr nmi. Otherwise we could
> trigger an unhandled nmi. To ensure that all other nmi handlers are
> called, the perfctr's nmi handler must have the lowest priority. Then,
> the handler will be the last in the chain.

But the cases this break are, external NMI buttons, broken firmware that
causes SERRs on the PCI bus, and any other general hardware failures.

So what the perf handler does is really unacceptable.  The only reason we
are noticing this now is because I put the nmi_watchdog on top of the perf
subsystem, so it always has a user and will trigger NOTIFY_STOP.  Before,
it never had a registerd user so instead returned NOTIFY_DONE and
everything worked great.

Cheers,
Don

Re: A question of perf NMI handler

[Peter Zijlstra]

On Wed, 2010-08-04 at 10:00 -0400, Don Zickus wrote:
> On Wed, Aug 04, 2010 at 12:01:16PM +0200, Robert Richter wrote:
> > There is no general mechanism for recording the NMI source (except if
> > it was external triggered, e.g. by the southbridge). Also, all nmis
> > are mapped to NMI vector 2 and therefore there is no way to find out
> > the reason by using apic mask registers.
> 
> This is no different than a shared interrupt, no?  All the nmi handlers
> need to check their own sources to see if they triggered it.  You can't
> expect the generic nmi handler to determine this.

Sure, but the problem is that the PMU can't reliably do that.

> > Now, if multiple perfctrs trigger an nmi, it may happen that a handler
> > has nothing to do because the counter was already handled by the
> > previous one. Thus, it is valid to have unhandled nmis caused by
> > perfctrs.
> > 
> > So, with counters enabled we always have to return stop for *all* nmis
> > as we cannot detect that it was an perfctr nmi. Otherwise we could
> > trigger an unhandled nmi. To ensure that all other nmi handlers are
> > called, the perfctr's nmi handler must have the lowest priority. Then,
> > the handler will be the last in the chain.
> 
> But the cases this break are, external NMI buttons, broken firmware that
> causes SERRs on the PCI bus, and any other general hardware failures.

It breaks broken firmware? :-) and you care?

> So what the perf handler does is really unacceptable.  The only reason we
> are noticing this now is because I put the nmi_watchdog on top of the perf
> subsystem, so it always has a user and will trigger NOTIFY_STOP.  Before,
> it never had a registerd user so instead returned NOTIFY_DONE and
> everything worked great.

Right so I looked up your thing and while that limits the damage in that
at some point it will let NMIs pass, it will still consume too many.
Meaning that Yinghai will have to potentially press his NMI button
several times before it registers.

Re: A question of perf NMI handler

[Don Zickus]

On Wed, Aug 04, 2010 at 04:11:33PM +0200, Peter Zijlstra wrote:
> On Wed, 2010-08-04 at 10:00 -0400, Don Zickus wrote:
> > On Wed, Aug 04, 2010 at 12:01:16PM +0200, Robert Richter wrote:
> > > There is no general mechanism for recording the NMI source (except if
> > > it was external triggered, e.g. by the southbridge). Also, all nmis
> > > are mapped to NMI vector 2 and therefore there is no way to find out
> > > the reason by using apic mask registers.
> > 
> > This is no different than a shared interrupt, no?  All the nmi handlers
> > need to check their own sources to see if they triggered it.  You can't
> > expect the generic nmi handler to determine this.
> 
> Sure, but the problem is that the PMU can't reliably do that.

Right, but that is because there is no bit that says the PMU generated the
nmi.  But for the most part checking to see if the PMU is >0 is good
enough, no?

> 
> > > Now, if multiple perfctrs trigger an nmi, it may happen that a handler
> > > has nothing to do because the counter was already handled by the
> > > previous one. Thus, it is valid to have unhandled nmis caused by
> > > perfctrs.
> > > 
> > > So, with counters enabled we always have to return stop for *all* nmis
> > > as we cannot detect that it was an perfctr nmi. Otherwise we could
> > > trigger an unhandled nmi. To ensure that all other nmi handlers are
> > > called, the perfctr's nmi handler must have the lowest priority. Then,
> > > the handler will be the last in the chain.
> > 
> > But the cases this break are, external NMI buttons, broken firmware that
> > causes SERRs on the PCI bus, and any other general hardware failures.
> 
> It breaks broken firmware? :-) and you care?

Absolutely.  When a customer complains they upgraded their RHEL kernel and
the box suddenly hangs on boot trying to access the storage device, yes I
care.  Because a flood of NMIs would indiciate something is fishy with
the firmware (in this case it was a network card though it hung on storage
access).  Swallowing the NMIs would just cause everyone to waste weeks of
their time trying to figure it out (you don't want to know how many weeks
were wasted in RHEL-6 across multiple machines only to find out it was
broken firmware on a card that no one suspected as being the culprit).

As much as I hate broken firmware, it is becoming common place, and the
faster the kernel can point it out through unknown nmis, the faster we can
get the vendor involved to fix it.

> 
> > So what the perf handler does is really unacceptable.  The only reason we
> > are noticing this now is because I put the nmi_watchdog on top of the perf
> > subsystem, so it always has a user and will trigger NOTIFY_STOP.  Before,
> > it never had a registerd user so instead returned NOTIFY_DONE and
> > everything worked great.
> 
> Right so I looked up your thing and while that limits the damage in that
> at some point it will let NMIs pass, it will still consume too many.
> Meaning that Yinghai will have to potentially press his NMI button
> several times before it registers.

Ok.  Thanks for reviewing.  How does it consume to many?  I probably don't
understand how perf is being used in the non-simple scenarios.

Cheers,
Don

Re: A question of perf NMI handler

[Peter Zijlstra]

On Wed, 2010-08-04 at 10:52 -0400, Don Zickus wrote:
> > Right so I looked up your thing and while that limits the damage in that
> > at some point it will let NMIs pass, it will still consume too many.
> > Meaning that Yinghai will have to potentially press his NMI button
> > several times before it registers.
> 
> Ok.  Thanks for reviewing.  How does it consume to many?  I probably don't
> understand how perf is being used in the non-simple scenarios.

Suppose you have 4 counters (AMD, intel-nhm+), when more than 2 overflow
the first will raise the PMI, if the other 2+ overflow before we disable
the PMU it will try to raise 2+ more PMIs, but because hardware only has
a single interrupt pending bit it will at most cause a single extra
interrupt after we finish servicing the first one.

So then the first interrupt will see 3+ overflows, return 3+, and will
thus eat 2+ NMIs, only one of which will be the pending interrupt,
leaving 1+ NMIs from other sources to consume unhandled.

In which case Yinghai will have to press his NMI button 2+ times before
it registers.

That said, that might be a better situation than always consuming
unknown NMIs.. 

Re: A question of perf NMI handler

[Cyrill Gorcunov]

On Wed, Aug 04, 2010 at 05:02:41PM +0200, Peter Zijlstra wrote:
> On Wed, 2010-08-04 at 10:52 -0400, Don Zickus wrote:
> > > Right so I looked up your thing and while that limits the damage in that
> > > at some point it will let NMIs pass, it will still consume too many.
> > > Meaning that Yinghai will have to potentially press his NMI button
> > > several times before it registers.
> > 
> > Ok.  Thanks for reviewing.  How does it consume to many?  I probably don't
> > understand how perf is being used in the non-simple scenarios.
> 
> Suppose you have 4 counters (AMD, intel-nhm+), when more than 2 overflow
> the first will raise the PMI, if the other 2+ overflow before we disable
> the PMU it will try to raise 2+ more PMIs, but because hardware only has
> a single interrupt pending bit it will at most cause a single extra
> interrupt after we finish servicing the first one.
> 
> So then the first interrupt will see 3+ overflows, return 3+, and will
> thus eat 2+ NMIs, only one of which will be the pending interrupt,
> leaving 1+ NMIs from other sources to consume unhandled.
> 
> In which case Yinghai will have to press his NMI button 2+ times before
> it registers.
> 
> That said, that might be a better situation than always consuming
> unknown NMIs.. 
> 

Well, first I guess having Yinghai CC'ed is a bonus ;)
The second thing is that I don't get why perf handler can't be _last_
call in default_do_nmi, if there were any nmi with reason (serr or parity)
I think they should be calling first which of course don't eliminate
the former issue but somewhat make it weaken.

	-- Cyrill

Re: A question of perf NMI handler

[Don Zickus]

On Wed, Aug 04, 2010 at 07:18:58PM +0400, Cyrill Gorcunov wrote:
> On Wed, Aug 04, 2010 at 05:02:41PM +0200, Peter Zijlstra wrote:
> > On Wed, 2010-08-04 at 10:52 -0400, Don Zickus wrote:
> > > > Right so I looked up your thing and while that limits the damage in that
> > > > at some point it will let NMIs pass, it will still consume too many.
> > > > Meaning that Yinghai will have to potentially press his NMI button
> > > > several times before it registers.
> > > 
> > > Ok.  Thanks for reviewing.  How does it consume to many?  I probably don't
> > > understand how perf is being used in the non-simple scenarios.
> > 
> > Suppose you have 4 counters (AMD, intel-nhm+), when more than 2 overflow
> > the first will raise the PMI, if the other 2+ overflow before we disable
> > the PMU it will try to raise 2+ more PMIs, but because hardware only has
> > a single interrupt pending bit it will at most cause a single extra
> > interrupt after we finish servicing the first one.
> > 
> > So then the first interrupt will see 3+ overflows, return 3+, and will
> > thus eat 2+ NMIs, only one of which will be the pending interrupt,
> > leaving 1+ NMIs from other sources to consume unhandled.
> > 
> > In which case Yinghai will have to press his NMI button 2+ times before
> > it registers.
> > 
> > That said, that might be a better situation than always consuming
> > unknown NMIs.. 
> > 
> 
> Well, first I guess having Yinghai CC'ed is a bonus ;)
> The second thing is that I don't get why perf handler can't be _last_
> call in default_do_nmi, if there were any nmi with reason (serr or parity)
> I think they should be calling first which of course don't eliminate
> the former issue but somewhat make it weaken.

Because the reason registers are never set.  If they were, then the code
wouldn't have to walk the notify_chain. :-)

Unknown nmis are unknown nmis, nobody is claiming them.  Even worse, there
are customers that want to register their nmi handler below the perf
handler to claim all the unknown nmis, so they can be logged on the system
before being rebooted.

Cheers,
Don

Re: A question of perf NMI handler

[Cyrill Gorcunov]

On Wed, Aug 04, 2010 at 11:50:02AM -0400, Don Zickus wrote:
...
> > 
> > Well, first I guess having Yinghai CC'ed is a bonus ;)
> > The second thing is that I don't get why perf handler can't be _last_
> > call in default_do_nmi, if there were any nmi with reason (serr or parity)
> > I think they should be calling first which of course don't eliminate
> > the former issue but somewhat make it weaken.
> 
> Because the reason registers are never set.  If they were, then the code
> wouldn't have to walk the notify_chain. :-)
> 

maybe we're talking about different things. i meant that if there is nmi
with a reason (from 0x61) the handling of such nmi should be done before
notify_die I think (if only I not miss something behind).

> Unknown nmis are unknown nmis, nobody is claiming them.  Even worse, there
> are customers that want to register their nmi handler below the perf
> handler to claim all the unknown nmis, so they can be logged on the system
> before being rebooted.

well, perhaps we might need some kind of perf_chain in notifier code and
call for it after die_nmi chain, so the customers you mention may add own
chain for being called last.

> 
> Cheers,
> Don
> 
	-- Cyrill

Re: A question of perf NMI handler

[Don Zickus]

On Wed, Aug 04, 2010 at 08:10:46PM +0400, Cyrill Gorcunov wrote:
> On Wed, Aug 04, 2010 at 11:50:02AM -0400, Don Zickus wrote:
> ...
> > > 
> > > Well, first I guess having Yinghai CC'ed is a bonus ;)
> > > The second thing is that I don't get why perf handler can't be _last_
> > > call in default_do_nmi, if there were any nmi with reason (serr or parity)
> > > I think they should be calling first which of course don't eliminate
> > > the former issue but somewhat make it weaken.
> > 
> > Because the reason registers are never set.  If they were, then the code
> > wouldn't have to walk the notify_chain. :-)
> > 
> 
> maybe we're talking about different things. i meant that if there is nmi
> with a reason (from 0x61) the handling of such nmi should be done before
> notify_die I think (if only I not miss something behind).

No we are talking about the same thing. :-)  And that code is already
there.  The problem is the bits in register 0x61 are not always set
correctly in the case of SERRs (well at least in all the cases I have
dealt with).  So you can easily can a flood of unknown nmis from an SERR
and register 0x61 would have the PERR/SERR bits set to 0.  Fun, huh?

Cheers,
Don

Re: A question of perf NMI handler

[Cyrill Gorcunov]

On Wed, Aug 04, 2010 at 12:20:26PM -0400, Don Zickus wrote:
...
> > > 
> > > Because the reason registers are never set.  If they were, then the code
> > > wouldn't have to walk the notify_chain. :-)
> > > 
> > 
> > maybe we're talking about different things. i meant that if there is nmi
> > with a reason (from 0x61) the handling of such nmi should be done before
> > notify_die I think (if only I not miss something behind).
> 
> No we are talking about the same thing. :-)  And that code is already

seems not actually ;)

> there.  The problem is the bits in register 0x61 are not always set
> correctly in the case of SERRs (well at least in all the cases I have
> dealt with).  So you can easily can a flood of unknown nmis from an SERR
> and register 0x61 would have the PERR/SERR bits set to 0.  Fun, huh?

if there is nothing in nmi_sc the code flows into another branch. And
it hits the problem of perf events eating all nmi giving no chance the
others. So we take if (!(reason & 0xc0)) case and hit DIE_NMI_IPI
(/me scratching the head why it's not under CONFIG_X86_LOCAL_APIC) and
drop all code, unpleasant.

> 
> Cheers,
> Don
> 
	-- Cyrill

Re: A question of perf NMI handler

[Robert Richter]

(cc'ing Andi)

On 04.08.10 12:39:30, Cyrill Gorcunov wrote:
> On Wed, Aug 04, 2010 at 12:20:26PM -0400, Don Zickus wrote:

> > there.  The problem is the bits in register 0x61 are not always set
> > correctly in the case of SERRs (well at least in all the cases I have
> > dealt with).  So you can easily can a flood of unknown nmis from an SERR
> > and register 0x61 would have the PERR/SERR bits set to 0.  Fun, huh?
> 
> if there is nothing in nmi_sc the code flows into another branch. And
> it hits the problem of perf events eating all nmi giving no chance the
> others. So we take if (!(reason & 0xc0)) case and hit DIE_NMI_IPI
> (/me scratching the head why it's not under CONFIG_X86_LOCAL_APIC) and
> drop all code, unpleasant.

Only the upper 2 bits in io_61h indicate the nmi reason, so in case of
(!(reason & 0xc0)) the source simply can not be determined and all nmi
handlers in the chain must be called (DIE_NMI/DIE_NMI_IPI). The
perfctr handler then stops it.

So you can decide to either get an unrecovered nmi panic triggered by
a perfctr or losing unknown nmis from other sources. Maybe this can be
fixed by implementing handlers for those sources.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: A question of perf NMI handler

[Andi Kleen]

> Only the upper 2 bits in io_61h indicate the nmi reason, so in case of
> (!(reason & 0xc0)) the source simply can not be determined and all nmi
> handlers in the chain must be called (DIE_NMI/DIE_NMI_IPI). The
> perfctr handler then stops it.
> 
> So you can decide to either get an unrecovered nmi panic triggered by
> a perfctr or losing unknown nmis from other sources. Maybe this can be
> fixed by implementing handlers for those sources.

This is a tricky area. Me and Ying have been looking at this recently.

Hardware traditionally signals NMI when it has a uncontained error and really 
expects the OS to shut down to prevent data corruption spreading. i

Unfortunately especially for some older hardware
there can be cases where this is not expressed in port 61.
But the default behaviour of Linux for this today is quite wrong.

Some cases can be also determined with the help of APEI, which
can give you more information about the error (and tell you
if shutdown is needed).

But of course we can still have performance counter and other NMI
users.

So the right flow might be something like

- check software events (like crash dump or reboot)
- check perfctrs
- check APEI
- check port 61 for known events (it's probably a good idea
to check perfctrs first because accessing io ports is quite slow.
But the perfctr handler has to make sure it doesn't eat unknown
events, otherwise error handling would be impacted)
- check other event sources
- shutdown (depending on the chipset likely)

This means the NMI users who cannot determine themselves if a event
happened and eat everything (like oprofile today) would need to be fixed.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: A question of perf NMI handler

[Cyrill Gorcunov]

On Wed, Aug 04, 2010 at 08:48:06PM +0200, Robert Richter wrote:
> (cc'ing Andi)
> 
> On 04.08.10 12:39:30, Cyrill Gorcunov wrote:
> > On Wed, Aug 04, 2010 at 12:20:26PM -0400, Don Zickus wrote:
> 
> > > there.  The problem is the bits in register 0x61 are not always set
> > > correctly in the case of SERRs (well at least in all the cases I have
> > > dealt with).  So you can easily can a flood of unknown nmis from an SERR
> > > and register 0x61 would have the PERR/SERR bits set to 0.  Fun, huh?
> > 
> > if there is nothing in nmi_sc the code flows into another branch. And
> > it hits the problem of perf events eating all nmi giving no chance the
> > others. So we take if (!(reason & 0xc0)) case and hit DIE_NMI_IPI
> > (/me scratching the head why it's not under CONFIG_X86_LOCAL_APIC) and
> > drop all code, unpleasant.
> 
> Only the upper 2 bits in io_61h indicate the nmi reason, so in case of
> (!(reason & 0xc0)) the source simply can not be determined and all nmi
> handlers in the chain must be called (DIE_NMI/DIE_NMI_IPI). The
> perfctr handler then stops it.

yes, that is what I meant by nmi_sc register. I think we need to restucturize
current default_do_nmi handler but how to be with perfs I don't know at moment
if perf register gets overflowed (ie already has pedning nmi) but we handle
it in early nmi cycle this would lead to strange results. Need to think.

> 
> So you can decide to either get an unrecovered nmi panic triggered by
> a perfctr or losing unknown nmis from other sources. Maybe this can be
> fixed by implementing handlers for those sources.
> 
> -Robert
> 
> -- 
> Advanced Micro Devices, Inc.
> Operating System Research Center
> 
	-- Cyrill

Re: A question of perf NMI handler

[Robert Richter]

On 04.08.10 15:26:34, Cyrill Gorcunov wrote:

> yes, that is what I meant by nmi_sc register. I think we need to restucturize
> current default_do_nmi handler but how to be with perfs I don't know at moment
> if perf register gets overflowed (ie already has pedning nmi) but we handle
> it in early nmi cycle this would lead to strange results. Need to think.
> 
> > 
> > So you can decide to either get an unrecovered nmi panic triggered by
> > a perfctr or losing unknown nmis from other sources. Maybe this can be
> > fixed by implementing handlers for those sources.

I was playing around with it yesterday trying to fix this. My idea is
to skip an unkown nmi if the privious nmi was a *handled* perfctr
nmi. I will probably post an rfc patch early next week.

Another problem I encountered is that unknown nmis from the chipset
are not reenabled, thus when hitting the nmi button I only see one
unknown nmi message per boot, if I reenable it, I get an nmi
storm firing nmi_watchdog. Uhh....

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: A question of perf NMI handler

[Don Zickus]

On Fri, Aug 06, 2010 at 08:52:03AM +0200, Robert Richter wrote:
> On 04.08.10 15:26:34, Cyrill Gorcunov wrote:
> 
> > yes, that is what I meant by nmi_sc register. I think we need to restucturize
> > current default_do_nmi handler but how to be with perfs I don't know at moment
> > if perf register gets overflowed (ie already has pedning nmi) but we handle
> > it in early nmi cycle this would lead to strange results. Need to think.
> > 
> > > 
> > > So you can decide to either get an unrecovered nmi panic triggered by
> > > a perfctr or losing unknown nmis from other sources. Maybe this can be
> > > fixed by implementing handlers for those sources.
> 
> I was playing around with it yesterday trying to fix this. My idea is
> to skip an unkown nmi if the privious nmi was a *handled* perfctr

You might want to add a little more logic that says *handled* _and_ had
more than one perfctr trigger.  Most of the time only one perfctr is
probably triggering, so you might be eating unknown_nmi's needlessly.

Just a thought.

> nmi. I will probably post an rfc patch early next week.
> 
> Another problem I encountered is that unknown nmis from the chipset
> are not reenabled, thus when hitting the nmi button I only see one
> unknown nmi message per boot, if I reenable it, I get an nmi
> storm firing nmi_watchdog. Uhh....

Interesting.

Cheers,
Don

[PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 06.08.10 10:21:31, Don Zickus wrote:
> On Fri, Aug 06, 2010 at 08:52:03AM +0200, Robert Richter wrote:

> > I was playing around with it yesterday trying to fix this. My idea is
> > to skip an unkown nmi if the privious nmi was a *handled* perfctr
> 
> You might want to add a little more logic that says *handled* _and_ had
> more than one perfctr trigger.  Most of the time only one perfctr is
> probably triggering, so you might be eating unknown_nmi's needlessly.
> 
> Just a thought.

Yes, that's true. It could be implemented on top of the patch below.

> 
> > nmi. I will probably post an rfc patch early next week.

Here it comes:

>From d2739578199d881ae6a9537c1b96a0efd1cdea43 Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@amd.com>
Date: Thu, 5 Aug 2010 16:19:59 +0200
Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

When perfctrs are running it is valid to have unhandled nmis, two
events could trigger 'simultaneously' raising two back-to-back
NMIs. If the first NMI handles both, the latter will be empty and daze
the CPU.

The solution to avoid an 'unknown nmi' massage in this case was simply
to stop the nmi handler chain when perfctrs are runnning by stating
the nmi was handled. This has the drawback that a) we can not detect
unknown nmis anymore, and b) subsequent nmi handlers are not called.

This patch addresses this. Now, we drop this unknown NMI only if the
previous NMI was handling a perfctr. Otherwise we pass it and let the
kernel handle the unknown nmi. The check runs only if no nmi handler
could handle the nmi (DIE_NMIUNKNOWN case).

We could improve this further by checking if perf was handling more
than one counter. Otherwise we may pass the unknown nmi too.

Signed-off-by: Robert Richter <robert.richter@amd.com>
---
 arch/x86/kernel/cpu/perf_event.c |   39 +++++++++++++++++++++++++++++--------
 1 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index f2da20f..c3cd159 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1200,12 +1200,16 @@ void perf_events_lapic_init(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 }
 
+static DEFINE_PER_CPU(unsigned int, perfctr_handled);
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
 			 unsigned long cmd, void *__args)
 {
 	struct die_args *args = __args;
 	struct pt_regs *regs;
+	unsigned int this_nmi;
+	unsigned int prev_nmi;
 
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
@@ -1214,7 +1218,26 @@ perf_event_nmi_handler(struct notifier_block *self,
 	case DIE_NMI:
 	case DIE_NMI_IPI:
 		break;
-
+	case DIE_NMIUNKNOWN:
+		/*
+		 * This one could be our NMI, two events could trigger
+		 * 'simultaneously' raising two back-to-back NMIs. If
+		 * the first NMI handles both, the latter will be
+		 * empty and daze the CPU.
+		 *
+		 * So, we drop this unknown NMI if the previous NMI
+		 * was handling a perfctr. Otherwise we pass it and
+		 * let the kernel handle the unknown nmi.
+		 *
+		 * Note: this could be improved if we drop unknown
+		 * NMIs only if we handled more than one perfctr in
+		 * the previous NMI.
+		 */
+		this_nmi = percpu_read(irq_stat.__nmi_count);
+		prev_nmi = __get_cpu_var(perfctr_handled);
+		if (this_nmi == prev_nmi + 1)
+			return NOTIFY_STOP;
+		return NOTIFY_DONE;
 	default:
 		return NOTIFY_DONE;
 	}
@@ -1222,14 +1245,12 @@ perf_event_nmi_handler(struct notifier_block *self,
 	regs = args->regs;
 
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
-	/*
-	 * Can't rely on the handled return value to say it was our NMI, two
-	 * events could trigger 'simultaneously' raising two back-to-back NMIs.
-	 *
-	 * If the first NMI handles both, the latter will be empty and daze
-	 * the CPU.
-	 */
-	x86_pmu.handle_irq(regs);
+
+	if (!x86_pmu.handle_irq(regs))
+		return NOTIFY_DONE;
+
+	/* handled */
+	__get_cpu_var(perfctr_handled) = percpu_read(irq_stat.__nmi_count);
 
 	return NOTIFY_STOP;
 }
-- 
1.7.1.1

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Mon, Aug 09, 2010 at 09:48:29PM +0200, Robert Richter wrote:
> On 06.08.10 10:21:31, Don Zickus wrote:
> > On Fri, Aug 06, 2010 at 08:52:03AM +0200, Robert Richter wrote:
> 
> > > I was playing around with it yesterday trying to fix this. My idea is
> > > to skip an unkown nmi if the privious nmi was a *handled* perfctr
> > 
> > You might want to add a little more logic that says *handled* _and_ had
> > more than one perfctr trigger.  Most of the time only one perfctr is
> > probably triggering, so you might be eating unknown_nmi's needlessly.
> > 
> > Just a thought.
> 
> Yes, that's true. It could be implemented on top of the patch below.
> 
> > 
> > > nmi. I will probably post an rfc patch early next week.
> 
> Here it comes:
> 

Thanks Robert! Looks good to me, one nit below.

> From d2739578199d881ae6a9537c1b96a0efd1cdea43 Mon Sep 17 00:00:00 2001
> From: Robert Richter <robert.richter@amd.com>
> Date: Thu, 5 Aug 2010 16:19:59 +0200
> Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs
>
... 
> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
> index f2da20f..c3cd159 100644
> --- a/arch/x86/kernel/cpu/perf_event.c
> +++ b/arch/x86/kernel/cpu/perf_event.c
> @@ -1200,12 +1200,16 @@ void perf_events_lapic_init(void)
>  	apic_write(APIC_LVTPC, APIC_DM_NMI);
>  }
>  
> +static DEFINE_PER_CPU(unsigned int, perfctr_handled);
> +
>  static int __kprobes
>  perf_event_nmi_handler(struct notifier_block *self,
>  			 unsigned long cmd, void *__args)
>  {
>  	struct die_args *args = __args;
>  	struct pt_regs *regs;
> +	unsigned int this_nmi;
> +	unsigned int prev_nmi;
>  
>  	if (!atomic_read(&active_events))
>  		return NOTIFY_DONE;
> @@ -1214,7 +1218,26 @@ perf_event_nmi_handler(struct notifier_block *self,
>  	case DIE_NMI:
>  	case DIE_NMI_IPI:
>  		break;
> -
> +	case DIE_NMIUNKNOWN:
> +		/*
> +		 * This one could be our NMI, two events could trigger
> +		 * 'simultaneously' raising two back-to-back NMIs. If
> +		 * the first NMI handles both, the latter will be
> +		 * empty and daze the CPU.
> +		 *
> +		 * So, we drop this unknown NMI if the previous NMI
> +		 * was handling a perfctr. Otherwise we pass it and
> +		 * let the kernel handle the unknown nmi.
> +		 *
> +		 * Note: this could be improved if we drop unknown
> +		 * NMIs only if we handled more than one perfctr in
> +		 * the previous NMI.
> +		 */
> +		this_nmi = percpu_read(irq_stat.__nmi_count);
> +		prev_nmi = __get_cpu_var(perfctr_handled);
> +		if (this_nmi == prev_nmi + 1)
> +			return NOTIFY_STOP;
> +		return NOTIFY_DONE;
>  	default:
>  		return NOTIFY_DONE;
>  	}
> @@ -1222,14 +1245,12 @@ perf_event_nmi_handler(struct notifier_block *self,
>  	regs = args->regs;
>  
>  	apic_write(APIC_LVTPC, APIC_DM_NMI);

If only I'm not missing something this apic_write should go up to
"case DIE_NMIUNKNOWN" site, no?

	-- Cyrill

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 09.08.10 16:02:45, Cyrill Gorcunov wrote:

> > @@ -1222,14 +1245,12 @@ perf_event_nmi_handler(struct notifier_block *self,
> >  	regs = args->regs;
> >  
> >  	apic_write(APIC_LVTPC, APIC_DM_NMI);
> 
> If only I'm not missing something this apic_write should go up to
> "case DIE_NMIUNKNOWN" site, no?

This seems to be code from the non-nmi implementation and can be
removed at all, which should be a separate patch. The vector is
already set up.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 10, 2010 at 09:42:00AM +0200, Robert Richter wrote:
> On 09.08.10 16:02:45, Cyrill Gorcunov wrote:
> 
> > > @@ -1222,14 +1245,12 @@ perf_event_nmi_handler(struct notifier_block *self,
> > >  	regs = args->regs;
> > >  
> > >  	apic_write(APIC_LVTPC, APIC_DM_NMI);
> > 
> > If only I'm not missing something this apic_write should go up to
> > "case DIE_NMIUNKNOWN" site, no?
> 
> This seems to be code from the non-nmi implementation and can be
> removed at all, which should be a separate patch. The vector is
> already set up.
> 
> -Robert
>

No, this is just a short way to unmask LVTPC (which is required for
cpus). Actually lookig into this snippet I found that in p4 pmu
I've made one redundant unmaksing operation. will update as only
this area settle down.

	-- Cyrill

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 10.08.10 12:16:27, Cyrill Gorcunov wrote:
> On Tue, Aug 10, 2010 at 09:42:00AM +0200, Robert Richter wrote:
> > On 09.08.10 16:02:45, Cyrill Gorcunov wrote:
> > 
> > > > @@ -1222,14 +1245,12 @@ perf_event_nmi_handler(struct notifier_block *self,
> > > >  	regs = args->regs;
> > > >  
> > > >  	apic_write(APIC_LVTPC, APIC_DM_NMI);
> > > 
> > > If only I'm not missing something this apic_write should go up to
> > > "case DIE_NMIUNKNOWN" site, no?
> > 
> > This seems to be code from the non-nmi implementation and can be
> > removed at all, which should be a separate patch. The vector is
> > already set up.
> > 
> > -Robert
> >
> 
> No, this is just a short way to unmask LVTPC (which is required for
> cpus). Actually lookig into this snippet I found that in p4 pmu
> I've made one redundant unmaksing operation. will update as only
> this area settle down.

The vector is setup in hw_perf_enable() and then never masked. The
perfctrs nmi is alwayes enabled since then. I still see no reason for
unmasking it again with every nmi. Once you handle the nmi it is also
enabled.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 10, 2010 at 06:41:24PM +0200, Robert Richter wrote:
> On 10.08.10 12:16:27, Cyrill Gorcunov wrote:
> > On Tue, Aug 10, 2010 at 09:42:00AM +0200, Robert Richter wrote:
> > > On 09.08.10 16:02:45, Cyrill Gorcunov wrote:
> > > 
> > > > > @@ -1222,14 +1245,12 @@ perf_event_nmi_handler(struct notifier_block *self,
> > > > >  	regs = args->regs;
> > > > >  
> > > > >  	apic_write(APIC_LVTPC, APIC_DM_NMI);
> > > > 
> > > > If only I'm not missing something this apic_write should go up to
> > > > "case DIE_NMIUNKNOWN" site, no?
> > > 
> > > This seems to be code from the non-nmi implementation and can be
> > > removed at all, which should be a separate patch. The vector is
> > > already set up.
> > > 
> > > -Robert
> > >
> > 
> > No, this is just a short way to unmask LVTPC (which is required for
> > cpus). Actually lookig into this snippet I found that in p4 pmu
> > I've made one redundant unmaksing operation. will update as only
> > this area settle down.
> 
> The vector is setup in hw_perf_enable() and then never masked. The
> perfctrs nmi is alwayes enabled since then. I still see no reason for
> unmasking it again with every nmi. Once you handle the nmi it is also
> enabled.
> 

It gets masked on NMI arrival, at least for some models (Core Duo, P4,
P6 M and I suspect more theh that, that was the reason oprofile has
it, also there is a note in SDM V3a page 643).

> -Robert
> 
> -- 
> Advanced Micro Devices, Inc.
> Operating System Research Center
> 
	-- Cyrill

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 10.08.10 13:24:51, Cyrill Gorcunov wrote:

> It gets masked on NMI arrival, at least for some models (Core Duo, P4,
> P6 M and I suspect more theh that, that was the reason oprofile has
> it, also there is a note in SDM V3a page 643).

Yes, that's right, I never noticed that. Maybe it is better to
implement the apic write it in model specific code then.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 10, 2010 at 09:05:41PM +0200, Robert Richter wrote:
> On 10.08.10 13:24:51, Cyrill Gorcunov wrote:
> 
> > It gets masked on NMI arrival, at least for some models (Core Duo, P4,
> > P6 M and I suspect more theh that, that was the reason oprofile has
> > it, also there is a note in SDM V3a page 643).
> 
> Yes, that's right, I never noticed that. Maybe it is better to
> implement the apic write it in model specific code then.
> 

Perhaps we can make it simplier I think, ie like it was before -- we just
move it under your new DIE_NMIUNKNOWN, in separate patch of course. Though
I'm fine with either way.

(actually it's interesting to know wouldn't we leave lvt masked when
 we hit 'second delayed nmi has arrived' situation, I guess we didn't
 hit it before in real yet :-)

> -Robert
> 
> -- 
> Advanced Micro Devices, Inc.
> Operating System Research Center
> 
	-- Cyrill

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 10.08.10 15:24:28, Cyrill Gorcunov wrote:
> On Tue, Aug 10, 2010 at 09:05:41PM +0200, Robert Richter wrote:
> > On 10.08.10 13:24:51, Cyrill Gorcunov wrote:
> > 
> > > It gets masked on NMI arrival, at least for some models (Core Duo, P4,
> > > P6 M and I suspect more theh that, that was the reason oprofile has
> > > it, also there is a note in SDM V3a page 643).
> > 
> > Yes, that's right, I never noticed that. Maybe it is better to
> > implement the apic write it in model specific code then.
> > 
> 
> Perhaps we can make it simplier I think, ie like it was before -- we just
> move it under your new DIE_NMIUNKNOWN, in separate patch of course. Though
> I'm fine with either way.

I do not understand why you want to put this in the 'unknown'
path. Isn't it necessary to unmask the vector with every call of the
nmi handler?

-Robert

> 
> (actually it's interesting to know wouldn't we leave lvt masked when
>  we hit 'second delayed nmi has arrived' situation, I guess we didn't
>  hit it before in real yet :-)
> 
> > -Robert
> > 
> > -- 
> > Advanced Micro Devices, Inc.
> > Operating System Research Center
> > 
> 	-- Cyrill
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Thu, Aug 12, 2010 at 03:24:19PM +0200, Robert Richter wrote:
> On 10.08.10 15:24:28, Cyrill Gorcunov wrote:
> > On Tue, Aug 10, 2010 at 09:05:41PM +0200, Robert Richter wrote:
> > > On 10.08.10 13:24:51, Cyrill Gorcunov wrote:
> > > 
> > > > It gets masked on NMI arrival, at least for some models (Core Duo, P4,
> > > > P6 M and I suspect more theh that, that was the reason oprofile has
> > > > it, also there is a note in SDM V3a page 643).
> > > 
> > > Yes, that's right, I never noticed that. Maybe it is better to
> > > implement the apic write it in model specific code then.
> > > 
> > 
> > Perhaps we can make it simplier I think, ie like it was before -- we just
> > move it under your new DIE_NMIUNKNOWN, in separate patch of course. Though
> > I'm fine with either way.
> 
> I do not understand why you want to put this in the 'unknown'
> path. Isn't it necessary to unmask the vector with every call of the
> nmi handler?
> 
> -Robert
> 

Heh, it's simple - I'm screwed. Robert you're right, of course it should NOT
be under every 'unknown' nmi. I thought about small optimization here, but
I think all this should be done only _after_ your patch is merged.

Sorry for confuse ;)
 
	-- Cyrill

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Mon, Aug 09, 2010 at 09:48:29PM +0200, Robert Richter wrote:
> On 06.08.10 10:21:31, Don Zickus wrote:
> > On Fri, Aug 06, 2010 at 08:52:03AM +0200, Robert Richter wrote:
> 
> > > I was playing around with it yesterday trying to fix this. My idea is
> > > to skip an unkown nmi if the privious nmi was a *handled* perfctr
> > 
> > You might want to add a little more logic that says *handled* _and_ had
> > more than one perfctr trigger.  Most of the time only one perfctr is
> > probably triggering, so you might be eating unknown_nmi's needlessly.
> > 
> > Just a thought.
> 
> Yes, that's true. It could be implemented on top of the patch below.

I did, but the changes basically revert the bulk of your patch.

> 
> > 
> > > nmi. I will probably post an rfc patch early next week.
> 
> Here it comes:
> 
> From d2739578199d881ae6a9537c1b96a0efd1cdea43 Mon Sep 17 00:00:00 2001
> From: Robert Richter <robert.richter@amd.com>
> Date: Thu, 5 Aug 2010 16:19:59 +0200
> Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

On top of Robert's patch:
(compiled tested only because I don't have a fancy button to trigger
unknown nmis)

>From 548cf5148f47618854a0eff22b1d55db71b6f8fc Mon Sep 17 00:00:00 2001
From: Don Zickus <dzickus@redhat.com>
Date: Tue, 10 Aug 2010 16:40:03 -0400
Subject: [PATCH] perf, x86: only skip NMIs when multiple perfctrs trigger

A small optimization on top of Robert's patch that limits the
skipping of NMI's to cases where we detect multiple perfctr events
have happened.

Signed-off-by: Don Zickus <dzickus@redhat.com>

---
 arch/x86/kernel/cpu/perf_event.c |   34 ++++++++++++++++++++--------------
 1 files changed, 20 insertions(+), 14 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index c3cd159..066046d 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1154,7 +1154,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		/*
 		 * event overflow
 		 */
-		handled		= 1;
+		handled		+= 1;
 		data.period	= event->hw.last_period;
 
 		if (!x86_perf_event_set_period(event))
@@ -1200,7 +1200,7 @@ void perf_events_lapic_init(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 }
 
-static DEFINE_PER_CPU(unsigned int, perfctr_handled);
+static DEFINE_PER_CPU(unsigned int, perfctr_skip);
 
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
@@ -1208,8 +1208,7 @@ perf_event_nmi_handler(struct notifier_block *self,
 {
 	struct die_args *args = __args;
 	struct pt_regs *regs;
-	unsigned int this_nmi;
-	unsigned int prev_nmi;
+	int handled = 0;
 
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
@@ -1229,14 +1228,11 @@ perf_event_nmi_handler(struct notifier_block *self,
 		 * was handling a perfctr. Otherwise we pass it and
 		 * let the kernel handle the unknown nmi.
 		 *
-		 * Note: this could be improved if we drop unknown
-		 * NMIs only if we handled more than one perfctr in
-		 * the previous NMI.
 		 */
-		this_nmi = percpu_read(irq_stat.__nmi_count);
-		prev_nmi = __get_cpu_var(perfctr_handled);
-		if (this_nmi == prev_nmi + 1)
+		if (__get_cpu_var(perfctr_skip)){
+			__get_cpu_var(perfctr_skip) -=1;
 			return NOTIFY_STOP;
+		}
 		return NOTIFY_DONE;
 	default:
 		return NOTIFY_DONE;
@@ -1246,11 +1242,21 @@ perf_event_nmi_handler(struct notifier_block *self,
 
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 
-	if (!x86_pmu.handle_irq(regs))
+	handled = x86_pmu.handle_irq(regs);
+	if (!handled)
+		/* not our NMI */
 		return NOTIFY_DONE;
-
-	/* handled */
-	__get_cpu_var(perfctr_handled) = percpu_read(irq_stat.__nmi_count);
+	else if (handled > 1)
+		/*
+		 * More than one perfctr triggered.  This could have
+		 * caused a second NMI that we must now skip because
+		 * we have already handled it.  Remember it.
+		 *
+		 * NOTE: We have no way of knowing if a second NMI was
+		 * actually triggered, so we may accidentally skip a valid
+		 * unknown nmi later.
+		 */
+		__get_cpu_var(perfctr_skip) +=1;
 
 	return NOTIFY_STOP;
 }
-- 
1.7.2

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Frederic Weisbecker]

On Tue, Aug 10, 2010 at 04:48:56PM -0400, Don Zickus wrote:
> On Mon, Aug 09, 2010 at 09:48:29PM +0200, Robert Richter wrote:
> > On 06.08.10 10:21:31, Don Zickus wrote:
> > > On Fri, Aug 06, 2010 at 08:52:03AM +0200, Robert Richter wrote:
> > 
> > > > I was playing around with it yesterday trying to fix this. My idea is
> > > > to skip an unkown nmi if the privious nmi was a *handled* perfctr
> > > 
> > > You might want to add a little more logic that says *handled* _and_ had
> > > more than one perfctr trigger.  Most of the time only one perfctr is
> > > probably triggering, so you might be eating unknown_nmi's needlessly.
> > > 
> > > Just a thought.
> > 
> > Yes, that's true. It could be implemented on top of the patch below.
> 
> I did, but the changes basically revert the bulk of your patch.
> 
> > 
> > > 
> > > > nmi. I will probably post an rfc patch early next week.
> > 
> > Here it comes:
> > 
> > From d2739578199d881ae6a9537c1b96a0efd1cdea43 Mon Sep 17 00:00:00 2001
> > From: Robert Richter <robert.richter@amd.com>
> > Date: Thu, 5 Aug 2010 16:19:59 +0200
> > Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs
> 
> On top of Robert's patch:
> (compiled tested only because I don't have a fancy button to trigger
> unknown nmis)
> 
> From 548cf5148f47618854a0eff22b1d55db71b6f8fc Mon Sep 17 00:00:00 2001
> From: Don Zickus <dzickus@redhat.com>
> Date: Tue, 10 Aug 2010 16:40:03 -0400
> Subject: [PATCH] perf, x86: only skip NMIs when multiple perfctrs trigger
> 
> A small optimization on top of Robert's patch that limits the
> skipping of NMI's to cases where we detect multiple perfctr events
> have happened.



Yeah, I think that's more reasonable. This lowers even more the chances of
losing important hardware errors.

One comment though:


> 
> Signed-off-by: Don Zickus <dzickus@redhat.com>
> 
> ---
>  arch/x86/kernel/cpu/perf_event.c |   34 ++++++++++++++++++++--------------
>  1 files changed, 20 insertions(+), 14 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
> index c3cd159..066046d 100644
> --- a/arch/x86/kernel/cpu/perf_event.c
> +++ b/arch/x86/kernel/cpu/perf_event.c
> @@ -1154,7 +1154,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>  		/*
>  		 * event overflow
>  		 */
> -		handled		= 1;
> +		handled		+= 1;
>  		data.period	= event->hw.last_period;
>  
>  		if (!x86_perf_event_set_period(event))
> @@ -1200,7 +1200,7 @@ void perf_events_lapic_init(void)
>  	apic_write(APIC_LVTPC, APIC_DM_NMI);
>  }
>  
> -static DEFINE_PER_CPU(unsigned int, perfctr_handled);
> +static DEFINE_PER_CPU(unsigned int, perfctr_skip);
>  
>  static int __kprobes
>  perf_event_nmi_handler(struct notifier_block *self,
> @@ -1208,8 +1208,7 @@ perf_event_nmi_handler(struct notifier_block *self,
>  {
>  	struct die_args *args = __args;
>  	struct pt_regs *regs;
> -	unsigned int this_nmi;
> -	unsigned int prev_nmi;
> +	int handled = 0;
>  
>  	if (!atomic_read(&active_events))
>  		return NOTIFY_DONE;
> @@ -1229,14 +1228,11 @@ perf_event_nmi_handler(struct notifier_block *self,
>  		 * was handling a perfctr. Otherwise we pass it and
>  		 * let the kernel handle the unknown nmi.
>  		 *
> -		 * Note: this could be improved if we drop unknown
> -		 * NMIs only if we handled more than one perfctr in
> -		 * the previous NMI.
>  		 */
> -		this_nmi = percpu_read(irq_stat.__nmi_count);
> -		prev_nmi = __get_cpu_var(perfctr_handled);
> -		if (this_nmi == prev_nmi + 1)
> +		if (__get_cpu_var(perfctr_skip)){
> +			__get_cpu_var(perfctr_skip) -=1;
>  			return NOTIFY_STOP;
> +		}
>  		return NOTIFY_DONE;
>  	default:
>  		return NOTIFY_DONE;
> @@ -1246,11 +1242,21 @@ perf_event_nmi_handler(struct notifier_block *self,
>  
>  	apic_write(APIC_LVTPC, APIC_DM_NMI);
>  
> -	if (!x86_pmu.handle_irq(regs))
> +	handled = x86_pmu.handle_irq(regs);
> +	if (!handled)
> +		/* not our NMI */
>  		return NOTIFY_DONE;
> -
> -	/* handled */
> -	__get_cpu_var(perfctr_handled) = percpu_read(irq_stat.__nmi_count);
> +	else if (handled > 1)
> +		/*
> +		 * More than one perfctr triggered.  This could have
> +		 * caused a second NMI that we must now skip because
> +		 * we have already handled it.  Remember it.
> +		 *
> +		 * NOTE: We have no way of knowing if a second NMI was
> +		 * actually triggered, so we may accidentally skip a valid
> +		 * unknown nmi later.
> +		 */
> +		__get_cpu_var(perfctr_skip) +=1;



May be make it just a pending bit. I mean not something that can
go further 1, because you can't have more than 1 pending anyway. I don't
know how that could happen you get accidental perctr_skip > 1, may be
expected pending NMIs that don't happen somehow, but better be paranoid with
that, as it's about trying not to miss hardware errors.

Thanks.

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 10.08.10 22:44:55, Frederic Weisbecker wrote:
> On Tue, Aug 10, 2010 at 04:48:56PM -0400, Don Zickus wrote:
> > @@ -1200,7 +1200,7 @@ void perf_events_lapic_init(void)
> >  	apic_write(APIC_LVTPC, APIC_DM_NMI);
> >  }
> >  
> > -static DEFINE_PER_CPU(unsigned int, perfctr_handled);
> > +static DEFINE_PER_CPU(unsigned int, perfctr_skip);

Yes, using perfctr_skip is better to understand ...

> > @@ -1229,14 +1228,11 @@ perf_event_nmi_handler(struct notifier_block *self,
> >  		 * was handling a perfctr. Otherwise we pass it and
> >  		 * let the kernel handle the unknown nmi.
> >  		 *
> > -		 * Note: this could be improved if we drop unknown
> > -		 * NMIs only if we handled more than one perfctr in
> > -		 * the previous NMI.
> >  		 */
> > -		this_nmi = percpu_read(irq_stat.__nmi_count);
> > -		prev_nmi = __get_cpu_var(perfctr_handled);
> > -		if (this_nmi == prev_nmi + 1)
> > +		if (__get_cpu_var(perfctr_skip)){
> > +			__get_cpu_var(perfctr_skip) -=1;
> >  			return NOTIFY_STOP;
> > +		}
> >  		return NOTIFY_DONE;
> >  	default:
> >  		return NOTIFY_DONE;
> > @@ -1246,11 +1242,21 @@ perf_event_nmi_handler(struct notifier_block *self,
> >  
> >  	apic_write(APIC_LVTPC, APIC_DM_NMI);
> >  
> > -	if (!x86_pmu.handle_irq(regs))
> > +	handled = x86_pmu.handle_irq(regs);
> > +	if (!handled)
> > +		/* not our NMI */
> >  		return NOTIFY_DONE;
> > -
> > -	/* handled */
> > -	__get_cpu_var(perfctr_handled) = percpu_read(irq_stat.__nmi_count);
> > +	else if (handled > 1)
> > +		/*
> > +		 * More than one perfctr triggered.  This could have
> > +		 * caused a second NMI that we must now skip because
> > +		 * we have already handled it.  Remember it.
> > +		 *
> > +		 * NOTE: We have no way of knowing if a second NMI was
> > +		 * actually triggered, so we may accidentally skip a valid
> > +		 * unknown nmi later.
> > +		 */
> > +		__get_cpu_var(perfctr_skip) +=1;

... but this will not work. You have to mark the *absolute* nmi number
here. If you only raise a flag, the next unknown nmi will be dropped,
every. Because, in between there could have been other nmis that
stopped the chain and thus the 'unknown' path is not executed. The
trick in my patch is that you *know*, which nmi you want to skip.

I will send an updated version of my patch.

-Robert

> 
> 
> 
> May be make it just a pending bit. I mean not something that can
> go further 1, because you can't have more than 1 pending anyway. I don't
> know how that could happen you get accidental perctr_skip > 1, may be
> expected pending NMIs that don't happen somehow, but better be paranoid with
> that, as it's about trying not to miss hardware errors.
> 
> Thanks.
> 
> 

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Wed, Aug 11, 2010 at 01:10:46PM +0200, Robert Richter wrote:
> > > +		 *
> > > +		 * NOTE: We have no way of knowing if a second NMI was
> > > +		 * actually triggered, so we may accidentally skip a valid
> > > +		 * unknown nmi later.
> > > +		 */
> > > +		__get_cpu_var(perfctr_skip) +=1;
> 
> ... but this will not work. You have to mark the *absolute* nmi number
> here. If you only raise a flag, the next unknown nmi will be dropped,
> every. Because, in between there could have been other nmis that
> stopped the chain and thus the 'unknown' path is not executed. The
> trick in my patch is that you *know*, which nmi you want to skip.

I guess I am confused.  The way I read your patch was that you assumed the
next NMI would be the one you skip and if there was another NMI in between
the handled one and the one to skip, you would not skip it (nmi count !=
prev + 1) and it would produce an accidental unknown nmi.

I tried to change that with my patch by setting the skip flag which would
be drained on the next unknown nmi, independent of where it is in the NMI
backlog of NMIs.

Did I misread something?

Cheers,
Don

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 11.08.10 08:44:43, Don Zickus wrote:
> On Wed, Aug 11, 2010 at 01:10:46PM +0200, Robert Richter wrote:
> > > > +		 *
> > > > +		 * NOTE: We have no way of knowing if a second NMI was
> > > > +		 * actually triggered, so we may accidentally skip a valid
> > > > +		 * unknown nmi later.
> > > > +		 */
> > > > +		__get_cpu_var(perfctr_skip) +=1;
> > 
> > ... but this will not work. You have to mark the *absolute* nmi number
> > here. If you only raise a flag, the next unknown nmi will be dropped,
> > every. Because, in between there could have been other nmis that
> > stopped the chain and thus the 'unknown' path is not executed. The
> > trick in my patch is that you *know*, which nmi you want to skip.
> 
> I guess I am confused.  The way I read your patch was that you assumed the
> next NMI would be the one you skip and if there was another NMI in between
> the handled one and the one to skip, you would not skip it (nmi count !=
> prev + 1) and it would produce an accidental unknown nmi.

That's how it works.

> I tried to change that with my patch by setting the skip flag which would
> be drained on the next unknown nmi, independent of where it is in the NMI
> backlog of NMIs.

"setting the skip flag which would be drained on the next unknown nmi"

That's what is wrong, it drops every unknown nmi, no matter when it is
detected. In between there could be 1000's of valid other nmis
handled. You even could have been returned from nmi mode. But still,
the next unknown nmi will be dropped. Your patch could accumulate also
the number of unknown nmis to skip, and then, if 'real' unknown nmis
happen, all of them will be dropped.

-Robert

> 
> Did I misread something?
> 
> Cheers,
> Don
> 

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Wed, Aug 11, 2010 at 04:03:07PM +0200, Robert Richter wrote:
> On 11.08.10 08:44:43, Don Zickus wrote:
> > On Wed, Aug 11, 2010 at 01:10:46PM +0200, Robert Richter wrote:
> > > > > +		 *
> > > > > +		 * NOTE: We have no way of knowing if a second NMI was
> > > > > +		 * actually triggered, so we may accidentally skip a valid
> > > > > +		 * unknown nmi later.
> > > > > +		 */
> > > > > +		__get_cpu_var(perfctr_skip) +=1;
> > > 
> > > ... but this will not work. You have to mark the *absolute* nmi number
> > > here. If you only raise a flag, the next unknown nmi will be dropped,
> > > every. Because, in between there could have been other nmis that
> > > stopped the chain and thus the 'unknown' path is not executed. The
> > > trick in my patch is that you *know*, which nmi you want to skip.
> > 
> > I guess I am confused.  The way I read your patch was that you assumed the
> > next NMI would be the one you skip and if there was another NMI in between
> > the handled one and the one to skip, you would not skip it (nmi count !=
> > prev + 1) and it would produce an accidental unknown nmi.
> 
> That's how it works.
> 
> > I tried to change that with my patch by setting the skip flag which would
> > be drained on the next unknown nmi, independent of where it is in the NMI
> > backlog of NMIs.
> 
> "setting the skip flag which would be drained on the next unknown nmi"
> 
> That's what is wrong, it drops every unknown nmi, no matter when it is

Well as Frederic pointed out the skip variable will never go past one, so
it will only drop at most one unknown nmi.

> detected. In between there could be 1000's of valid other nmis
> handled. You even could have been returned from nmi mode. But still,
> the next unknown nmi will be dropped. Your patch could accumulate also

That was the intent.  Can we guarantee that in the rare cases where the
perfctr is generating two nmis, that they will be back-to-back?

I think Huang tried to cap my approach even further my creating a time
window in which the two nmis had to happen.  That gives us the flexibility
to handle nmis that are not back to back, but yet deal with the case where
two perfctrs fired but we are unsure if it generated a second nmi and we
falsely set the skip flag.

Cheers,
Don

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Frederic Weisbecker]

On Wed, Aug 11, 2010 at 01:10:46PM +0200, Robert Richter wrote:
> On 10.08.10 22:44:55, Frederic Weisbecker wrote:
> > On Tue, Aug 10, 2010 at 04:48:56PM -0400, Don Zickus wrote:
> > > @@ -1200,7 +1200,7 @@ void perf_events_lapic_init(void)
> > >  	apic_write(APIC_LVTPC, APIC_DM_NMI);
> > >  }
> > >  
> > > -static DEFINE_PER_CPU(unsigned int, perfctr_handled);
> > > +static DEFINE_PER_CPU(unsigned int, perfctr_skip);
> 
> Yes, using perfctr_skip is better to understand ...
> 
> > > @@ -1229,14 +1228,11 @@ perf_event_nmi_handler(struct notifier_block *self,
> > >  		 * was handling a perfctr. Otherwise we pass it and
> > >  		 * let the kernel handle the unknown nmi.
> > >  		 *
> > > -		 * Note: this could be improved if we drop unknown
> > > -		 * NMIs only if we handled more than one perfctr in
> > > -		 * the previous NMI.
> > >  		 */
> > > -		this_nmi = percpu_read(irq_stat.__nmi_count);
> > > -		prev_nmi = __get_cpu_var(perfctr_handled);
> > > -		if (this_nmi == prev_nmi + 1)
> > > +		if (__get_cpu_var(perfctr_skip)){
> > > +			__get_cpu_var(perfctr_skip) -=1;
> > >  			return NOTIFY_STOP;
> > > +		}
> > >  		return NOTIFY_DONE;
> > >  	default:
> > >  		return NOTIFY_DONE;
> > > @@ -1246,11 +1242,21 @@ perf_event_nmi_handler(struct notifier_block *self,
> > >  
> > >  	apic_write(APIC_LVTPC, APIC_DM_NMI);
> > >  
> > > -	if (!x86_pmu.handle_irq(regs))
> > > +	handled = x86_pmu.handle_irq(regs);
> > > +	if (!handled)
> > > +		/* not our NMI */
> > >  		return NOTIFY_DONE;
> > > -
> > > -	/* handled */
> > > -	__get_cpu_var(perfctr_handled) = percpu_read(irq_stat.__nmi_count);
> > > +	else if (handled > 1)
> > > +		/*
> > > +		 * More than one perfctr triggered.  This could have
> > > +		 * caused a second NMI that we must now skip because
> > > +		 * we have already handled it.  Remember it.
> > > +		 *
> > > +		 * NOTE: We have no way of knowing if a second NMI was
> > > +		 * actually triggered, so we may accidentally skip a valid
> > > +		 * unknown nmi later.
> > > +		 */
> > > +		__get_cpu_var(perfctr_skip) +=1;
> 
> ... but this will not work. You have to mark the *absolute* nmi number
> here. If you only raise a flag, the next unknown nmi will be dropped,
> every.



Isn't it what we want? Only the next unknown nmi gets dropped.




> Because, in between there could have been other nmis that
> stopped the chain and thus the 'unknown' path is not executed.



I'm not sure what you mean here. Are you thinking about a third
NMI source that triggers while we are still handling the first
NMI in the back to back sequence?



> The trick in my patch is that you *know*, which nmi you want to skip.


Well with the flag you also know which nmi you want to skip.

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 13.08.10 00:37:48, Frederic Weisbecker wrote:

> > 
> > ... but this will not work. You have to mark the *absolute* nmi number
> > here. If you only raise a flag, the next unknown nmi will be dropped,
> > every.
> 
> 
> 
> Isn't it what we want? Only the next unknown nmi gets dropped.
> 
> 
> 
> 
> > Because, in between there could have been other nmis that
> > stopped the chain and thus the 'unknown' path is not executed.
> 
> 
> 
> I'm not sure what you mean here. Are you thinking about a third
> NMI source that triggers while we are still handling the first
> NMI in the back to back sequence?
> 
> 
> 
> > The trick in my patch is that you *know*, which nmi you want to skip.
> 
> 
> Well with the flag you also know which nmi you want to skip.

We cannot assume that all cpus have the same behavior here. Imagine a
cpu that handles 2 counters and *does not* trigger a back-to-back
nmi. With flags only implemented, the next unknown nmi will be dropped
anyway, no matter when it fires. We have to check the nmi sequence.

The next thing you have to be aware is, a registered nmi handler is
not called with every nmi. If there was another nmi source and a
handler with higher priority was returning a stop, when all other
subsequent handlers are not called. Esp. 'unknown nmi' is called only
in rare cases. So, a handler might not get noticed of an nmi. This
means, if a handler gets called a 2nd time, it must not necessarily
the next (2nd) nmi.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Frederic Weisbecker]

On Fri, Aug 13, 2010 at 10:22:30AM +0200, Robert Richter wrote:
> On 13.08.10 00:37:48, Frederic Weisbecker wrote:
> 
> > > 
> > > ... but this will not work. You have to mark the *absolute* nmi number
> > > here. If you only raise a flag, the next unknown nmi will be dropped,
> > > every.
> > 
> > 
> > 
> > Isn't it what we want? Only the next unknown nmi gets dropped.
> > 
> > 
> > 
> > 
> > > Because, in between there could have been other nmis that
> > > stopped the chain and thus the 'unknown' path is not executed.
> > 
> > 
> > 
> > I'm not sure what you mean here. Are you thinking about a third
> > NMI source that triggers while we are still handling the first
> > NMI in the back to back sequence?
> > 
> > 
> > 
> > > The trick in my patch is that you *know*, which nmi you want to skip.
> > 
> > 
> > Well with the flag you also know which nmi you want to skip.
> 
> We cannot assume that all cpus have the same behavior here. Imagine a
> cpu that handles 2 counters and *does not* trigger a back-to-back
> nmi. With flags only implemented, the next unknown nmi will be dropped
> anyway, no matter when it fires. We have to check the nmi sequence.



I'd expect it to be an ABI. NMIs can't nest, but if one triggers while
servicing another, it should trigger right after (once we "iret", which
reenables NMIs).

But I haven't checked intel or amd documentation about that.


 
> The next thing you have to be aware is, a registered nmi handler is
> not called with every nmi. If there was another nmi source and a
> handler with higher priority was returning a stop, when all other
> subsequent handlers are not called. Esp. 'unknown nmi' is called only
> in rare cases. So, a handler might not get noticed of an nmi. This
> means, if a handler gets called a 2nd time, it must not necessarily
> the next (2nd) nmi.


Yeah, in this case we can just clear the __ger_cpu_var(next_nmi_skip)
when another handler service the next NMI.

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 13.08.10 21:28:07, Frederic Weisbecker wrote:

> > We cannot assume that all cpus have the same behavior here. Imagine a
> > cpu that handles 2 counters and *does not* trigger a back-to-back
> > nmi. With flags only implemented, the next unknown nmi will be dropped
> > anyway, no matter when it fires. We have to check the nmi sequence.
> 
> 
> 
> I'd expect it to be an ABI. NMIs can't nest, but if one triggers while
> servicing another, it should trigger right after (once we "iret", which
> reenables NMIs).
> 
> But I haven't checked intel or amd documentation about that.

Yes, nmis are nested and if there is another source firing it will be
retriggered. But the question is, if multiple counters trigger, does
this mean multiple nmis are fired, esp. if all counters were served in
the first run? This very much depends on the cpu implementation and it
will be hard to find documentation about this detail.

> > The next thing you have to be aware is, a registered nmi handler is
> > not called with every nmi. If there was another nmi source and a
> > handler with higher priority was returning a stop, when all other
> > subsequent handlers are not called. Esp. 'unknown nmi' is called only
> > in rare cases. So, a handler might not get noticed of an nmi. This
> > means, if a handler gets called a 2nd time, it must not necessarily
> > the next (2nd) nmi.
> 
> 
> Yeah, in this case we can just clear the __ger_cpu_var(next_nmi_skip)
> when another handler service the next NMI.

Yes, this might work too. But then you end up at the same complexity.
Even worse, you have to check and unset the flag with each perf nmi.
If you store the nmi number, it will only be read in then 'unknown'
nmi path again. And, you can't unset the flag for nmis by other
sources what do not pass the perf nmi handler.

So, overall, I don't see advantages in using a flag. The
implementation of storing the nmi number is simple and straight
forward with no or less impact on performance or memory usage.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Wed, Aug 11, 2010 at 04:44:55AM +0200, Frederic Weisbecker wrote:
> May be make it just a pending bit. I mean not something that can
> go further 1, because you can't have more than 1 pending anyway. I don't
> know how that could happen you get accidental perctr_skip > 1, may be
> expected pending NMIs that don't happen somehow, but better be paranoid with
> that, as it's about trying not to miss hardware errors.

I guess I was thinking about the SMI case where it drains the perfctr(s)
and then retriggers them but I guess even in that case the most you can
have is one extra NMI.  So yeah, you are probably right, I should have
used a flag instead of incrementing.

Cheers,
Don

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Huang Ying]

On Wed, 2010-08-11 at 04:48 +0800, Don Zickus wrote:
> > From d2739578199d881ae6a9537c1b96a0efd1cdea43 Mon Sep 17 00:00:00 2001
> > From: Robert Richter <robert.richter@amd.com>
> > Date: Thu, 5 Aug 2010 16:19:59 +0200
> > Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs
> 
> On top of Robert's patch:
> (compiled tested only because I don't have a fancy button to trigger
> unknown nmis)

You can trigger unknown NMIs via apic->send_IPI_mask(cpu_mask,
NMI_VECTOR).

How about the algorithm as follow:

int perf_event_nmi_handler()
{
	...
	switch (cmd) {
	case DIE_NMIUNKNOWN:
		if (per_cpu(perfctr_prev_handled) > 1
		    && rdtsc() - per_cpu(perfctr_handled_timestamp) < 1000)
			return NOTIFY_STOP;
		else
			return NOTIFY_DONE;
	}
	...
	handled = x86_pmu.handle_irq(regs);
	per_cpu(perfctr_prev_handled) = per_cpu(perfctr_handled);
	per_cpu(perfctr_handled) = handled;
	if (handled) {
		per_cpu(perfctr_handled_timestamp) = rdtsc();
		return NOTIFY_STOP;
	} else
		return NOTIFY_DONE;
}

Best Regards,
Huang Ying

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Wed, Aug 11, 2010 at 11:19:39AM +0800, Huang Ying wrote:
> On Wed, 2010-08-11 at 04:48 +0800, Don Zickus wrote:
> > > From d2739578199d881ae6a9537c1b96a0efd1cdea43 Mon Sep 17 00:00:00 2001
> > > From: Robert Richter <robert.richter@amd.com>
> > > Date: Thu, 5 Aug 2010 16:19:59 +0200
> > > Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs
> > 
> > On top of Robert's patch:
> > (compiled tested only because I don't have a fancy button to trigger
> > unknown nmis)
> 
> You can trigger unknown NMIs via apic->send_IPI_mask(cpu_mask,
> NMI_VECTOR).
> 
> How about the algorithm as follow:

Heh, I thought about the following too, just couldn't figure out an easy
way to timestamp.  I forgot about the rdtsc(). :-)

The only thing that might screw this up would be an SMI which takes longer
than 1000 but that should be rare and would probably be related to the
unknown NMI anyway.  Also under virt that would probably break (due to
time jumping) but until they emulate the perfctr, it won't matter. :-)

Cheers,
Don

Re: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

[Peter Zijlstra]

On Wed, 2010-08-11 at 08:36 -0400, Don Zickus wrote:
> The only thing that might screw this up would be an SMI which takes longer
> than 1000 but that should be rare and would probably be related to the
> unknown NMI anyway. 

Long running SMIs aren't nearly as rare as you'd want them to be.
Hitting one in exactly the right spot will be, but given the numbers its
going to happen and make us scratch our heads..

[PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

I was debuging this a little more, see version 2 below.

-Robert

--

>From 8bb831af56d118b85fc38e0ddc2e516f7504b9fb Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@amd.com>
Date: Thu, 5 Aug 2010 16:19:59 +0200
Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

When perfctrs are running it is valid to have unhandled nmis, two
events could trigger 'simultaneously' raising two back-to-back
NMIs. If the first NMI handles both, the latter will be empty and daze
the CPU.

The solution to avoid an 'unknown nmi' massage in this case was simply
to stop the nmi handler chain when perfctrs are runnning by stating
the nmi was handled. This has the drawback that a) we can not detect
unknown nmis anymore, and b) subsequent nmi handlers are not called.

This patch addresses this. Now, we check this unknown NMI if it could
be a perfctr back-to-back NMI. Otherwise we pass it and let the kernel
handle the unknown nmi.

This is a debug log:

cpu #6, nmi #32333, skip_nmi #32330, handled = 1, time = 1934364430
cpu #6, nmi #32334, skip_nmi #32330, handled = 1, time = 1934704616
cpu #6, nmi #32335, skip_nmi #32336, handled = 2, time = 1936032320
cpu #6, nmi #32336, skip_nmi #32336, handled = 0, time = 1936034139
cpu #6, nmi #32337, skip_nmi #32336, handled = 1, time = 1936120100
cpu #6, nmi #32338, skip_nmi #32336, handled = 1, time = 1936404607
cpu #6, nmi #32339, skip_nmi #32336, handled = 1, time = 1937983416
cpu #6, nmi #32340, skip_nmi #32341, handled = 2, time = 1938201032
cpu #6, nmi #32341, skip_nmi #32341, handled = 0, time = 1938202830
cpu #6, nmi #32342, skip_nmi #32341, handled = 1, time = 1938443743
cpu #6, nmi #32343, skip_nmi #32341, handled = 1, time = 1939956552
cpu #6, nmi #32344, skip_nmi #32341, handled = 1, time = 1940073224
cpu #6, nmi #32345, skip_nmi #32341, handled = 1, time = 1940485677
cpu #6, nmi #32346, skip_nmi #32347, handled = 2, time = 1941947772
cpu #6, nmi #32347, skip_nmi #32347, handled = 1, time = 1941949818
cpu #6, nmi #32348, skip_nmi #32347, handled = 0, time = 1941951591
Uhhuh. NMI received for unknown reason 00 on CPU 6.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue

Deltas:

nmi #32334 340186
nmi #32335 1327704
nmi #32336 1819      <<<< back-to-back nmi [1]
nmi #32337 85961
nmi #32338 284507
nmi #32339 1578809
nmi #32340 217616
nmi #32341 1798      <<<< back-to-back nmi [2]
nmi #32342 240913
nmi #32343 1512809
nmi #32344 116672
nmi #32345 412453
nmi #32346 1462095   <<<< 1st nmi (standard) handling 2 counters
nmi #32347 2046      <<<< 2nd nmi (back-to-back) handling one counter
nmi #32348 1773      <<<< 3rd nmi (back-to-back) handling no counter! [3]

For  back-to-back nmi detection there are the following rules:

The perfctr nmi handler was handling more than one counter and no
counter was handled in the subsequent nmi (see [1] and [2] above).

There is another case if there are two subsequent back-to-back nmis
[3]. In this case we measure the time between the first and the
2nd. The 2nd is detected as back-to-back because the first handled
more than one counter. The time between the 1st and the 2nd is used to
calculate a range for which we assume a back-to-back nmi. Now, the 3rd
nmi triggers, we measure again the time delta and compare it with the
first delta from which we know it was a back-to-back nmi. If the 3rd
nmi is within the range, it is also a back-to-back nmi and we drop it.

Signed-off-by: Robert Richter <robert.richter@amd.com>
---
 arch/x86/kernel/cpu/perf_event.c |   96 +++++++++++++++++++++++++++++++++----
 1 files changed, 85 insertions(+), 11 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index f2da20f..b79a235 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1154,7 +1154,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		/*
 		 * event overflow
 		 */
-		handled		= 1;
+		handled++;
 		data.period	= event->hw.last_period;
 
 		if (!x86_perf_event_set_period(event))
@@ -1200,12 +1200,24 @@ void perf_events_lapic_init(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 }
 
+struct perfctr_nmi {
+	u64		timestamp;
+	u64		range;
+	unsigned int	marked;
+	int		handled;
+};
+
+static DEFINE_PER_CPU(struct perfctr_nmi, nmi);
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
 			 unsigned long cmd, void *__args)
 {
 	struct die_args *args = __args;
-	struct pt_regs *regs;
+	unsigned int this_nmi;
+	int handled;
+	u64 timestamp;
+	u64 delta;
 
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
@@ -1214,22 +1226,84 @@ perf_event_nmi_handler(struct notifier_block *self,
 	case DIE_NMI:
 	case DIE_NMI_IPI:
 		break;
+	case DIE_NMIUNKNOWN:
+		this_nmi = percpu_read(irq_stat.__nmi_count);
+		if (this_nmi != __get_cpu_var(nmi).marked)
+			return NOTIFY_DONE;
+
+		/*
+		 * This one could be our NMI, two events could trigger
+		 * 'simultaneously' raising two back-to-back NMIs. If
+		 * the first NMI handles both, the latter will be
+		 * empty and daze the CPU.
+		 *
+		 * So, we check this unknown NMI and drop it if it is
+		 * a perfctr back-to-back nmi. Otherwise we pass it
+		 * and let the kernel handle the unknown nmi.
+		 */
+
+		if (!cpu_has_tsc)
+			/*
+			 * no timestamps available the cannot detect
+			 * back-to-back nmis, drop it
+			 */
+			return NOTIFY_STOP;
+
+		if (__get_cpu_var(nmi).handled > 1)
+			/*
+			 *  we have handled more than one counter,
+			 *  this must be a back-to-back nmi
+			 */
+			return NOTIFY_STOP;
+
+		rdtscll(delta);
+		delta -= __get_cpu_var(nmi).timestamp;
+		if (delta < __get_cpu_var(nmi).range)
+			/* the delta is small, it is a back-to-back nmi */
+			return NOTIFY_STOP;
+
+		/* not a perfctr back-to-back nmi, let it pass */
+		return NOTIFY_DONE;
 
 	default:
 		return NOTIFY_DONE;
 	}
 
-	regs = args->regs;
+	this_nmi = percpu_read(irq_stat.__nmi_count);
 
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
-	/*
-	 * Can't rely on the handled return value to say it was our NMI, two
-	 * events could trigger 'simultaneously' raising two back-to-back NMIs.
-	 *
-	 * If the first NMI handles both, the latter will be empty and daze
-	 * the CPU.
-	 */
-	x86_pmu.handle_irq(regs);
+
+	handled = x86_pmu.handle_irq(args->regs);
+
+	if (!handled)
+		return NOTIFY_DONE;
+
+	if ((handled == 1) && (__get_cpu_var(nmi).marked != this_nmi))
+		/* this may not trigger back-to-back nmis */
+		return NOTIFY_STOP;
+
+	if (!cpu_has_tsc)
+		goto mark_next_nmi;
+
+	rdtscll(timestamp);
+	if (__get_cpu_var(nmi).marked == this_nmi) {
+		/*
+		 * this was a back-to-back nmi, calculate back-to-back
+		 * time delta and define the back-to-back range that
+		 * is twice the delta
+		 */
+		delta = timestamp;
+		delta -= __get_cpu_var(nmi).timestamp;
+
+		__get_cpu_var(nmi).range = delta << 1;
+	}
+
+	__get_cpu_var(nmi).timestamp	= timestamp;
+	__get_cpu_var(nmi).handled	= handled;
+
+ mark_next_nmi:
+	/* the next nmi could be a back-to-back nmi */
+	__get_cpu_var(nmi).marked	= this_nmi + 1;
 
 	return NOTIFY_STOP;
 }
-- 
1.7.1.1


-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 12.08.10 00:00:58, Robert Richter wrote:
> I was debuging this a little more, see version 2 below.

I was testing the patch further, it properly filters perfctr
back-to-back nmis. I was able to reliable detect unknown nmis
triggered by the nmi button during high load perf sessions with
multiple counters, no false positives.

> 
> -Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Thu, Aug 12, 2010 at 03:10:07PM +0200, Robert Richter wrote:
> On 12.08.10 00:00:58, Robert Richter wrote:
> > I was debuging this a little more, see version 2 below.
> 
> I was testing the patch further, it properly filters perfctr
> back-to-back nmis. I was able to reliable detect unknown nmis
> triggered by the nmi button during high load perf sessions with
> multiple counters, no false positives.

For my own curiousity, what type of high load perf sessions are you using
to test this.  I don't know perf well enough to have it generate events
across multiple perfctrs.

Thanks,
Don

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 12.08.10 14:21:26, Don Zickus wrote:

> > I was testing the patch further, it properly filters perfctr
> > back-to-back nmis. I was able to reliable detect unknown nmis
> > triggered by the nmi button during high load perf sessions with
> > multiple counters, no false positives.
> 
> For my own curiousity, what type of high load perf sessions are you using
> to test this.  I don't know perf well enough to have it generate events
> across multiple perfctrs.

You put load on all cpus and then start something like the following:

 perf record -e cycles -e instructions -e cache-references \
 	-e cache-misses -e branch-misses -a -- sleep 10

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Thu, Aug 12, 2010 at 12:00:58AM +0200, Robert Richter wrote:
> I was debuging this a little more, see version 2 below.
> 
> -Robert
> 
> --
> 
> From 8bb831af56d118b85fc38e0ddc2e516f7504b9fb Mon Sep 17 00:00:00 2001
> From: Robert Richter <robert.richter@amd.com>
> Date: Thu, 5 Aug 2010 16:19:59 +0200
> Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs
> 
> When perfctrs are running it is valid to have unhandled nmis, two
> events could trigger 'simultaneously' raising two back-to-back
> NMIs. If the first NMI handles both, the latter will be empty and daze
> the CPU.
> 
> The solution to avoid an 'unknown nmi' massage in this case was simply
> to stop the nmi handler chain when perfctrs are runnning by stating
> the nmi was handled. This has the drawback that a) we can not detect
> unknown nmis anymore, and b) subsequent nmi handlers are not called.
> 
> This patch addresses this. Now, we check this unknown NMI if it could
> be a perfctr back-to-back NMI. Otherwise we pass it and let the kernel
> handle the unknown nmi.

Seems to cover my concerns.  Great work!

ACK

Cheers,
Don

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Frederic Weisbecker]

On Thu, Aug 12, 2010 at 12:00:58AM +0200, Robert Richter wrote:
> I was debuging this a little more, see version 2 below.
> 
> -Robert
> 
> --
> 
> From 8bb831af56d118b85fc38e0ddc2e516f7504b9fb Mon Sep 17 00:00:00 2001
> From: Robert Richter <robert.richter@amd.com>
> Date: Thu, 5 Aug 2010 16:19:59 +0200
> Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs
> 
> When perfctrs are running it is valid to have unhandled nmis, two
> events could trigger 'simultaneously' raising two back-to-back
> NMIs. If the first NMI handles both, the latter will be empty and daze
> the CPU.
> 
> The solution to avoid an 'unknown nmi' massage in this case was simply
> to stop the nmi handler chain when perfctrs are runnning by stating
> the nmi was handled. This has the drawback that a) we can not detect
> unknown nmis anymore, and b) subsequent nmi handlers are not called.
> 
> This patch addresses this. Now, we check this unknown NMI if it could
> be a perfctr back-to-back NMI. Otherwise we pass it and let the kernel
> handle the unknown nmi.
> 
> This is a debug log:
> 
> cpu #6, nmi #32333, skip_nmi #32330, handled = 1, time = 1934364430
> cpu #6, nmi #32334, skip_nmi #32330, handled = 1, time = 1934704616
> cpu #6, nmi #32335, skip_nmi #32336, handled = 2, time = 1936032320
> cpu #6, nmi #32336, skip_nmi #32336, handled = 0, time = 1936034139
> cpu #6, nmi #32337, skip_nmi #32336, handled = 1, time = 1936120100
> cpu #6, nmi #32338, skip_nmi #32336, handled = 1, time = 1936404607
> cpu #6, nmi #32339, skip_nmi #32336, handled = 1, time = 1937983416
> cpu #6, nmi #32340, skip_nmi #32341, handled = 2, time = 1938201032
> cpu #6, nmi #32341, skip_nmi #32341, handled = 0, time = 1938202830
> cpu #6, nmi #32342, skip_nmi #32341, handled = 1, time = 1938443743
> cpu #6, nmi #32343, skip_nmi #32341, handled = 1, time = 1939956552
> cpu #6, nmi #32344, skip_nmi #32341, handled = 1, time = 1940073224
> cpu #6, nmi #32345, skip_nmi #32341, handled = 1, time = 1940485677
> cpu #6, nmi #32346, skip_nmi #32347, handled = 2, time = 1941947772
> cpu #6, nmi #32347, skip_nmi #32347, handled = 1, time = 1941949818
> cpu #6, nmi #32348, skip_nmi #32347, handled = 0, time = 1941951591
> Uhhuh. NMI received for unknown reason 00 on CPU 6.
> Do you have a strange power saving mode enabled?
> Dazed and confused, but trying to continue
> 
> Deltas:
> 
> nmi #32334 340186
> nmi #32335 1327704
> nmi #32336 1819      <<<< back-to-back nmi [1]
> nmi #32337 85961
> nmi #32338 284507
> nmi #32339 1578809
> nmi #32340 217616
> nmi #32341 1798      <<<< back-to-back nmi [2]
> nmi #32342 240913
> nmi #32343 1512809
> nmi #32344 116672
> nmi #32345 412453
> nmi #32346 1462095   <<<< 1st nmi (standard) handling 2 counters
> nmi #32347 2046      <<<< 2nd nmi (back-to-back) handling one counter
> nmi #32348 1773      <<<< 3rd nmi (back-to-back) handling no counter! [3]
> 
> For  back-to-back nmi detection there are the following rules:
> 
> The perfctr nmi handler was handling more than one counter and no
> counter was handled in the subsequent nmi (see [1] and [2] above).
> 
> There is another case if there are two subsequent back-to-back nmis
> [3]. In this case we measure the time between the first and the
> 2nd. The 2nd is detected as back-to-back because the first handled
> more than one counter. The time between the 1st and the 2nd is used to
> calculate a range for which we assume a back-to-back nmi. Now, the 3rd
> nmi triggers, we measure again the time delta and compare it with the
> first delta from which we know it was a back-to-back nmi. If the 3rd
> nmi is within the range, it is also a back-to-back nmi and we drop it.
> 
> Signed-off-by: Robert Richter <robert.richter@amd.com>
> ---



That time based thing looks a bit complicated.

I'm still not sure why you don't want to use a simple flag:

After handled a perf NMI:

	if (handled more than one counter)
		__get_cpu_var(skip_unknown) = 1;

While handling an unknown NMI:

	if (__get_cpu_var(skip_unknown)) {
		__get_cpu_var(skip_unknow) = 0;
		return NOTIFY_STOP;
	}

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Peter Zijlstra]

On Thu, 2010-08-12 at 00:00 +0200, Robert Richter wrote:
> From 8bb831af56d118b85fc38e0ddc2e516f7504b9fb Mon Sep 17 00:00:00 2001
> From: Robert Richter <robert.richter@amd.com>
> Date: Thu, 5 Aug 2010 16:19:59 +0200
> Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs
> 
> When perfctrs are running it is valid to have unhandled nmis, two
> events could trigger 'simultaneously' raising two back-to-back
> NMIs. If the first NMI handles both, the latter will be empty and daze
> the CPU.
> 
> The solution to avoid an 'unknown nmi' massage in this case was simply
> to stop the nmi handler chain when perfctrs are runnning by stating
> the nmi was handled. This has the drawback that a) we can not detect
> unknown nmis anymore, and b) subsequent nmi handlers are not called.
> 
> This patch addresses this. Now, we check this unknown NMI if it could
> be a perfctr back-to-back NMI. Otherwise we pass it and let the kernel
> handle the unknown nmi.
> 
> This is a debug log:

> 
> Deltas:
> 

> nmi #32346 1462095   <<<< 1st nmi (standard) handling 2 counters
> nmi #32347 2046      <<<< 2nd nmi (back-to-back) handling one counter
> nmi #32348 1773      <<<< 3rd nmi (back-to-back) handling no counter! [3]
> 
> For  back-to-back nmi detection there are the following rules:
> 
> The perfctr nmi handler was handling more than one counter and no
> counter was handled in the subsequent nmi (see [1] and [2] above).
> 
> There is another case if there are two subsequent back-to-back nmis
> [3]. In this case we measure the time between the first and the
> 2nd. The 2nd is detected as back-to-back because the first handled
> more than one counter. The time between the 1st and the 2nd is used to
> calculate a range for which we assume a back-to-back nmi. Now, the 3rd
> nmi triggers, we measure again the time delta and compare it with the
> first delta from which we know it was a back-to-back nmi. If the 3rd
> nmi is within the range, it is also a back-to-back nmi and we drop it. 

I liked the one without funny timestamps in better, the whole timestamps
thing just feels too fragile.

Relying on handled > 1 to arm the back-to-back filter seems doable.

(Also, you didn't deal with the TSC going backwards..)

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Mon, Aug 16, 2010 at 04:48:36PM +0200, Peter Zijlstra wrote:
...
> > There is another case if there are two subsequent back-to-back nmis
> > [3]. In this case we measure the time between the first and the
> > 2nd. The 2nd is detected as back-to-back because the first handled
> > more than one counter. The time between the 1st and the 2nd is used to
> > calculate a range for which we assume a back-to-back nmi. Now, the 3rd
> > nmi triggers, we measure again the time delta and compare it with the
> > first delta from which we know it was a back-to-back nmi. If the 3rd
> > nmi is within the range, it is also a back-to-back nmi and we drop it. 
> 
> I liked the one without funny timestamps in better, the whole timestamps
> thing just feels too fragile.
> 

Me too, the former Roberts patch (if I'm not missing something) looks good
to me.

>
> Relying on handled > 1 to arm the back-to-back filter seems doable.
> 

It's doable _but_ I think there is nothing we can do, there is no
way (at least I known of) to check if there is latched nmi from
perf counters. We only can assume that if there multiple counters
overflowed most probably the next unknown nmi has the same nature,
ie it came from perf. Yes, we can loose real unknown nmi in this
case but I think this is justified trade off. If an user need
a precise counting of unknown nmis he should not arm perf events
at all, if there an user with nmi button (guys where did you get this
magic buttuns? i need one ;) he better to not arm perf events too
otherwise he might have to click twice

(and of course we should keep in mind Andi's proposal but it
 is a next step I think).

> (Also, you didn't deal with the TSC going backwards..)
> 
	-- Cyrill

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 16.08.10 12:27:06, Cyrill Gorcunov wrote:
> On Mon, Aug 16, 2010 at 04:48:36PM +0200, Peter Zijlstra wrote:
> > I liked the one without funny timestamps in better, the whole timestamps
> > thing just feels too fragile.
> > 
> 
> Me too, the former Roberts patch (if I'm not missing something) looks good
> to me.
> 
> >
> > Relying on handled > 1 to arm the back-to-back filter seems doable.

Peter, I will rip out the timestamp code from the -v2 patch. My first
patch does not deal with a 2-1-0 sequence, so it has false positives.
We do not necessarily need the timestamps if back-to-back nmis are
rare. Without using timestamps the statistically lost ratio for
unknown nmis will be as the ratio for back-to-back nmis, with
timestamps we could catch almost every unknown nmi. So if we encounter
problems we could still implement timestamp code on top.

> It's doable _but_ I think there is nothing we can do, there is no
> way (at least I known of) to check if there is latched nmi from
> perf counters. We only can assume that if there multiple counters
> overflowed most probably the next unknown nmi has the same nature,
> ie it came from perf.

As said, I think with timestamps we could be able to detect 100% of
the unknown nmis. I guess we get now more than 90% with mutliple
counters, and 100% with a single counter running. So, this is already
more than a simple improvement.

> Yes, we can loose real unknown nmi in this
> case but I think this is justified trade off. If an user need
> a precise counting of unknown nmis he should not arm perf events
> at all, if there an user with nmi button (guys where did you get this
> magic buttuns? i need one ;) he better to not arm perf events too
> otherwise he might have to click twice
> 
> (and of course we should keep in mind Andi's proposal but it
>  is a next step I think).

Yes, this patch is the first step, now we can change the nmi handler
priority. The perf handler must not have the lowest priority anymore.

> > (Also, you didn't deal with the TSC going backwards..)

Does this also happen in the case of a back-to-back nmi? I don't know
the conditions for a backward running TSC. Maybe, if an nmi is
retriggered the TSC wont be adjusted by a negative offset, I don't
know...

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Mon, Aug 16, 2010 at 07:16:10PM +0200, Robert Richter wrote:
> On 16.08.10 12:27:06, Cyrill Gorcunov wrote:
> > On Mon, Aug 16, 2010 at 04:48:36PM +0200, Peter Zijlstra wrote:
> > > I liked the one without funny timestamps in better, the whole timestamps
> > > thing just feels too fragile.
> > > 
> > 
> > Me too, the former Roberts patch (if I'm not missing something) looks good
> > to me.
> > 
> > >
> > > Relying on handled > 1 to arm the back-to-back filter seems doable.
> 

I suspect Peter was supposed to be in To: field ;)

> Peter, I will rip out the timestamp code from the -v2 patch. My first
> patch does not deal with a 2-1-0 sequence, so it has false positives.
> We do not necessarily need the timestamps if back-to-back nmis are
> rare. Without using timestamps the statistically lost ratio for
> unknown nmis will be as the ratio for back-to-back nmis, with
> timestamps we could catch almost every unknown nmi. So if we encounter
> problems we could still implement timestamp code on top.
> 
> > It's doable _but_ I think there is nothing we can do, there is no
> > way (at least I known of) to check if there is latched nmi from
> > perf counters. We only can assume that if there multiple counters
> > overflowed most probably the next unknown nmi has the same nature,
> > ie it came from perf.
> 
> As said, I think with timestamps we could be able to detect 100% of
> the unknown nmis. I guess we get now more than 90% with mutliple
> counters, and 100% with a single counter running. So, this is already
> more than a simple improvement.

Robert, I think we still may miss unknown irq, consider the case when
unknown nmi is latched while you handle nmi from perf and what is
more interesting several counters may be overflowed. So you set
delta small enough and second (unknown nmi) will be in range and
treated as being perf back-to-back, or I miss something from patch?

> 
> > Yes, we can loose real unknown nmi in this
> > case but I think this is justified trade off. If an user need
> > a precise counting of unknown nmis he should not arm perf events
> > at all, if there an user with nmi button (guys where did you get this
> > magic buttuns? i need one ;) he better to not arm perf events too
> > otherwise he might have to click twice
> > 
> > (and of course we should keep in mind Andi's proposal but it
> >  is a next step I think).
> 
> Yes, this patch is the first step, now we can change the nmi handler
> priority. The perf handler must not have the lowest priority anymore.
> 
> > > (Also, you didn't deal with the TSC going backwards..)
> 
> Does this also happen in the case of a back-to-back nmi? I don't know
> the conditions for a backward running TSC. Maybe, if an nmi is
> retriggered the TSC wont be adjusted by a negative offset, I don't
> know...

I never heard of backward running tsc, though tsc is a strange beast.

> 
> -Robert
> 
> -- 
> Advanced Micro Devices, Inc.
> Operating System Research Center
> 
	-- Cyrill

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Peter Zijlstra]

On Mon, 2010-08-16 at 23:06 +0400, Cyrill Gorcunov wrote:
> I never heard of backward running tsc, though tsc is a strange beast.
> 
Its not supposed to happen, but then there's BIOS failure-add that frobs
the TSC from SMIs and fun TSC artifacts around CPU frequency changes and
people resetting TSC in S-states etc..

In short, never trust the TSC to be even remotely sane.

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Mon, Aug 16, 2010 at 09:13:16PM +0200, Peter Zijlstra wrote:
> On Mon, 2010-08-16 at 23:06 +0400, Cyrill Gorcunov wrote:
> > I never heard of backward running tsc, though tsc is a strange beast.
> > 
> Its not supposed to happen, but then there's BIOS failure-add that frobs
> the TSC from SMIs and fun TSC artifacts around CPU frequency changes and
> people resetting TSC in S-states etc..

grr :/

> 
> In short, never trust the TSC to be even remotely sane.
> 

ok, good to know, thanks!

	-- Cyrill

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 16.08.10 15:06:59, Cyrill Gorcunov wrote:
> I suspect Peter was supposed to be in To: field ;)

It was the attempt to answer 2 mails with one. :)

> Robert, I think we still may miss unknown irq, consider the case when
> unknown nmi is latched while you handle nmi from perf and what is
> more interesting several counters may be overflowed. So you set
> delta small enough and second (unknown nmi) will be in range and
> treated as being perf back-to-back, or I miss something from patch?

Yes, that's true, but before you have to enable your Infinite
Improbability Drive.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v2] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 17, 2010 at 12:55:25AM +0200, Robert Richter wrote:
> On 16.08.10 15:06:59, Cyrill Gorcunov wrote:
> > I suspect Peter was supposed to be in To: field ;)
> 
> It was the attempt to answer 2 mails with one. :)
> 

ah, I see ;)

> > Robert, I think we still may miss unknown irq, consider the case when
> > unknown nmi is latched while you handle nmi from perf and what is
> > more interesting several counters may be overflowed. So you set
> > delta small enough and second (unknown nmi) will be in range and
> > treated as being perf back-to-back, or I miss something from patch?
> 
> Yes, that's true, but before you have to enable your Infinite
> Improbability Drive.
> 

not at all, it's absolutely legit to happen ;)

> -Robert
> 
> -- 
> Advanced Micro Devices, Inc.
> Operating System Research Center
> 
	-- Cyrill

[PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

Peter,

this is version 3 without timestamp code. Compared to -v1 it
implements the handling for 2-1-0 nmi sequences.

-Robert

>From 71a206394d8a3536b033247ec14ad037fef77236 Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@amd.com>
Date: Tue, 17 Aug 2010 16:42:03 +0200
Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

When perfctrs are running it is valid to have unhandled nmis, two
events could trigger 'simultaneously' raising two back-to-back
NMIs. If the first NMI handles both, the latter will be empty and daze
the CPU.

The solution to avoid an 'unknown nmi' massage in this case was simply
to stop the nmi handler chain when perfctrs are runnning by stating
the nmi was handled. This has the drawback that a) we can not detect
unknown nmis anymore, and b) subsequent nmi handlers are not called.

This patch addresses this. Now, we check this unknown NMI if it could
be a perfctr back-to-back NMI. Otherwise we pass it and let the kernel
handle the unknown nmi.

This is a debug log:

cpu #6, nmi #32333, skip_nmi #32330, handled = 1, time = 1934364430
cpu #6, nmi #32334, skip_nmi #32330, handled = 1, time = 1934704616
cpu #6, nmi #32335, skip_nmi #32336, handled = 2, time = 1936032320
cpu #6, nmi #32336, skip_nmi #32336, handled = 0, time = 1936034139
cpu #6, nmi #32337, skip_nmi #32336, handled = 1, time = 1936120100
cpu #6, nmi #32338, skip_nmi #32336, handled = 1, time = 1936404607
cpu #6, nmi #32339, skip_nmi #32336, handled = 1, time = 1937983416
cpu #6, nmi #32340, skip_nmi #32341, handled = 2, time = 1938201032
cpu #6, nmi #32341, skip_nmi #32341, handled = 0, time = 1938202830
cpu #6, nmi #32342, skip_nmi #32341, handled = 1, time = 1938443743
cpu #6, nmi #32343, skip_nmi #32341, handled = 1, time = 1939956552
cpu #6, nmi #32344, skip_nmi #32341, handled = 1, time = 1940073224
cpu #6, nmi #32345, skip_nmi #32341, handled = 1, time = 1940485677
cpu #6, nmi #32346, skip_nmi #32347, handled = 2, time = 1941947772
cpu #6, nmi #32347, skip_nmi #32347, handled = 1, time = 1941949818
cpu #6, nmi #32348, skip_nmi #32347, handled = 0, time = 1941951591
Uhhuh. NMI received for unknown reason 00 on CPU 6.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue

Deltas:

nmi #32334 340186
nmi #32335 1327704
nmi #32336 1819      <<<< back-to-back nmi [1]
nmi #32337 85961
nmi #32338 284507
nmi #32339 1578809
nmi #32340 217616
nmi #32341 1798      <<<< back-to-back nmi [2]
nmi #32342 240913
nmi #32343 1512809
nmi #32344 116672
nmi #32345 412453
nmi #32346 1462095   <<<< 1st nmi (standard) handling 2 counters
nmi #32347 2046      <<<< 2nd nmi (back-to-back) handling one counter
nmi #32348 1773      <<<< 3rd nmi (back-to-back) handling no counter! [3]

For  back-to-back nmi detection there are the following rules:

The perfctr nmi handler was handling more than one counter and no
counter was handled in the subsequent nmi (see [1] and [2] above).

There is another case if there are two subsequent back-to-back nmis
[3]. The 2nd is detected as back-to-back because the first handled
more than one counter. If the second handles one counter and the 3rd
handles nothing, we drop the 3rd nmi because it could be a
back-to-back nmi.

Signed-off-by: Robert Richter <robert.richter@amd.com>
---
 arch/x86/kernel/cpu/perf_event.c |   65 ++++++++++++++++++++++++++++++-------
 1 files changed, 52 insertions(+), 13 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index f2da20f..bbcc89e 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1154,7 +1154,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		/*
 		 * event overflow
 		 */
-		handled		= 1;
+		handled++;
 		data.period	= event->hw.last_period;
 
 		if (!x86_perf_event_set_period(event))
@@ -1200,12 +1200,20 @@ void perf_events_lapic_init(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 }
 
+struct perfctr_nmi {
+	unsigned int	marked;
+	int		handled;
+};
+
+static DEFINE_PER_CPU(struct perfctr_nmi, nmi);
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
 			 unsigned long cmd, void *__args)
 {
 	struct die_args *args = __args;
-	struct pt_regs *regs;
+	unsigned int this_nmi;
+	int handled;
 
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
@@ -1214,22 +1222,53 @@ perf_event_nmi_handler(struct notifier_block *self,
 	case DIE_NMI:
 	case DIE_NMI_IPI:
 		break;
-
+	case DIE_NMIUNKNOWN:
+		this_nmi = percpu_read(irq_stat.__nmi_count);
+		if (this_nmi != __get_cpu_var(nmi).marked)
+			/* let the kernel handle the unknown nmi */
+			return NOTIFY_DONE;
+		/*
+		 * This one is a perfctr back-to-back nmi. Two events
+		 * trigger 'simultaneously' raising two back-to-back
+		 * NMIs. If the first NMI handles both, the latter
+		 * will be empty and daze the CPU. So, we drop it to
+		 * avoid false-positive 'unknown nmi' messages.
+		 */
+		return NOTIFY_STOP;
 	default:
 		return NOTIFY_DONE;
 	}
 
-	regs = args->regs;
-
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
-	/*
-	 * Can't rely on the handled return value to say it was our NMI, two
-	 * events could trigger 'simultaneously' raising two back-to-back NMIs.
-	 *
-	 * If the first NMI handles both, the latter will be empty and daze
-	 * the CPU.
-	 */
-	x86_pmu.handle_irq(regs);
+
+	handled = x86_pmu.handle_irq(args->regs);
+	if (!handled)
+		return NOTIFY_DONE;
+
+	this_nmi = percpu_read(irq_stat.__nmi_count);
+	if (handled > 1)
+		goto mark_nmi;
+	if ((__get_cpu_var(nmi).marked == this_nmi)
+	    && (__get_cpu_var(nmi).handled > 1))
+		/*
+		 * We could have two subsequent back-to-back nmis: The
+		 * first handles more than one counter, the 2nd
+		 * handles only one counter and the 3rd handles no
+		 * counter.
+		 *
+		 * This is the 2nd nmi because the previous was
+		 * handling more than one counter. We will mark the
+		 * next (3rd) and then drop it if unhandled.
+		 */
+		goto mark_nmi;
+
+	/* this may not trigger back-to-back nmis */
+	return NOTIFY_STOP;
+
+ mark_nmi:
+	/* the next nmi could be a back-to-back nmi */
+	__get_cpu_var(nmi).marked	= this_nmi + 1;
+	__get_cpu_var(nmi).handled	= handled;
 
 	return NOTIFY_STOP;
 }
-- 
1.7.1.1

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 17, 2010 at 05:22:25PM +0200, Robert Richter wrote:
> Peter,
> 
> this is version 3 without timestamp code. Compared to -v1 it
> implements the handling for 2-1-0 nmi sequences.
> 
> -Robert
> 
> From 71a206394d8a3536b033247ec14ad037fef77236 Mon Sep 17 00:00:00 2001
> From: Robert Richter <robert.richter@amd.com>
> Date: Tue, 17 Aug 2010 16:42:03 +0200
> Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs
> 
... 
> Signed-off-by: Robert Richter <robert.richter@amd.com>
> ---
...

Looks good to me, thanks a lot Robert!

	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Peter Zijlstra]

On Tue, 2010-08-17 at 17:22 +0200, Robert Richter wrote:
> +       this_nmi = percpu_read(irq_stat.__nmi_count);
> +       if (handled > 1)
> +               goto mark_nmi;
> +       if ((__get_cpu_var(nmi).marked == this_nmi)
> +           && (__get_cpu_var(nmi).handled > 1))
> +               /*
> +                * We could have two subsequent back-to-back nmis: The
> +                * first handles more than one counter, the 2nd
> +                * handles only one counter and the 3rd handles no
> +                * counter.
> +                *
> +                * This is the 2nd nmi because the previous was
> +                * handling more than one counter. We will mark the
> +                * next (3rd) and then drop it if unhandled.
> +                */
> +               goto mark_nmi;
> +
> +       /* this may not trigger back-to-back nmis */
> +       return NOTIFY_STOP;
> +
> + mark_nmi:
> +       /* the next nmi could be a back-to-back nmi */
> +       __get_cpu_var(nmi).marked       = this_nmi + 1;
> +       __get_cpu_var(nmi).handled      = handled;
>  
>         return NOTIFY_STOP;
>  } 

I queued it with that part changed to:

+       this_nmi = percpu_read(irq_stat.__nmi_count);
+       if ((handled > 1) ||
+               /* the next nmi could be a back-to-back nmi */
+           ((__get_cpu_var(nmi).marked == this_nmi) &&
+            (__get_cpu_var(nmi).handled > 1))) {
+               /*
+                * We could have two subsequent back-to-back nmis: The
+                * first handles more than one counter, the 2nd
+                * handles only one counter and the 3rd handles no
+                * counter.
+                *
+                * This is the 2nd nmi because the previous was
+                * handling more than one counter. We will mark the
+                * next (3rd) and then drop it if unhandled.
+                */
+               __get_cpu_var(nmi).marked       = this_nmi + 1;
+               __get_cpu_var(nmi).handled      = handled;
+       }

        return NOTIFY_STOP;
 }

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 19.08.10 06:45:53, Peter Zijlstra wrote:
> I queued it with that part changed to:
> 
> +       this_nmi = percpu_read(irq_stat.__nmi_count);
> +       if ((handled > 1) ||
> +               /* the next nmi could be a back-to-back nmi */
> +           ((__get_cpu_var(nmi).marked == this_nmi) &&
> +            (__get_cpu_var(nmi).handled > 1))) {
> +               /*
> +                * We could have two subsequent back-to-back nmis: The
> +                * first handles more than one counter, the 2nd
> +                * handles only one counter and the 3rd handles no
> +                * counter.
> +                *
> +                * This is the 2nd nmi because the previous was
> +                * handling more than one counter. We will mark the
> +                * next (3rd) and then drop it if unhandled.
> +                */
> +               __get_cpu_var(nmi).marked       = this_nmi + 1;
> +               __get_cpu_var(nmi).handled      = handled;
> +       }
> 
>         return NOTIFY_STOP;
>  }

I am fine with this. Thanks Peter.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Thu, Aug 19, 2010 at 12:45:53PM +0200, Peter Zijlstra wrote:
> 
> I queued it with that part changed to:

I realized the other day this change doesn't cover the nehalem, core and p4
cases which use

intel_pmu_handle_irq
p4_pmu_handle_irq

as their handlers.  Though that patch can go on top of Robert's.

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Peter Zijlstra]

On Thu, 2010-08-19 at 10:12 -0400, Don Zickus wrote:
> On Thu, Aug 19, 2010 at 12:45:53PM +0200, Peter Zijlstra wrote:
> > 
> > I queued it with that part changed to:
> 
> I realized the other day this change doesn't cover the nehalem, core and p4
> cases which use
> 
> intel_pmu_handle_irq
> p4_pmu_handle_irq
> 
> as their handlers.  Though that patch can go on top of Robert's.

Something like this?

---
Index: linux-2.6/arch/x86/kernel/cpu/perf_event_intel.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/cpu/perf_event_intel.c
+++ linux-2.6/arch/x86/kernel/cpu/perf_event_intel.c
@@ -713,6 +713,7 @@ static int intel_pmu_handle_irq(struct p
 	struct cpu_hw_events *cpuc;
 	int bit, loops;
 	u64 ack, status;
+	int handled = 0;
 
 	perf_sample_data_init(&data, 0);
 
@@ -743,12 +744,16 @@ again:
 	/*
 	 * PEBS overflow sets bit 62 in the global status register
 	 */
-	if (__test_and_clear_bit(62, (unsigned long *)&status))
+	if (__test_and_clear_bit(62, (unsigned long *)&status)) {
+		handled++;
 		x86_pmu.drain_pebs(regs);
+	}
 
 	for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
 		struct perf_event *event = cpuc->events[bit];
 
+		handled++;
+
 		if (!test_bit(bit, cpuc->active_mask))
 			continue;
 
@@ -772,7 +777,7 @@ again:
 
 done:
 	intel_pmu_enable_all(0);
-	return 1;
+	return handled;
 }
 
 static struct event_constraint *
Index: linux-2.6/arch/x86/kernel/cpu/perf_event_p4.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/cpu/perf_event_p4.c
+++ linux-2.6/arch/x86/kernel/cpu/perf_event_p4.c
@@ -673,7 +673,7 @@ static int p4_pmu_handle_irq(struct pt_r
 		if (!overflow && (val & (1ULL << (x86_pmu.cntval_bits - 1))))
 			continue;
 
-		handled += overflow;
+		handled += !!overflow;
 
 		/* event overflow for sure */
 		data.period = event->hw.last_period;
@@ -690,7 +690,7 @@ static int p4_pmu_handle_irq(struct pt_r
 		inc_irq_stat(apic_perf_irqs);
 	}
 
-	return handled > 0;
+	return handled;
 }
 
 /*

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Thu, Aug 19, 2010 at 04:27:13PM +0200, Peter Zijlstra wrote:
> On Thu, 2010-08-19 at 10:12 -0400, Don Zickus wrote:
> > On Thu, Aug 19, 2010 at 12:45:53PM +0200, Peter Zijlstra wrote:
> > > 
> > > I queued it with that part changed to:
> > 
> > I realized the other day this change doesn't cover the nehalem, core and p4
> > cases which use
> > 
> > intel_pmu_handle_irq
> > p4_pmu_handle_irq
> > 
> > as their handlers.  Though that patch can go on top of Robert's.
> 
> Something like this?

Looks correct.  I'll try to test the perf_event_intel.c path though I
haven't had much luck getting multiple events on the nehalem box I have
(the amd box was easy).

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Thu, Aug 19, 2010 at 04:27:13PM +0200, Peter Zijlstra wrote:
...
> Index: linux-2.6/arch/x86/kernel/cpu/perf_event_p4.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/cpu/perf_event_p4.c
> +++ linux-2.6/arch/x86/kernel/cpu/perf_event_p4.c
> @@ -673,7 +673,7 @@ static int p4_pmu_handle_irq(struct pt_r
>  		if (!overflow && (val & (1ULL << (x86_pmu.cntval_bits - 1))))
>  			continue;
>  
> -		handled += overflow;
> +		handled += !!overflow;

No need to !! here, overflowed returns [0;1] though a small nit
and could be updated later :)

>  
>  		/* event overflow for sure */
>  		data.period = event->hw.last_period;
> @@ -690,7 +690,7 @@ static int p4_pmu_handle_irq(struct pt_r
>  		inc_irq_stat(apic_perf_irqs);
>  	}
>  
> -	return handled > 0;
> +	return handled;
>  }
>  
>  /*
> 
	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Peter Zijlstra]

On Thu, 2010-08-19 at 21:43 +0400, Cyrill Gorcunov wrote:
> > @@ -673,7 +673,7 @@ static int p4_pmu_handle_irq(struct pt_r
> >               if (!overflow && (val & (1ULL << (x86_pmu.cntval_bits
> - 1))))
> >                       continue;
> >  
> > -             handled += overflow;
> > +             handled += !!overflow;
> 
> No need to !! here, overflowed returns [0;1] though a small nit
> and could be updated later :) 

done.

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Thu, Aug 19, 2010 at 04:27:13PM +0200, Peter Zijlstra wrote:
>  		x86_pmu.drain_pebs(regs);
> +	}
>  
>  	for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
>  		struct perf_event *event = cpuc->events[bit];
>  
> +		handled++;
> +
>  		if (!test_bit(bit, cpuc->active_mask))
>  			continue;

Sorry I didn't notice this earlier, but I think you want that handled++
after the if(..) continue pieces.  Otherwise you will always have a number
>1. :-)

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Peter Zijlstra]

On Thu, 2010-08-19 at 17:58 -0400, Don Zickus wrote:
> On Thu, Aug 19, 2010 at 04:27:13PM +0200, Peter Zijlstra wrote:
> >  		x86_pmu.drain_pebs(regs);
> > +	}
> >  
> >  	for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
> >  		struct perf_event *event = cpuc->events[bit];
> >  
> > +		handled++;
> > +
> >  		if (!test_bit(bit, cpuc->active_mask))
> >  			continue;
> 
> Sorry I didn't notice this earlier, but I think you want that handled++
> after the if(..) continue pieces.  Otherwise you will always have a number
> >1. :-)

Only if there's any remaining set bits in the overflow status reg,
right?

But if we want to be paranoid, we should also check if the event that
generated the overflow actually had the INT flag enabled, I guess ;-)

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Thu, Aug 19, 2010 at 04:27:13PM +0200, Peter Zijlstra wrote:
> On Thu, 2010-08-19 at 10:12 -0400, Don Zickus wrote:
> > On Thu, Aug 19, 2010 at 12:45:53PM +0200, Peter Zijlstra wrote:
> > > 
> > > I queued it with that part changed to:
> > 
> > I realized the other day this change doesn't cover the nehalem, core and p4
> > cases which use
> > 
> > intel_pmu_handle_irq
> > p4_pmu_handle_irq
> > 
> > as their handlers.  Though that patch can go on top of Robert's.
> 
> Something like this?

I tested this patch and Robert's on an AMD box and Nehalem box.  Both
worked as intended.  However I did notice that whenever the AMD box
detected handled >1, it was shortly followed by an unknown_nmi that was
properly eaten with Robert's logic.  Whereas on the Nehalem box I saw a
lot of 'handled > 1' messages but very very few of them were followed by
an unknown_nmi message (and those messages that did come were properly
eaten).

Maybe that is just the differences in the cpu designs.

Of course I had to make the one change I mentioned previously for the
perf_event_intel.c file (moving the handled++ logic down a few lines).

I didn't run the test on a P4 box.

Looks great, thanks guys!

Cheers,
Don

> 
> ---
> Index: linux-2.6/arch/x86/kernel/cpu/perf_event_intel.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/cpu/perf_event_intel.c
> +++ linux-2.6/arch/x86/kernel/cpu/perf_event_intel.c
> @@ -713,6 +713,7 @@ static int intel_pmu_handle_irq(struct p
>  	struct cpu_hw_events *cpuc;
>  	int bit, loops;
>  	u64 ack, status;
> +	int handled = 0;
>  
>  	perf_sample_data_init(&data, 0);
>  
> @@ -743,12 +744,16 @@ again:
>  	/*
>  	 * PEBS overflow sets bit 62 in the global status register
>  	 */
> -	if (__test_and_clear_bit(62, (unsigned long *)&status))
> +	if (__test_and_clear_bit(62, (unsigned long *)&status)) {
> +		handled++;
>  		x86_pmu.drain_pebs(regs);
> +	}
>  
>  	for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
>  		struct perf_event *event = cpuc->events[bit];
>  
> +		handled++;
> +
>  		if (!test_bit(bit, cpuc->active_mask))
>  			continue;
>  
> @@ -772,7 +777,7 @@ again:
>  
>  done:
>  	intel_pmu_enable_all(0);
> -	return 1;
> +	return handled;
>  }
>  
>  static struct event_constraint *
> Index: linux-2.6/arch/x86/kernel/cpu/perf_event_p4.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/cpu/perf_event_p4.c
> +++ linux-2.6/arch/x86/kernel/cpu/perf_event_p4.c
> @@ -673,7 +673,7 @@ static int p4_pmu_handle_irq(struct pt_r
>  		if (!overflow && (val & (1ULL << (x86_pmu.cntval_bits - 1))))
>  			continue;
>  
> -		handled += overflow;
> +		handled += !!overflow;
>  
>  		/* event overflow for sure */
>  		data.period = event->hw.last_period;
> @@ -690,7 +690,7 @@ static int p4_pmu_handle_irq(struct pt_r
>  		inc_irq_stat(apic_perf_irqs);
>  	}
>  
> -	return handled > 0;
> +	return handled;
>  }
>  
>  /*
> 

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Ingo Molnar]

* Don Zickus <dzickus@redhat.com> wrote:

> On Thu, Aug 19, 2010 at 04:27:13PM +0200, Peter Zijlstra wrote:
> > On Thu, 2010-08-19 at 10:12 -0400, Don Zickus wrote:
> > > On Thu, Aug 19, 2010 at 12:45:53PM +0200, Peter Zijlstra wrote:
> > > > 
> > > > I queued it with that part changed to:
> > > 
> > > I realized the other day this change doesn't cover the nehalem, core and p4
> > > cases which use
> > > 
> > > intel_pmu_handle_irq
> > > p4_pmu_handle_irq
> > > 
> > > as their handlers.  Though that patch can go on top of Robert's.
> > 
> > Something like this?
> 
> I tested this patch and Robert's on an AMD box and Nehalem box.  Both
> worked as intended.  However I did notice that whenever the AMD box
> detected handled >1, it was shortly followed by an unknown_nmi that was
> properly eaten with Robert's logic.  Whereas on the Nehalem box I saw a
> lot of 'handled > 1' messages but very very few of them were followed by
> an unknown_nmi message (and those messages that did come were properly
> eaten).
> 
> Maybe that is just the differences in the cpu designs.
> 
> Of course I had to make the one change I mentioned previously for the
> perf_event_intel.c file (moving the handled++ logic down a few lines).
> 
> I didn't run the test on a P4 box.
> 
> Looks great, thanks guys!

Please someone send the final version with a changelog, with all the acks and 
tested-by's added, so that i can send it Linuswards.

Thanks,

	Ingo

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Peter Zijlstra]

On Fri, 2010-08-20 at 10:16 +0200, Ingo Molnar wrote:
> 
> 
> Please someone send the final version with a changelog, with all the acks and 
> tested-by's added, so that i can send it Linuswards. 

I've got two patches queued, one from Robert (with a slight
modification, which removes a goto) and one from myself converting the
intel and p4 irq handler.

They're available at:

  programming.kicks-ass.net/sekrit/patches.tar.bz2

I did add Tested-by tags, although I suspect its not the exact same bits
Don tested..

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On 8/20/10, Peter Zijlstra <peterz@infradead.org> wrote:
> On Fri, 2010-08-20 at 10:16 +0200, Ingo Molnar wrote:
>>
>>
>> Please someone send the final version with a changelog, with all the acks
>> and
>> tested-by's added, so that i can send it Linuswards.
>
> I've got two patches queued, one from Robert (with a slight
> modification, which removes a goto) and one from myself converting the
> intel and p4 irq handler.
>
> They're available at:
>
>   programming.kicks-ass.net/sekrit/patches.tar.bz2
>
> I did add Tested-by tags, although I suspect its not the exact same bits
> Don tested..
>

My Ack if needed

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Fri, Aug 20, 2010 at 12:04:11PM +0200, Peter Zijlstra wrote:
> On Fri, 2010-08-20 at 10:16 +0200, Ingo Molnar wrote:
> > 
> > 
> > Please someone send the final version with a changelog, with all the acks and 
> > tested-by's added, so that i can send it Linuswards. 
> 
> I've got two patches queued, one from Robert (with a slight
> modification, which removes a goto) and one from myself converting the
> intel and p4 irq handler.
> 
> They're available at:
> 
>   programming.kicks-ass.net/sekrit/patches.tar.bz2
> 
> I did add Tested-by tags, although I suspect its not the exact same bits
> Don tested..

I can re-test those exact patches later today if needed.

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Ingo Molnar]

* Don Zickus <dzickus@redhat.com> wrote:

> On Fri, Aug 20, 2010 at 12:04:11PM +0200, Peter Zijlstra wrote:
> > On Fri, 2010-08-20 at 10:16 +0200, Ingo Molnar wrote:
> > > 
> > > 
> > > Please someone send the final version with a changelog, with all the acks and 
> > > tested-by's added, so that i can send it Linuswards. 
> > 
> > I've got two patches queued, one from Robert (with a slight
> > modification, which removes a goto) and one from myself converting the
> > intel and p4 irq handler.
> > 
> > They're available at:
> > 
> >   programming.kicks-ass.net/sekrit/patches.tar.bz2
> > 
> > I did add Tested-by tags, although I suspect its not the exact same bits
> > Don tested..
> 
> I can re-test those exact patches later today if needed.

Please pick up -tip in an hour or two and holler if something looks wrong.

Thanks,

	Ingo

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Fri, Aug 20, 2010 at 03:27:37PM +0200, Ingo Molnar wrote:
> 
> * Don Zickus <dzickus@redhat.com> wrote:
> 
> > On Fri, Aug 20, 2010 at 12:04:11PM +0200, Peter Zijlstra wrote:
> > > On Fri, 2010-08-20 at 10:16 +0200, Ingo Molnar wrote:
> > > > 
> > > > 
> > > > Please someone send the final version with a changelog, with all the acks and 
> > > > tested-by's added, so that i can send it Linuswards. 
> > > 
> > > I've got two patches queued, one from Robert (with a slight
> > > modification, which removes a goto) and one from myself converting the
> > > intel and p4 irq handler.
> > > 
> > > They're available at:
> > > 
> > >   programming.kicks-ass.net/sekrit/patches.tar.bz2
> > > 
> > > I did add Tested-by tags, although I suspect its not the exact same bits
> > > Don tested..
> > 
> > I can re-test those exact patches later today if needed.
> 
> Please pick up -tip in an hour or two and holler if something looks wrong.

Easy enough.

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Ingo Molnar]

it's not working so well, i'm getting:

 Uhhuh. NMI received for unknown reason 00 on CPU 9.
 Do you have a strange power saving mode enabled?
 Dazed and confused, but trying to continue

on a nehalem box, after a perf top and perf stat run.

Thanks,

	Ingo

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Fri, Aug 20, 2010 at 04:17:03PM +0200, Ingo Molnar wrote:
> 
> it's not working so well, i'm getting:
> 
>  Uhhuh. NMI received for unknown reason 00 on CPU 9.
>  Do you have a strange power saving mode enabled?
>  Dazed and confused, but trying to continue
> 
> on a nehalem box, after a perf top and perf stat run.
> 
> Thanks,
> 
> 	Ingo
> 

Might not this be the case of AAN93 erratum, which says the following

 | The processor can be configured to issue a PMI (performance monitor interrupt)
 | upon overflow of the IA32_FIXED_CTR0 MSR (309H). A single PMI should be observed on
 | overflow of IA32_FIXED_CTR0, however multiple PMIs are observed when this
 | erratum occurs.

Just a thought.

	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Fri, Aug 20, 2010 at 04:17:03PM +0200, Ingo Molnar wrote:
> 
> it's not working so well, i'm getting:
> 
>  Uhhuh. NMI received for unknown reason 00 on CPU 9.
>  Do you have a strange power saving mode enabled?
>  Dazed and confused, but trying to continue
> 
> on a nehalem box, after a perf top and perf stat run.

After applying the patch below, I ran the following commands on my nehalem
box without reproducing what you are seeing.  Any thing else I can run
that might trigger it? (I also ran them on an amd phenom quad-core box).

I used 2.6.32-rc2 plus Robert's and 2 of PeterZ's patches.

perf top
perf stat -a -e cycles -e instructions -e cache-references -e cache-misses -e branch-misses -- sleep 5
perf record -f -a -e cycles -e instructions -e cache-references -e cache-misses -e branch-misses -- sleep 5

Cheers,
Don


>From 198be1044fa603bc9582a5c19134fdf9a433fff0 Mon Sep 17 00:00:00 2001
From: Don Zickus <dzickus@redhat.com>
Date: Tue, 24 Aug 2010 17:43:17 -0400
Subject: [PATCH] [x86] perf: rename nmi variable to avoid clash with entry point

There is already an entry point named .nmi in entry.S and that seems to clash
with the per_cpu variable nmi defined in commit f3a860d8.  Renaming this
variable avoids the namespace collision.

Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 arch/x86/kernel/cpu/perf_event.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index dd2fceb..2a05ea4 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1205,7 +1205,7 @@ struct pmu_nmi_state {
 	int		handled;
 };
 
-static DEFINE_PER_CPU(struct pmu_nmi_state, nmi);
+static DEFINE_PER_CPU_PAGE_ALIGNED(struct pmu_nmi_state, pmu_nmi);
 
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
@@ -1224,7 +1224,7 @@ perf_event_nmi_handler(struct notifier_block *self,
 		break;
 	case DIE_NMIUNKNOWN:
 		this_nmi = percpu_read(irq_stat.__nmi_count);
-		if (this_nmi != __get_cpu_var(nmi).marked)
+		if (this_nmi != __get_cpu_var(pmu_nmi).marked)
 			/* let the kernel handle the unknown nmi */
 			return NOTIFY_DONE;
 		/*
@@ -1248,8 +1248,8 @@ perf_event_nmi_handler(struct notifier_block *self,
 	this_nmi = percpu_read(irq_stat.__nmi_count);
 	if ((handled > 1) ||
 		/* the next nmi could be a back-to-back nmi */
-	    ((__get_cpu_var(nmi).marked == this_nmi) &&
-	     (__get_cpu_var(nmi).handled > 1))) {
+	    ((__get_cpu_var(pmu_nmi).marked == this_nmi) &&
+	     (__get_cpu_var(pmu_nmi).handled > 1))) {
 		/*
 		 * We could have two subsequent back-to-back nmis: The
 		 * first handles more than one counter, the 2nd
@@ -1260,8 +1260,8 @@ perf_event_nmi_handler(struct notifier_block *self,
 		 * handling more than one counter. We will mark the
 		 * next (3rd) and then drop it if unhandled.
 		 */
-		__get_cpu_var(nmi).marked	= this_nmi + 1;
-		__get_cpu_var(nmi).handled	= handled;
+		__get_cpu_var(pmu_nmi).marked	= this_nmi + 1;
+		__get_cpu_var(pmu_nmi).handled	= handled;
 	}
 
 	return NOTIFY_STOP;
-- 
1.7.2.1

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 19.08.10 21:50:17, Don Zickus wrote:
> I tested this patch and Robert's on an AMD box and Nehalem box.  Both
> worked as intended.  However I did notice that whenever the AMD box
> detected handled >1, it was shortly followed by an unknown_nmi that was
> properly eaten with Robert's logic.  Whereas on the Nehalem box I saw a
> lot of 'handled > 1' messages but very very few of them were followed by
> an unknown_nmi message (and those messages that did come were properly
> eaten).

Don, thanks for testing this.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

[tip:perf/urgent] perf, x86: Fix handle_irq return values

[tip-bot for Peter Zijlstra]

Commit-ID:  4a31bebe71ab2e4f6ecd6e5f9f2ac9f0ff38ff76
Gitweb:     http://git.kernel.org/tip/4a31bebe71ab2e4f6ecd6e5f9f2ac9f0ff38ff76
Author:     Peter Zijlstra <a.p.zijlstra@chello.nl>
AuthorDate: Thu, 19 Aug 2010 16:28:00 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 20 Aug 2010 15:00:06 +0200

perf, x86: Fix handle_irq return values

Now that we rely on the number of handled overflows, ensure all handle_irq
implementations actually return the right number.

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Tested-by: Don Zickus <dzickus@redhat.com>
LKML-Reference: <1282228033.2605.204.camel@laptop>
---
 arch/x86/kernel/cpu/perf_event_intel.c |    9 +++++++--
 arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
index d8d86d0..4539b4b 100644
--- a/arch/x86/kernel/cpu/perf_event_intel.c
+++ b/arch/x86/kernel/cpu/perf_event_intel.c
@@ -713,6 +713,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	struct cpu_hw_events *cpuc;
 	int bit, loops;
 	u64 ack, status;
+	int handled = 0;
 
 	perf_sample_data_init(&data, 0);
 
@@ -743,12 +744,16 @@ again:
 	/*
 	 * PEBS overflow sets bit 62 in the global status register
 	 */
-	if (__test_and_clear_bit(62, (unsigned long *)&status))
+	if (__test_and_clear_bit(62, (unsigned long *)&status)) {
+		handled++;
 		x86_pmu.drain_pebs(regs);
+	}
 
 	for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
 		struct perf_event *event = cpuc->events[bit];
 
+		handled++;
+
 		if (!test_bit(bit, cpuc->active_mask))
 			continue;
 
@@ -772,7 +777,7 @@ again:
 
 done:
 	intel_pmu_enable_all(0);
-	return 1;
+	return handled;
 }
 
 static struct event_constraint *
diff --git a/arch/x86/kernel/cpu/perf_event_p4.c b/arch/x86/kernel/cpu/perf_event_p4.c
index febb12c..d470c91 100644
--- a/arch/x86/kernel/cpu/perf_event_p4.c
+++ b/arch/x86/kernel/cpu/perf_event_p4.c
@@ -690,7 +690,7 @@ static int p4_pmu_handle_irq(struct pt_regs *regs)
 		inc_irq_stat(apic_perf_irqs);
 	}
 
-	return handled > 0;
+	return handled;
 }
 
 /*

[tip:perf/urgent] perf, x86: Try to handle unknown nmis with an enabled PMU

[tip-bot for Robert Richter]

Commit-ID:  8e3e42b4f88602bb6e1f0b74afd80ff4703f79ce
Gitweb:     http://git.kernel.org/tip/8e3e42b4f88602bb6e1f0b74afd80ff4703f79ce
Author:     Robert Richter <robert.richter@amd.com>
AuthorDate: Tue, 17 Aug 2010 16:42:03 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 20 Aug 2010 15:00:05 +0200

perf, x86: Try to handle unknown nmis with an enabled PMU

When the PMU is enabled it is valid to have unhandled nmis, two
events could trigger 'simultaneously' raising two back-to-back
NMIs. If the first NMI handles both, the latter will be empty and daze
the CPU.

The solution to avoid an 'unknown nmi' massage in this case was simply
to stop the nmi handler chain when the PMU is enabled by stating
the nmi was handled. This has the drawback that a) we can not detect
unknown nmis anymore, and b) subsequent nmi handlers are not called.

This patch addresses this. Now, we check this unknown NMI if it could
be a PMU back-to-back NMI. Otherwise we pass it and let the kernel
handle the unknown nmi.

This is a debug log:

cpu #6, nmi #32333, skip_nmi #32330, handled = 1, time = 1934364430
cpu #6, nmi #32334, skip_nmi #32330, handled = 1, time = 1934704616
cpu #6, nmi #32335, skip_nmi #32336, handled = 2, time = 1936032320
cpu #6, nmi #32336, skip_nmi #32336, handled = 0, time = 1936034139
cpu #6, nmi #32337, skip_nmi #32336, handled = 1, time = 1936120100
cpu #6, nmi #32338, skip_nmi #32336, handled = 1, time = 1936404607
cpu #6, nmi #32339, skip_nmi #32336, handled = 1, time = 1937983416
cpu #6, nmi #32340, skip_nmi #32341, handled = 2, time = 1938201032
cpu #6, nmi #32341, skip_nmi #32341, handled = 0, time = 1938202830
cpu #6, nmi #32342, skip_nmi #32341, handled = 1, time = 1938443743
cpu #6, nmi #32343, skip_nmi #32341, handled = 1, time = 1939956552
cpu #6, nmi #32344, skip_nmi #32341, handled = 1, time = 1940073224
cpu #6, nmi #32345, skip_nmi #32341, handled = 1, time = 1940485677
cpu #6, nmi #32346, skip_nmi #32347, handled = 2, time = 1941947772
cpu #6, nmi #32347, skip_nmi #32347, handled = 1, time = 1941949818
cpu #6, nmi #32348, skip_nmi #32347, handled = 0, time = 1941951591
Uhhuh. NMI received for unknown reason 00 on CPU 6.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue

Deltas:

nmi #32334 340186
nmi #32335 1327704
nmi #32336 1819      <<<< back-to-back nmi [1]
nmi #32337 85961
nmi #32338 284507
nmi #32339 1578809
nmi #32340 217616
nmi #32341 1798      <<<< back-to-back nmi [2]
nmi #32342 240913
nmi #32343 1512809
nmi #32344 116672
nmi #32345 412453
nmi #32346 1462095   <<<< 1st nmi (standard) handling 2 counters
nmi #32347 2046      <<<< 2nd nmi (back-to-back) handling one counter
nmi #32348 1773      <<<< 3rd nmi (back-to-back) handling no counter! [3]

For  back-to-back nmi detection there are the following rules:

The PMU nmi handler was handling more than one counter and no
counter was handled in the subsequent nmi (see [1] and [2] above).

There is another case if there are two subsequent back-to-back nmis
[3]. The 2nd is detected as back-to-back because the first handled
more than one counter. If the second handles one counter and the 3rd
handles nothing, we drop the 3rd nmi because it could be a
back-to-back nmi.

Signed-off-by: Robert Richter <robert.richter@amd.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Tested-by: Don Zickus <dzickus@redhat.com>
LKML-Reference: <20100817152225.GQ26154@erda.amd.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 arch/x86/kernel/cpu/perf_event.c |   59 +++++++++++++++++++++++++++++--------
 1 files changed, 46 insertions(+), 13 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index f2da20f..dd2fceb 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1154,7 +1154,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		/*
 		 * event overflow
 		 */
-		handled		= 1;
+		handled++;
 		data.period	= event->hw.last_period;
 
 		if (!x86_perf_event_set_period(event))
@@ -1200,12 +1200,20 @@ void perf_events_lapic_init(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 }
 
+struct pmu_nmi_state {
+	unsigned int	marked;
+	int		handled;
+};
+
+static DEFINE_PER_CPU(struct pmu_nmi_state, nmi);
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
 			 unsigned long cmd, void *__args)
 {
 	struct die_args *args = __args;
-	struct pt_regs *regs;
+	unsigned int this_nmi;
+	int handled;
 
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
@@ -1214,22 +1222,47 @@ perf_event_nmi_handler(struct notifier_block *self,
 	case DIE_NMI:
 	case DIE_NMI_IPI:
 		break;
-
+	case DIE_NMIUNKNOWN:
+		this_nmi = percpu_read(irq_stat.__nmi_count);
+		if (this_nmi != __get_cpu_var(nmi).marked)
+			/* let the kernel handle the unknown nmi */
+			return NOTIFY_DONE;
+		/*
+		 * This one is a PMU back-to-back nmi. Two events
+		 * trigger 'simultaneously' raising two back-to-back
+		 * NMIs. If the first NMI handles both, the latter
+		 * will be empty and daze the CPU. So, we drop it to
+		 * avoid false-positive 'unknown nmi' messages.
+		 */
+		return NOTIFY_STOP;
 	default:
 		return NOTIFY_DONE;
 	}
 
-	regs = args->regs;
-
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
-	/*
-	 * Can't rely on the handled return value to say it was our NMI, two
-	 * events could trigger 'simultaneously' raising two back-to-back NMIs.
-	 *
-	 * If the first NMI handles both, the latter will be empty and daze
-	 * the CPU.
-	 */
-	x86_pmu.handle_irq(regs);
+
+	handled = x86_pmu.handle_irq(args->regs);
+	if (!handled)
+		return NOTIFY_DONE;
+
+	this_nmi = percpu_read(irq_stat.__nmi_count);
+	if ((handled > 1) ||
+		/* the next nmi could be a back-to-back nmi */
+	    ((__get_cpu_var(nmi).marked == this_nmi) &&
+	     (__get_cpu_var(nmi).handled > 1))) {
+		/*
+		 * We could have two subsequent back-to-back nmis: The
+		 * first handles more than one counter, the 2nd
+		 * handles only one counter and the 3rd handles no
+		 * counter.
+		 *
+		 * This is the 2nd nmi because the previous was
+		 * handling more than one counter. We will mark the
+		 * next (3rd) and then drop it if unhandled.
+		 */
+		__get_cpu_var(nmi).marked	= this_nmi + 1;
+		__get_cpu_var(nmi).handled	= handled;
+	}
 
 	return NOTIFY_STOP;
 }

Re: A question of perf NMI handler

[Andi Kleen]

Don Zickus <dzickus@redhat.com> writes:

> On Wed, Aug 04, 2010 at 08:10:46PM +0400, Cyrill Gorcunov wrote:
>> On Wed, Aug 04, 2010 at 11:50:02AM -0400, Don Zickus wrote:
>> ...
>> > > 
>> > > Well, first I guess having Yinghai CC'ed is a bonus ;)
>> > > The second thing is that I don't get why perf handler can't be _last_
>> > > call in default_do_nmi, if there were any nmi with reason (serr or parity)
>> > > I think they should be calling first which of course don't eliminate
>> > > the former issue but somewhat make it weaken.
>> > 
>> > Because the reason registers are never set.  If they were, then the code
>> > wouldn't have to walk the notify_chain. :-)
>> > 
>> 
>> maybe we're talking about different things. i meant that if there is nmi
>> with a reason (from 0x61) the handling of such nmi should be done before
>> notify_die I think (if only I not miss something behind).
>
> No we are talking about the same thing. :-)  And that code is already
> there.  The problem is the bits in register 0x61 are not always set
> correctly in the case of SERRs (well at least in all the cases I have
> dealt with).  So you can easily can a flood of unknown nmis from an SERR
> and register 0x61 would have the PERR/SERR bits set to 0.  Fun, huh?

Some of this can be handled by APEI on newer systems (if the platform
supports that).

But not all unfortunately if you consider older systems.

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only.

Re: A question of perf NMI handler

[Don Zickus]

On Wed, Aug 04, 2010 at 05:02:41PM +0200, Peter Zijlstra wrote:
> So then the first interrupt will see 3+ overflows, return 3+, and will
> thus eat 2+ NMIs, only one of which will be the pending interrupt,
> leaving 1+ NMIs from other sources to consume unhandled.
> 
> In which case Yinghai will have to press his NMI button 2+ times before
> it registers.
> 
> That said, that might be a better situation than always consuming
> unknown NMIs.. 

Well if the worse case scenario is only one extra NMI, then I can change
the logic in my patch to eat a max of 1 possible NMI instead of 2 as in the
example you gave above.

It still won't be 100% accurate but how often are people running perf
where they need 4 counters, have to hit an external nmi button or run into
broken firmware all at the same time? :-)

Cheers,
Don

Re: A question of perf NMI handler

[Andi Kleen]

Peter Zijlstra <peterz@infradead.org> writes:
>
> Suppose you have 4 counters (AMD, intel-nhm+), when more than 2 overflow
> the first will raise the PMI, if the other 2+ overflow before we disable
> the PMU it will try to raise 2+ more PMIs, but because hardware only has
> a single interrupt pending bit it will at most cause a single extra
> interrupt after we finish servicing the first one.
>
> So then the first interrupt will see 3+ overflows, return 3+, and will
> thus eat 2+ NMIs, only one of which will be the pending interrupt,
> leaving 1+ NMIs from other sources to consume unhandled.
>
> In which case Yinghai will have to press his NMI button 2+ times before
> it registers.
>
> That said, that might be a better situation than always consuming
> unknown NMIs.. 

One alternative would be to stop using NMIs for perf counters in common cases.

If you have PEBS and your events support PEBS then PEBS can give you a
lot of information inside the irq off region. That works for common
events at least.

Also traditionally interrupt off regions are shrinking in Linux,
so is it really still worth all the trouble just to profile inside them.

e.g. one could make nmi profiling an option with default off.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: A question of perf NMI handler

[Don Zickus]

On Wed, Aug 04, 2010 at 05:21:10PM +0800, Lin Ming wrote:
> With nmi_watchdog enabled, perf_event_nmi_handler always return
> NOTIFY_STOP(active_events > 0), and the notifier call chain will not
> call further.
> 
> If it was not perf NMI, does the perf nmi handler may stop the real NMI
> handler get called because NOTIFY_STOP is returned??

Yes I sent a cheap and dirty patch to address this a couple of weeks ago

http://lkml.indiana.edu/hypermail//linux/kernel/1007.2/02590.html

Unfortunately, no responded. :-(  Of course, it could have been so gross
no one wanted to comment on it. :-)

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 537 bytes --]

I'll test tip later today to see if I can reproduce it.

Cheers,
Don

Ingo Molnar <mingo@elte.hu> wrote:

>
>it's not working so well, i'm getting:
>
> Uhhuh. NMI received for unknown reason 00 on CPU 9.
> Do you have a strange power saving mode enabled?
> Dazed and confused, but trying to continue
>
>on a nehalem box, after a perf top and perf stat run.
>
>Thanks,
>
>	Ingo
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Ingo Molnar]

* Don Zickus <dzickus@redhat.com> wrote:

> I'll test tip later today to see if I can reproduce it.
> 
> Cheers,
> Don
> 
> Ingo Molnar <mingo@elte.hu> wrote:
> 
> >
> >it's not working so well, i'm getting:
> >
> > Uhhuh. NMI received for unknown reason 00 on CPU 9.
> > Do you have a strange power saving mode enabled?
> > Dazed and confused, but trying to continue
> >
> >on a nehalem box, after a perf top and perf stat run.

FYI, it does not trigger on an AMD box.

Thanks,

	Ingo

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Don Zickus <dzickus@redhat.com> wrote:
> 
> > I'll test tip later today to see if I can reproduce it.
> > 
> > Cheers,
> > Don
> > 
> > Ingo Molnar <mingo@elte.hu> wrote:
> > 
> > >
> > >it's not working so well, i'm getting:
> > >
> > > Uhhuh. NMI received for unknown reason 00 on CPU 9.
> > > Do you have a strange power saving mode enabled?
> > > Dazed and confused, but trying to continue
> > >
> > >on a nehalem box, after a perf top and perf stat run.
> 
> FYI, it does not trigger on an AMD box.

Ok, to not hold up the perf/urgent flow i zapped these two commits for 
the time being:

 4a31beb: perf, x86: Fix handle_irq return values
 8e3e42b: perf, x86: Try to handle unknown nmis with an enabled PMU

We can apply them if they take a form that dont introduce a different 
kind of (and more visible) regression.

Thanks,

	Ingo

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Mon, Aug 23, 2010 at 10:53:39AM +0200, Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > 
> > * Don Zickus <dzickus@redhat.com> wrote:
> > 
> > > I'll test tip later today to see if I can reproduce it.
> > > 
> > > Cheers,
> > > Don
> > > 
> > > Ingo Molnar <mingo@elte.hu> wrote:
> > > 
> > > >
> > > >it's not working so well, i'm getting:
> > > >
> > > > Uhhuh. NMI received for unknown reason 00 on CPU 9.
> > > > Do you have a strange power saving mode enabled?
> > > > Dazed and confused, but trying to continue
> > > >
> > > >on a nehalem box, after a perf top and perf stat run.
> > 
> > FYI, it does not trigger on an AMD box.
> 
> Ok, to not hold up the perf/urgent flow i zapped these two commits for 
> the time being:
> 
>  4a31beb: perf, x86: Fix handle_irq return values
>  8e3e42b: perf, x86: Try to handle unknown nmis with an enabled PMU
> 
> We can apply them if they take a form that dont introduce a different 
> kind of (and more visible) regression.
> 
> Thanks,
> 
> 	Ingo
> 

Btw, guys, I fail to see how new nmi_watchdog work, we have
default_do_nmi
  if (!(reason & 0xc0)) {
    if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT) == == NOTIFY_STOP)
      return
    if (nmi_watchdog_tick(regs, reason))
      return

but perf_event_nmi_handler returns NOTIFY_STOP when watchdog is perf event
and nmi_watchdog_tick _never_ called, or (most probably) I miss something?

	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 24.08.10 12:22:52, Cyrill Gorcunov wrote:
> Btw, guys, I fail to see how new nmi_watchdog work, we have
> default_do_nmi
>   if (!(reason & 0xc0)) {
>     if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT) == == NOTIFY_STOP)
>       return
>     if (nmi_watchdog_tick(regs, reason))
>       return
> 
> but perf_event_nmi_handler returns NOTIFY_STOP when watchdog is perf event
> and nmi_watchdog_tick _never_ called, or (most probably) I miss something?

The watchdog is disabled during profiling (perf and oprofile) by
calling disable_lapic_nmi_watchdog().

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 24, 2010 at 07:09:36PM +0200, Robert Richter wrote:
> On 24.08.10 12:22:52, Cyrill Gorcunov wrote:
> > Btw, guys, I fail to see how new nmi_watchdog work, we have
> > default_do_nmi
> >   if (!(reason & 0xc0)) {
> >     if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT) == == NOTIFY_STOP)
> >       return
> >     if (nmi_watchdog_tick(regs, reason))
> >       return
> > 
> > but perf_event_nmi_handler returns NOTIFY_STOP when watchdog is perf event
> > and nmi_watchdog_tick _never_ called, or (most probably) I miss something?
> 
> The watchdog is disabled during profiling (perf and oprofile) by
> calling disable_lapic_nmi_watchdog().
> 
> -Robert
>

Huh? iirc Don have switched nmi watchdog to native perf subsystem, ie watchdog
uses PERF_COUNT_HW_CPU_CYCLES event, letme check...
 
	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 24, 2010 at 09:20:22PM +0400, Cyrill Gorcunov wrote:
> On Tue, Aug 24, 2010 at 07:09:36PM +0200, Robert Richter wrote:
> > On 24.08.10 12:22:52, Cyrill Gorcunov wrote:
> > > Btw, guys, I fail to see how new nmi_watchdog work, we have
> > > default_do_nmi
> > >   if (!(reason & 0xc0)) {
> > >     if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT) == == NOTIFY_STOP)
> > >       return
> > >     if (nmi_watchdog_tick(regs, reason))
> > >       return
> > > 
> > > but perf_event_nmi_handler returns NOTIFY_STOP when watchdog is perf event
> > > and nmi_watchdog_tick _never_ called, or (most probably) I miss something?
> > 
> > The watchdog is disabled during profiling (perf and oprofile) by
> > calling disable_lapic_nmi_watchdog().
> > 
> > -Robert
> >
> 
> Huh? iirc Don have switched nmi watchdog to native perf subsystem, ie watchdog
> uses PERF_COUNT_HW_CPU_CYCLES event, letme check...
>  
> 	-- Cyrill

False alarm, perf watchdog uses own handler. Sorry for noise ;)

	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 23.08.10 04:53:39, Ingo Molnar wrote:
> Ok, to not hold up the perf/urgent flow i zapped these two commits for 
> the time being:
> 
>  4a31beb: perf, x86: Fix handle_irq return values
>  8e3e42b: perf, x86: Try to handle unknown nmis with an enabled PMU

Ingo, are those commits in some branch to fetch them for testing? I
did not find them in tip/master.

Thanks,

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 24, 2010 at 07:15:05PM +0200, Robert Richter wrote:
> On 23.08.10 04:53:39, Ingo Molnar wrote:
> > Ok, to not hold up the perf/urgent flow i zapped these two commits for 
> > the time being:
> > 
> >  4a31beb: perf, x86: Fix handle_irq return values
> >  8e3e42b: perf, x86: Try to handle unknown nmis with an enabled PMU
> 
> Ingo, are those commits in some branch to fetch them for testing? I
> did not find them in tip/master.
> 
> Thanks,
> 
> -Robert
> 
> -- 
> Advanced Micro Devices, Inc.
> Operating System Research Center
>

Robert, they should be in -tip tree, I have

| commit 4a31bebe71ab2e4f6ecd6e5f9f2ac9f0ff38ff76
| Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
| Date:   Thu Aug 19 16:28:00 2010 +0200
|
|    perf, x86: Fix handle_irq return values
    
	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Tue, Aug 24, 2010 at 09:28:06PM +0400, Cyrill Gorcunov wrote:
> On Tue, Aug 24, 2010 at 07:15:05PM +0200, Robert Richter wrote:
> > On 23.08.10 04:53:39, Ingo Molnar wrote:
> > > Ok, to not hold up the perf/urgent flow i zapped these two commits for 
> > > the time being:
> > > 
> > >  4a31beb: perf, x86: Fix handle_irq return values
> > >  8e3e42b: perf, x86: Try to handle unknown nmis with an enabled PMU
> > 
> > Ingo, are those commits in some branch to fetch them for testing? I
> > did not find them in tip/master.
> > 
> > Thanks,
> > 
> > -Robert
> > 
> > -- 
> > Advanced Micro Devices, Inc.
> > Operating System Research Center
> >
> 
> Robert, they should be in -tip tree, I have
> 
> | commit 4a31bebe71ab2e4f6ecd6e5f9f2ac9f0ff38ff76
> | Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
> | Date:   Thu Aug 19 16:28:00 2010 +0200
> |
> |    perf, x86: Fix handle_irq return values

Has anyone else seen kernel panics when enabling the LOCKUP_DETECTOR or
apply Robert's patch?  I keep getting a panic with unable to handle paging
request everytime either Robert's patch or the lockup detector uses
__get_cpu_var().  And only on 2.6.36 code.

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 24, 2010 at 02:46:58PM -0400, Don Zickus wrote:
> On Tue, Aug 24, 2010 at 09:28:06PM +0400, Cyrill Gorcunov wrote:
> > On Tue, Aug 24, 2010 at 07:15:05PM +0200, Robert Richter wrote:
> > > On 23.08.10 04:53:39, Ingo Molnar wrote:
> > > > Ok, to not hold up the perf/urgent flow i zapped these two commits for 
> > > > the time being:
> > > > 
> > > >  4a31beb: perf, x86: Fix handle_irq return values
> > > >  8e3e42b: perf, x86: Try to handle unknown nmis with an enabled PMU
> > > 
> > > Ingo, are those commits in some branch to fetch them for testing? I
> > > did not find them in tip/master.
> > > 
> > > Thanks,
> > > 
> > > -Robert
> > > 
> > > -- 
> > > Advanced Micro Devices, Inc.
> > > Operating System Research Center
> > >
> > 
> > Robert, they should be in -tip tree, I have
> > 
> > | commit 4a31bebe71ab2e4f6ecd6e5f9f2ac9f0ff38ff76
> > | Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > | Date:   Thu Aug 19 16:28:00 2010 +0200
> > |
> > |    perf, x86: Fix handle_irq return values
> 
> Has anyone else seen kernel panics when enabling the LOCKUP_DETECTOR or
> apply Robert's patch?  I keep getting a panic with unable to handle paging
> request everytime either Robert's patch or the lockup detector uses
> __get_cpu_var().  And only on 2.6.36 code.
> 
> Cheers,
> Don
> 

I use 2.6.34 atm. Letme try 2.6.36 (which might require some time to recompile).

	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 24, 2010 at 10:54:58PM +0400, Cyrill Gorcunov wrote:
...
> > > | commit 4a31bebe71ab2e4f6ecd6e5f9f2ac9f0ff38ff76
> > > | Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > > | Date:   Thu Aug 19 16:28:00 2010 +0200
> > > |
> > > |    perf, x86: Fix handle_irq return values
> > 
> > Has anyone else seen kernel panics when enabling the LOCKUP_DETECTOR or
> > apply Robert's patch?  I keep getting a panic with unable to handle paging
> > request everytime either Robert's patch or the lockup detector uses
> > __get_cpu_var().  And only on 2.6.36 code.
> > 
> > Cheers,
> > Don
> > 
> 
> I use 2.6.34 atm. Letme try 2.6.36 (which might require some time to recompile).
> 
> 	-- Cyrill

Don, for me it fails with somehow unrelated page handling fault - in
reiserfs_evict_inode O_o Fails in __get_cpu_var I suspect might means
a problem either in per-cpu allocator itself or we screw pointer somehow.
Weird.

	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Tue, Aug 24, 2010 at 11:52:40PM +0400, Cyrill Gorcunov wrote:
> > I use 2.6.34 atm. Letme try 2.6.36 (which might require some time to recompile).
> > 
> > 	-- Cyrill
> 
> Don, for me it fails with somehow unrelated page handling fault - in
> reiserfs_evict_inode O_o Fails in __get_cpu_var I suspect might means
> a problem either in per-cpu allocator itself or we screw pointer somehow.
> Weird.

I just found out (with the help of the crash utility and Dave A.) that
Robert's percpu struct nmi clashes with the exception entry point .nmi.
I only see this problem in 2.6.36, so I am not sure what changed with
regards to compiler flags to confuse variables with text segments.

But renaming the percpu struct nmi to nmidon fixed the problem for me (I
am open to other suggestions :-) ).

Regarding your reiserfs, what was the variable's name?

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Tue, Aug 24, 2010 at 04:27:18PM -0400, Don Zickus wrote:
> On Tue, Aug 24, 2010 at 11:52:40PM +0400, Cyrill Gorcunov wrote:
> > > I use 2.6.34 atm. Letme try 2.6.36 (which might require some time to recompile).
> > > 
> > > 	-- Cyrill
> > 
> > Don, for me it fails with somehow unrelated page handling fault - in
> > reiserfs_evict_inode O_o Fails in __get_cpu_var I suspect might means
> > a problem either in per-cpu allocator itself or we screw pointer somehow.
> > Weird.
> 
> I just found out (with the help of the crash utility and Dave A.) that
> Robert's percpu struct nmi clashes with the exception entry point .nmi.
> I only see this problem in 2.6.36, so I am not sure what changed with
> regards to compiler flags to confuse variables with text segments.
> 

yeah, I suspect name clashes here but then I did grep over per-cpu variables
in whole kernel and didn't find match so I thought the assumption was wrong,
but eventually it happens to be true but via other way :) good to know, thanks!

> But renaming the percpu struct nmi to nmidon fixed the problem for me (I
> am open to other suggestions :-) ).

nmi_don_zickus ;) well, I think nmi_pmu or something like that
might be a bit modest ;)

> 
> Regarding your reiserfs, what was the variable's name?
> 

It seems to be different, it's pity that I had only 80x25 vga
mode and was unable to snap the whole log. But actually I didn't
even check precisely all .config options I had set since I was
more interested in early stage where per-cpu access should already
happen rather then real init'ed environmen. But I think I'll be
moving completely to .36 this week so we will see how it goes.

> Cheers,
> Don
> 
	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Frederic Weisbecker]

On Wed, Aug 25, 2010 at 12:40:38AM +0400, Cyrill Gorcunov wrote:
> On Tue, Aug 24, 2010 at 04:27:18PM -0400, Don Zickus wrote:
> > On Tue, Aug 24, 2010 at 11:52:40PM +0400, Cyrill Gorcunov wrote:
> > > > I use 2.6.34 atm. Letme try 2.6.36 (which might require some time to recompile).
> > > > 
> > > > 	-- Cyrill
> > > 
> > > Don, for me it fails with somehow unrelated page handling fault - in
> > > reiserfs_evict_inode O_o Fails in __get_cpu_var I suspect might means
> > > a problem either in per-cpu allocator itself or we screw pointer somehow.
> > > Weird.
> > 
> > I just found out (with the help of the crash utility and Dave A.) that
> > Robert's percpu struct nmi clashes with the exception entry point .nmi.
> > I only see this problem in 2.6.36, so I am not sure what changed with
> > regards to compiler flags to confuse variables with text segments.
> > 
> 
> yeah, I suspect name clashes here but then I did grep over per-cpu variables
> in whole kernel and didn't find match so I thought the assumption was wrong,
> but eventually it happens to be true but via other way :) good to know, thanks!
> 
> > But renaming the percpu struct nmi to nmidon fixed the problem for me (I
> > am open to other suggestions :-) ).
> 
> nmi_don_zickus ;) well, I think nmi_pmu or something like that
> might be a bit modest ;)
> 
> > 
> > Regarding your reiserfs, what was the variable's name?
> > 
> 
> It seems to be different, it's pity that I had only 80x25 vga
> mode and was unable to snap the whole log. But actually I didn't
> even check precisely all .config options I had set since I was
> more interested in early stage where per-cpu access should already
> happen rather then real init'ed environmen. But I think I'll be
> moving completely to .36 this week so we will see how it goes.


There was a missing "return" in the end of reiserfs_evict_inode.

It has been fixed there:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=f4ae2faa40199b97b12f508234640bc565d166f8

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Thu, Aug 26, 2010 at 3:52 AM, Frederic Weisbecker <fweisbec@gmail.com> wrote:
...
>>
>> It seems to be different, it's pity that I had only 80x25 vga
>> mode and was unable to snap the whole log. But actually I didn't
>> even check precisely all .config options I had set since I was
>> more interested in early stage where per-cpu access should already
>> happen rather then real init'ed environmen. But I think I'll be
>> moving completely to .36 this week so we will see how it goes.
>
>
> There was a missing "return" in the end of reiserfs_evict_inode.
>
> It has been fixed there:
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=f4ae2faa40199b97b12f508234640bc565d166f8
>
>

Thanks for info, Frederic!

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 24.08.10 16:27:18, Don Zickus wrote:
> On Tue, Aug 24, 2010 at 11:52:40PM +0400, Cyrill Gorcunov wrote:
> > > I use 2.6.34 atm. Letme try 2.6.36 (which might require some time to recompile).
> > > 
> > > 	-- Cyrill
> > 
> > Don, for me it fails with somehow unrelated page handling fault - in
> > reiserfs_evict_inode O_o Fails in __get_cpu_var I suspect might means
> > a problem either in per-cpu allocator itself or we screw pointer somehow.
> > Weird.
> 
> I just found out (with the help of the crash utility and Dave A.) that
> Robert's percpu struct nmi clashes with the exception entry point .nmi.
> I only see this problem in 2.6.36, so I am not sure what changed with
> regards to compiler flags to confuse variables with text segments.

I was testing the patches also with CONFIG_LOCKUP_DETECTOR=y without
crashes.

> 
> But renaming the percpu struct nmi to nmidon fixed the problem for me (I
> am open to other suggestions :-) ).

Also I was looking at the differences between .35 and tip/perf/urgent
and could not find a namespace collision.

Hmm...

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Mon, Aug 23, 2010 at 10:53:39AM +0200, Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > 
> > * Don Zickus <dzickus@redhat.com> wrote:
> > 
> > > I'll test tip later today to see if I can reproduce it.
> > > 
> > > Cheers,
> > > Don
> > > 
> > > Ingo Molnar <mingo@elte.hu> wrote:
> > > 
> > > >
> > > >it's not working so well, i'm getting:
> > > >
> > > > Uhhuh. NMI received for unknown reason 00 on CPU 9.
> > > > Do you have a strange power saving mode enabled?
> > > > Dazed and confused, but trying to continue
> > > >
> > > >on a nehalem box, after a perf top and perf stat run.
> > 
> > FYI, it does not trigger on an AMD box.
> 
> Ok, to not hold up the perf/urgent flow i zapped these two commits for 
> the time being:
> 
>  4a31beb: perf, x86: Fix handle_irq return values
>  8e3e42b: perf, x86: Try to handle unknown nmis with an enabled PMU
> 
> We can apply them if they take a form that dont introduce a different 
> kind of (and more visible) regression.

So this patch fixes it, though I haven't convince myself why (perhaps
babysitting my 4 month old isn't helping :-))

The code now enters the loop and reprocesses the new status which properly
increments handled to 2 and thus the new logic takes care of it.

Cheers,
Don


diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
index 4539b4b..d16ebd8 100644
--- a/arch/x86/kernel/cpu/perf_event_intel.c
+++ b/arch/x86/kernel/cpu/perf_event_intel.c
@@ -738,6 +738,7 @@ again:
 
 	inc_irq_stat(apic_perf_irqs);
 	ack = status;
+	intel_pmu_ack_status(ack);
 
 	intel_pmu_lbr_read();
 
@@ -766,8 +767,6 @@ again:
 			x86_pmu_stop(event);
 	}
 
-	intel_pmu_ack_status(ack);
-
 	/*
 	 * Repeat if there is more work to be done:
 	 */

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 26.08.10 17:14:24, Don Zickus wrote:
> diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
> index 4539b4b..d16ebd8 100644
> --- a/arch/x86/kernel/cpu/perf_event_intel.c
> +++ b/arch/x86/kernel/cpu/perf_event_intel.c
> @@ -738,6 +738,7 @@ again:
>  
>  	inc_irq_stat(apic_perf_irqs);
>  	ack = status;
> +	intel_pmu_ack_status(ack);

Yes, not immediately ack'ing the status was suspect to me too. Though
it must then be the same counter that retriggers. Or, it is a cpu
bug. You could add a debug print of the status register for the case
the loop is reentered, would be interesting...

Thank for fixing this.

-Robert

>  
>  	intel_pmu_lbr_read();
>  
> @@ -766,8 +767,6 @@ again:
>  			x86_pmu_stop(event);
>  	}
>  
> -	intel_pmu_ack_status(ack);
> -
>  	/*
>  	 * Repeat if there is more work to be done:
>  	 */
> 

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Fri, Aug 27, 2010 at 09:51:32AM +0200, Robert Richter wrote:
> On 26.08.10 17:14:24, Don Zickus wrote:
> > diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
> > index 4539b4b..d16ebd8 100644
> > --- a/arch/x86/kernel/cpu/perf_event_intel.c
> > +++ b/arch/x86/kernel/cpu/perf_event_intel.c
> > @@ -738,6 +738,7 @@ again:
> >  
> >  	inc_irq_stat(apic_perf_irqs);
> >  	ack = status;
> > +	intel_pmu_ack_status(ack);
> 
> Yes, not immediately ack'ing the status was suspect to me too. Though
> it must then be the same counter that retriggers. Or, it is a cpu
> bug. You could add a debug print of the status register for the case
> the loop is reentered, would be interesting...

It seems to be the same counter.  I wonder if the act of processing it
'intel_save_and_restart' cleared the status bit on the perfselX bit but
not in the global status register.  Thus it triggered again and we
accidentally cleared it when we ack'd later in the code.

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 26.08.10 17:14:24, Don Zickus wrote:
> diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
> index 4539b4b..d16ebd8 100644
> --- a/arch/x86/kernel/cpu/perf_event_intel.c
> +++ b/arch/x86/kernel/cpu/perf_event_intel.c
> @@ -738,6 +738,7 @@ again:
>  
>  	inc_irq_stat(apic_perf_irqs);
>  	ack = status;
> +	intel_pmu_ack_status(ack);

I would slightly change the patch:

There is no need for the ack variable anymore, you could directly work
with the status.

I would call intel_pmu_ack_status() as close as possible after the
intel_pmu_get_status(), which is after 'again:'.

-Robert

>  
>  	intel_pmu_lbr_read();
>  
> @@ -766,8 +767,6 @@ again:
>  			x86_pmu_stop(event);
>  	}
>  
> -	intel_pmu_ack_status(ack);
> -
>  	/*
>  	 * Repeat if there is more work to be done:
>  	 */
> 

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Fri, Aug 27, 2010 at 10:10:38AM +0200, Robert Richter wrote:
> On 26.08.10 17:14:24, Don Zickus wrote:
> > diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
> > index 4539b4b..d16ebd8 100644
> > --- a/arch/x86/kernel/cpu/perf_event_intel.c
> > +++ b/arch/x86/kernel/cpu/perf_event_intel.c
> > @@ -738,6 +738,7 @@ again:
> >  
> >  	inc_irq_stat(apic_perf_irqs);
> >  	ack = status;
> > +	intel_pmu_ack_status(ack);
> 
> I would slightly change the patch:
> 
> There is no need for the ack variable anymore, you could directly work
> with the status.
> 
> I would call intel_pmu_ack_status() as close as possible after the
> intel_pmu_get_status(), which is after 'again:'.

Yeah, I can do that.  The other patch was just a proof of concept to see
what others thought.

What is funny is that this problem was masked by the
perf_event_nmi_handler swallowing all the nmis.  I wonder if we were
losing events as a result of this bug too because if you think about it,
we processed the first event, a second event came in and we accidentally
ack'd it, thus dropping it on the floor.  Now I wonder how the event was
ever reloaded, unless it was by accident because of how the scheduler
deals with perf counters (perf_start/stop all the time).

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 27.08.10 09:44:29, Don Zickus wrote:
> On Fri, Aug 27, 2010 at 10:10:38AM +0200, Robert Richter wrote:
> > On 26.08.10 17:14:24, Don Zickus wrote:
> > > diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
> > > index 4539b4b..d16ebd8 100644
> > > --- a/arch/x86/kernel/cpu/perf_event_intel.c
> > > +++ b/arch/x86/kernel/cpu/perf_event_intel.c
> > > @@ -738,6 +738,7 @@ again:
> > >  
> > >  	inc_irq_stat(apic_perf_irqs);
> > >  	ack = status;
> > > +	intel_pmu_ack_status(ack);
> > 
> > I would slightly change the patch:
> > 
> > There is no need for the ack variable anymore, you could directly work
> > with the status.
> > 
> > I would call intel_pmu_ack_status() as close as possible after the
> > intel_pmu_get_status(), which is after 'again:'.
> 
> Yeah, I can do that.  The other patch was just a proof of concept to see
> what others thought.
> 
> What is funny is that this problem was masked by the
> perf_event_nmi_handler swallowing all the nmis.  I wonder if we were
> losing events as a result of this bug too because if you think about it,
> we processed the first event, a second event came in and we accidentally
> ack'd it, thus dropping it on the floor.

Yes, this could be the case, but only for handled counters. So it
would be interesting to see for this case the status mask of the
current and previous get_status call.

> Now I wonder how the event was
> ever reloaded, unless it was by accident because of how the scheduler
> deals with perf counters (perf_start/stop all the time).

The nmi might be queued be the cpu regardless of of the overflow
state.

I am wondering why this happens at all, because events are disabled by
wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL, 0). Hmm, maybe this is exactly the
reason because the nmi could fire again after reenabling the counters.

Is there a reason for disabling all counters?

-Robert

> 
> Cheers,
> Don
> 

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Fri, Aug 27, 2010 at 04:05:23PM +0200, Robert Richter wrote:
> > What is funny is that this problem was masked by the
> > perf_event_nmi_handler swallowing all the nmis.  I wonder if we were
> > losing events as a result of this bug too because if you think about it,
> > we processed the first event, a second event came in and we accidentally
> > ack'd it, thus dropping it on the floor.
> 
> Yes, this could be the case, but only for handled counters. So it
> would be interesting to see for this case the status mask of the
> current and previous get_status call.

The status masks seem to be identical, 0x1 (and when I forced pmc0
unusable, everything was 0x2).

> 
> > Now I wonder how the event was
> > ever reloaded, unless it was by accident because of how the scheduler
> > deals with perf counters (perf_start/stop all the time).
> 
> The nmi might be queued be the cpu regardless of of the overflow
> state.
> 
> I am wondering why this happens at all, because events are disabled by
> wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL, 0). Hmm, maybe this is exactly the

Heh.  Not sure why it isn't working then.  Then again you shouldn't need
the loop if it was working I would think.

> reason because the nmi could fire again after reenabling the counters.
> 
> Is there a reason for disabling all counters?

It would be a nice to have that way we wouldn't have to 'eat' all these
extra nmis.  But I guess it isn't working correctly.

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 27.08.10 11:05:23, Don Zickus wrote:
> The status masks seem to be identical, 0x1 (and when I forced pmc0
> unusable, everything was 0x2).

So this should also happen if only one counter is running?
Back-to-back nmis actually only occur then 2 different counters
trigger simultaneously.

> > > Now I wonder how the event was
> > > ever reloaded, unless it was by accident because of how the scheduler
> > > deals with perf counters (perf_start/stop all the time).
> > 
> > The nmi might be queued be the cpu regardless of of the overflow
> > state.
> > 
> > I am wondering why this happens at all, because events are disabled by
> > wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL, 0). Hmm, maybe this is exactly the
> 
> Heh.  Not sure why it isn't working then.  Then again you shouldn't need
> the loop if it was working I would think.
> 
> > reason because the nmi could fire again after reenabling the counters.
> > 
> > Is there a reason for disabling all counters?
> 
> It would be a nice to have that way we wouldn't have to 'eat' all these
> extra nmis.  But I guess it isn't working correctly.

What about the erratum mentioned in this thread before? We might
identify affected cpus and return handled=2 for them. This solution
will be still better than before. And, all other cpu models have the
nmi detection fixed. In a next step we could try to use a timer for
detection.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Fri, Aug 27, 2010 at 10:10:38AM +0200, Robert Richter wrote:
> >  
> >  	inc_irq_stat(apic_perf_irqs);
> >  	ack = status;
> > +	intel_pmu_ack_status(ack);
> 
> I would slightly change the patch:
> 
> There is no need for the ack variable anymore, you could directly work
> with the status.
> 
> I would call intel_pmu_ack_status() as close as possible after the
> intel_pmu_get_status(), which is after 'again:'.
> 

Here is another version of the patch, which uses Robert's suggestion of
removing the ack variable

Cheers,
Don 

--

From: Don Zickus <dzickus@redhat.com>
Date: Fri, 27 Aug 2010 14:43:03 -0400
Subject: [PATCH] [x86] perf: fix accidentally ack'ing a second event on intel perf counter

During testing of a patch to stop having the perf subsytem swallow nmis,
it was uncovered that Nehalem boxes were randomly getting unknown nmis
when using the perf tool.

Moving the ack'ing of the PMI closer to when we get the status allows
the hardware to properly re-set the PMU bit signaling another PMI was
triggered during the processing of the first PMI.  This allows the new
logic for dealing with the shortcomings of multiple PMIs to handle the
extra NMI by 'eat'ing it later.

Now one can wonder why are we getting a second PMI when we disable all
the PMUs in the begining of the NMI handler to prevent such a case, for
that I do not know.  But I know the fix below helps deal with this quirk.

Tested on multiple Nehalems where the problem was occuring.  With the
patch, the code now loops a second time to handle the second PMI (whereas
before it was not).

Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 arch/x86/kernel/cpu/perf_event_intel.c |    6 ++----
 1 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
index 4539b4b..ee05c90 100644
--- a/arch/x86/kernel/cpu/perf_event_intel.c
+++ b/arch/x86/kernel/cpu/perf_event_intel.c
@@ -712,7 +712,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	struct perf_sample_data data;
 	struct cpu_hw_events *cpuc;
 	int bit, loops;
-	u64 ack, status;
+	u64 status;
 	int handled = 0;
 
 	perf_sample_data_init(&data, 0);
@@ -729,6 +729,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 
 	loops = 0;
 again:
+	intel_pmu_ack_status(status);
 	if (++loops > 100) {
 		WARN_ONCE(1, "perfevents: irq loop stuck!\n");
 		perf_event_print_debug();
@@ -737,7 +738,6 @@ again:
 	}
 
 	inc_irq_stat(apic_perf_irqs);
-	ack = status;
 
 	intel_pmu_lbr_read();
 
@@ -766,8 +766,6 @@ again:
 			x86_pmu_stop(event);
 	}
 
-	intel_pmu_ack_status(ack);
-
 	/*
 	 * Repeat if there is more work to be done:
 	 */
-- 
1.7.2.1

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Yinghai Lu]

On 08/27/2010 11:57 AM, Don Zickus wrote:
> On Fri, Aug 27, 2010 at 10:10:38AM +0200, Robert Richter wrote:
>>>  
>>>  	inc_irq_stat(apic_perf_irqs);
>>>  	ack = status;
>>> +	intel_pmu_ack_status(ack);
>>
>> I would slightly change the patch:
>>
>> There is no need for the ack variable anymore, you could directly work
>> with the status.
>>
>> I would call intel_pmu_ack_status() as close as possible after the
>> intel_pmu_get_status(), which is after 'again:'.
>>
> 
> Here is another version of the patch, which uses Robert's suggestion of
> removing the ack variable

Can you resend all three updated patches?

Thanks

Yinghai

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 27.08.10 14:57:44, Don Zickus wrote:
> From: Don Zickus <dzickus@redhat.com>
> Date: Fri, 27 Aug 2010 14:43:03 -0400
> Subject: [PATCH] [x86] perf: fix accidentally ack'ing a second event on intel perf counter
> 
> During testing of a patch to stop having the perf subsytem swallow nmis,
> it was uncovered that Nehalem boxes were randomly getting unknown nmis
> when using the perf tool.
> 
> Moving the ack'ing of the PMI closer to when we get the status allows
> the hardware to properly re-set the PMU bit signaling another PMI was
> triggered during the processing of the first PMI.  This allows the new
> logic for dealing with the shortcomings of multiple PMIs to handle the
> extra NMI by 'eat'ing it later.
> 
> Now one can wonder why are we getting a second PMI when we disable all
> the PMUs in the begining of the NMI handler to prevent such a case, for
> that I do not know.  But I know the fix below helps deal with this quirk.
> 
> Tested on multiple Nehalems where the problem was occuring.  With the
> patch, the code now loops a second time to handle the second PMI (whereas
> before it was not).
> 
> Signed-off-by: Don Zickus <dzickus@redhat.com>

Looks good to me. Thanks Don.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 20.08.10 11:25:10, Ingo Molnar wrote:

> > Ingo Molnar <mingo@elte.hu> wrote:
> > 
> > >
> > >it's not working so well, i'm getting:
> > >
> > > Uhhuh. NMI received for unknown reason 00 on CPU 9.
> > > Do you have a strange power saving mode enabled?
> > > Dazed and confused, but trying to continue
> > >
> > >on a nehalem box, after a perf top and perf stat run.
> 
> FYI, it does not trigger on an AMD box.

Ingo,

do you mean it does not trigger false positives on AMD? Both patches
applied on top of current tip/perf/urgent (c6db67c) are working on the
systems I have.

You might use the debug patch below for diagnostics.

-Robert

--

>From 1bbb5aa64e96360529c34a593a072e1a84114f04 Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@amd.com>
Date: Wed, 11 Aug 2010 18:14:00 +0200
Subject: [PATCH] debug

Signed-off-by: Robert Richter <robert.richter@amd.com>
---
 arch/x86/kernel/cpu/perf_event.c |   54 ++++++++++++++++++++++++++++++++++++-
 1 files changed, 52 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index dd2fceb..059ef09 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1203,10 +1203,43 @@ void perf_events_lapic_init(void)
 struct pmu_nmi_state {
 	unsigned int	marked;
 	int		handled;
+	u64		timestamp;
 };
 
 static DEFINE_PER_CPU(struct pmu_nmi_state, nmi);
 
+struct nmi_debug {
+	int		cpu;
+	unsigned int	this_nmi;
+	unsigned int	marked;
+	int		handled;
+	u64		timestamp;
+	u64		delta;
+};
+
+static DEFINE_PER_CPU(struct nmi_debug[16], nmi_debug);
+
+static void nmi_handler_debug(void)
+{
+	struct nmi_debug *debug;
+	int i;
+
+	if (!printk_ratelimit())
+		return;
+
+	for (i = 0; i < 16; i++) {
+		debug = &__get_cpu_var(nmi_debug)[i];
+		printk(KERN_EMERG
+		       "cpu #%d, nmi #%d, marked #%d, handled = %d, time = %llu, delta = %llu\n",
+		       debug->cpu,
+		       debug->this_nmi,
+		       debug->marked,
+		       debug->handled,
+		       debug->timestamp,
+		       debug->delta);
+	}
+}
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
 			 unsigned long cmd, void *__args)
@@ -1214,6 +1247,8 @@ perf_event_nmi_handler(struct notifier_block *self,
 	struct die_args *args = __args;
 	unsigned int this_nmi;
 	int handled;
+	struct nmi_debug *debug;
+	u64 timestamp;
 
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
@@ -1224,9 +1259,11 @@ perf_event_nmi_handler(struct notifier_block *self,
 		break;
 	case DIE_NMIUNKNOWN:
 		this_nmi = percpu_read(irq_stat.__nmi_count);
-		if (this_nmi != __get_cpu_var(nmi).marked)
+		if (this_nmi != __get_cpu_var(nmi).marked) {
+			nmi_handler_debug();
 			/* let the kernel handle the unknown nmi */
 			return NOTIFY_DONE;
+		}
 		/*
 		 * This one is a PMU back-to-back nmi. Two events
 		 * trigger 'simultaneously' raising two back-to-back
@@ -1242,10 +1279,21 @@ perf_event_nmi_handler(struct notifier_block *self,
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 
 	handled = x86_pmu.handle_irq(args->regs);
+	this_nmi = percpu_read(irq_stat.__nmi_count);
+
+	debug		= &__get_cpu_var(nmi_debug)[0xf & this_nmi];
+	debug->cpu	= smp_processor_id();
+	debug->this_nmi	= this_nmi;
+	debug->marked	= __get_cpu_var(nmi).marked;
+	debug->handled	= handled;
+	rdtscll(timestamp);
+	debug->delta    = timestamp - __get_cpu_var(nmi).timestamp;
+	__get_cpu_var(nmi).timestamp = timestamp;
+	debug->timestamp = timestamp;
+
 	if (!handled)
 		return NOTIFY_DONE;
 
-	this_nmi = percpu_read(irq_stat.__nmi_count);
 	if ((handled > 1) ||
 		/* the next nmi could be a back-to-back nmi */
 	    ((__get_cpu_var(nmi).marked == this_nmi) &&
@@ -1262,6 +1310,8 @@ perf_event_nmi_handler(struct notifier_block *self,
 		 */
 		__get_cpu_var(nmi).marked	= this_nmi + 1;
 		__get_cpu_var(nmi).handled	= handled;
+		debug->marked	= __get_cpu_var(nmi).marked;
+		debug->handled	= handled;
 	}
 
 	return NOTIFY_STOP;
-- 
1.7.1.1


-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Ingo Molnar]

* Robert Richter <robert.richter@amd.com> wrote:

> On 20.08.10 11:25:10, Ingo Molnar wrote:
> 
> > > Ingo Molnar <mingo@elte.hu> wrote:
> > > 
> > > >
> > > >it's not working so well, i'm getting:
> > > >
> > > > Uhhuh. NMI received for unknown reason 00 on CPU 9.
> > > > Do you have a strange power saving mode enabled?
> > > > Dazed and confused, but trying to continue
> > > >
> > > >on a nehalem box, after a perf top and perf stat run.
> > 
> > FYI, it does not trigger on an AMD box.
> 
> Ingo,
> 
> do you mean it does not trigger false positives on AMD? [...]

Correct, on AMD boxes i do not get this message during the first 'perf 
top' run:

> > > > Uhhuh. NMI received for unknown reason 00 on CPU 9.
> > > > Do you have a strange power saving mode enabled?
> > > > Dazed and confused, but trying to continue

Before the two patches i did not get these messages at all, on any of 
the systems.

> You might use the debug patch below for diagnostics.

Thanks, will try that and report back.

	Ingo

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 3600 bytes --]


* Ingo Molnar <mingo@elte.hu> wrote:

> > You might use the debug patch below for diagnostics.
> 
> Thanks, will try that and report back.

Here's a more detailed description of the regression introduced by:

  4a31beb: perf, x86: Fix handle_irq return values
  8e3e42b: perf, x86: Try to handle unknown nmis with an enabled PMU

Booting into the debug kernel the system boots up fine - no NMI 
messages, as expected.

Then when i start 'perf top' for the first time i get the NMI message 
with this debug output:

 cpu #15, nmi #160, marked #0, handled = 1, time = 333392635730, delta = 11238255
 cpu #15, nmi #161, marked #0, handled = 1, time = 333403779380, delta = 11143650
 cpu #15, nmi #162, marked #0, handled = 1, time = 333415418497, delta = 11639117
 cpu #15, nmi #163, marked #0, handled = 1, time = 333415467084, delta = 48587
 cpu #15, nmi #164, marked #0, handled = 1, time = 333415501531, delta = 34447
 cpu #15, nmi #165, marked #0, handled = 1, time = 333459918106, delta = 44416575
 cpu #15, nmi #166, marked #0, handled = 0, time = 333459923167, delta = 1666
 cpu #15, nmi #151, marked #0, handled = 1, time = 332978597882, delta = 11447002
 cpu #15, nmi #152, marked #0, handled = 1, time = 332978657151, delta = 59269
 cpu #15, nmi #153, marked #0, handled = 1, time = 332978667847, delta = 10696
 cpu #15, nmi #154, marked #0, handled = 1, time = 333023125757, delta = 44457910
 cpu #15, nmi #155, marked #0, handled = 1, time = 333291980833, delta = 268855076
 cpu #15, nmi #156, marked #0, handled = 1, time = 333325663125, delta = 33682292
 cpu #15, nmi #157, marked #0, handled = 1, time = 333348216481, delta = 22553356
 cpu #15, nmi #158, marked #0, handled = 1, time = 333370168887, delta = 21952406
 cpu #15, nmi #159, marked #0, handled = 1, time = 333381397475, delta = 11228588
 Uhhuh. NMI received for unknown reason 00 on CPU 15.
 Do you have a strange power saving mode enabled?
 Dazed and confused, but trying to continue

When i start perf top for a second time, no messages are printed at all. 
The reason is that on one of the CPUs NMIs are 'stuck':

 NMI: 78164 67099 6342 [*] 65677 66119 63796 65395 63995 65012 64151 65082 
      63483 64948 62926 65608 62630

CPU#2 is stuck at 6342.

The NMIs work fine on other CPUs and perf top works (sans the missing 
samples from CPU#2@@), and the NMIs keep ticking.

The CPU is:

 processor	: 2
 vendor_id	: GenuineIntel
 cpu family	: 6
 model		: 26
 model name	: Intel(R) Xeon(R) CPU           X55600 @ 2.80GHz
 stepping	: 5
 cpu MHz	: 2794.000
 cache size	: 8192 KB
 physical id	: 0
 siblings	: 8
 core id	: 1
 cpu cores	: 4
 apicid		: 2
 initial apicid	: 2
 fpu		: yes
 fpu_exception	: yes
 cpuid level	: 11
 wp		: yes
 flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
 bogomips	: 5599.98
 clflush size	: 64
 cache_alignment: 64
 address sizes	: 40 bits physical, 48 bits virtual
 power management:

The PMU init is:

 Performance Events: PEBS fmt1+, Nehalem events, Intel PMU driver.
 ... version:                3
 ... bit width:              48
 ... generic registers:      4
 ... value mask:             0000ffffffffffff
 ... max period:             000000007fffffff
 ... fixed-purpose events:   3
 ... event mask:             000000070000000f

I've attached the config as well.

Thanks,

	Ingo

[-- Attachment #2: config --]
[-- Type: text/plain, Size: 83223 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.36-rc2
# Thu Aug 26 07:41:31 2010
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_EARLY_RES=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
# CONFIG_KTIME_SCALAR is not set
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_LZO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
# CONFIG_TASK_XACCT is not set
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=20
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_CGROUPS is not set
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_LZO=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
CONFIG_PERF_COUNTERS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_COMPAT_BRK=y
CONFIG_SLAB=y
# CONFIG_SLUB is not set
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
# CONFIG_OPROFILE_EVENT_MULTIPLEX is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_OPTPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y

#
# GCOV-based kernel profiling
#
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_INTEGRITY is not set
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_INLINE_SPIN_TRYLOCK is not set
# CONFIG_INLINE_SPIN_TRYLOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK is not set
# CONFIG_INLINE_SPIN_LOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK_IRQ is not set
# CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
CONFIG_INLINE_SPIN_UNLOCK=y
# CONFIG_INLINE_SPIN_UNLOCK_BH is not set
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
# CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_READ_TRYLOCK is not set
# CONFIG_INLINE_READ_LOCK is not set
# CONFIG_INLINE_READ_LOCK_BH is not set
# CONFIG_INLINE_READ_LOCK_IRQ is not set
# CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
CONFIG_INLINE_READ_UNLOCK=y
# CONFIG_INLINE_READ_UNLOCK_BH is not set
CONFIG_INLINE_READ_UNLOCK_IRQ=y
# CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_WRITE_TRYLOCK is not set
# CONFIG_INLINE_WRITE_LOCK is not set
# CONFIG_INLINE_WRITE_LOCK_BH is not set
# CONFIG_INLINE_WRITE_LOCK_IRQ is not set
# CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
CONFIG_INLINE_WRITE_UNLOCK=y
# CONFIG_INLINE_WRITE_UNLOCK_BH is not set
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
# CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_FREEZER=y

#
# Processor type and features
#
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
# CONFIG_SPARSE_IRQ is not set
CONFIG_X86_MPPARSE=y
CONFIG_X86_EXTENDED_PLATFORM=y
# CONFIG_X86_VSMP is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
CONFIG_NO_BOOTMEM=y
# CONFIG_MEMTEST is not set
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=7
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
# CONFIG_AMD_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_IOMMU_API is not set
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=255
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set
CONFIG_X86_THERMAL_VECTOR=y
# CONFIG_I8K is not set
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
# CONFIG_MICROCODE_AMD is not set
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_NUMA=y
CONFIG_K8_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=6
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_COMPACTION is not set
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_MEMORY_FAILURE is not set
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_X86_LOW_RESERVE=64
CONFIG_MTRR=y
# CONFIG_MTRR_SANITIZER is not set
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
# CONFIG_EFI is not set
# CONFIG_SECCOMP is not set
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
# CONFIG_SCHED_HRTICK is not set
CONFIG_KEXEC=y
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x1000000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_COMPAT_VDSO=y
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y

#
# Power management and ACPI options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP=y
CONFIG_SUSPEND_NVS=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
# CONFIG_HIBERNATION is not set
# CONFIG_PM_RUNTIME is not set
CONFIG_PM_OPS=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
# CONFIG_ACPI_POWER_METER is not set
CONFIG_ACPI_SYSFS_POWER=y
# CONFIG_ACPI_EC_DEBUGFS is not set
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_NUMA=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_SBS=m
# CONFIG_ACPI_HED is not set
# CONFIG_ACPI_APEI is not set
# CONFIG_SFI is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_DEBUG=y
CONFIG_CPU_FREQ_STAT=m
CONFIG_CPU_FREQ_STAT_DETAILS=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m

#
# CPUFreq processor drivers
#
# CONFIG_X86_PCC_CPUFREQ is not set
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_POWERNOW_K8=y
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_P4_CLOCKMOD is not set

#
# shared options
#
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
# CONFIG_INTEL_IDLE is not set

#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCI_CNB20LE_QUIRK is not set
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=m
CONFIG_PCIEAER=y
# CONFIG_PCIE_ECRC is not set
# CONFIG_PCIEAER_INJECT is not set
CONFIG_PCIEASPM=y
# CONFIG_PCIEASPM_DEBUG is not set
CONFIG_ARCH_SUPPORTS_MSI=y
# CONFIG_PCI_MSI is not set
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
CONFIG_HT_IRQ=y
# CONFIG_PCI_IOV is not set
CONFIG_PCI_IOAPIC=y
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
CONFIG_PCCARD=y
CONFIG_PCMCIA=y
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y

#
# PC-card bridges
#
CONFIG_YENTA=y
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PD6729=m
CONFIG_I82092=m
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
# CONFIG_HOTPLUG_PCI_CPCI is not set
CONFIG_HOTPLUG_PCI_SHPC=m

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=y
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=y
# CONFIG_XFRM_SUB_POLICY is not set
CONFIG_XFRM_MIGRATE=y
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
# CONFIG_IP_MROUTE_MULTIPLE_TABLES is not set
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=y
CONFIG_TCP_CONG_CUBIC=m
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
# CONFIG_TCP_CONG_YEAH is not set
# CONFIG_TCP_CONG_ILLINOIS is not set
CONFIG_DEFAULT_BIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="bic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=m
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
# CONFIG_IPV6_MIP6 is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
CONFIG_IPV6_SIT=m
# CONFIG_IPV6_SIT_6RD is not set
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
CONFIG_NETFILTER_DEBUG=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
# CONFIG_NF_CT_PROTO_UDPLITE is not set
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
# CONFIG_NF_CT_NETLINK is not set
# CONFIG_NETFILTER_TPROXY is not set
CONFIG_NETFILTER_XTABLES=m

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m

#
# Xtables targets
#
# CONFIG_NETFILTER_XT_TARGET_CHECKSUM is not set
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
# CONFIG_NETFILTER_XT_TARGET_CT is not set
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
# CONFIG_NETFILTER_XT_TARGET_IDLETIMER is not set
# CONFIG_NETFILTER_XT_TARGET_LED is not set
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
# CONFIG_NETFILTER_XT_TARGET_RATEEST is not set
# CONFIG_NETFILTER_XT_TARGET_TEE is not set
# CONFIG_NETFILTER_XT_TARGET_TRACE is not set
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
# CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP is not set

#
# Xtables matches
#
# CONFIG_NETFILTER_XT_MATCH_CLUSTER is not set
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
# CONFIG_NETFILTER_XT_MATCH_CONNLIMIT is not set
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
# CONFIG_NETFILTER_XT_MATCH_CPU is not set
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
# CONFIG_NETFILTER_XT_MATCH_IPRANGE is not set
# CONFIG_NETFILTER_XT_MATCH_IPVS is not set
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
# CONFIG_NETFILTER_XT_MATCH_OSF is not set
# CONFIG_NETFILTER_XT_MATCH_OWNER is not set
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
# CONFIG_NETFILTER_XT_MATCH_RATEEST is not set
CONFIG_NETFILTER_XT_MATCH_REALM=m
# CONFIG_NETFILTER_XT_MATCH_RECENT is not set
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
# CONFIG_NETFILTER_XT_MATCH_TIME is not set
# CONFIG_NETFILTER_XT_MATCH_U32 is not set
CONFIG_IP_VS=m
# CONFIG_IP_VS_IPV6 is not set
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
# CONFIG_IP_VS_PROTO_SCTP is not set

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
# CONFIG_IP_NF_SECURITY is not set
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

#
# IPv6: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
# CONFIG_IP6_NF_SECURITY is not set

#
# DECnet: Netfilter Configuration
#
# CONFIG_DECNET_NF_GRABULATOR is not set
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
# CONFIG_BRIDGE_EBT_IP6 is not set
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m
# CONFIG_BRIDGE_EBT_NFLOG is not set
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m

#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_CCID3_RTO=100
CONFIG_IP_DCCP_TFRC_LIB=y

#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
# CONFIG_NET_DCCPPROBE is not set
CONFIG_IP_SCTP=m
# CONFIG_NET_SCTPPROBE is not set
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
# CONFIG_RDS is not set
CONFIG_TIPC=m
# CONFIG_TIPC_ADVANCED is not set
# CONFIG_TIPC_DEBUG is not set
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
# CONFIG_ATM_MPOA is not set
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
# CONFIG_L2TP is not set
CONFIG_STP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
# CONFIG_NET_DSA is not set
CONFIG_VLAN_8021Q=m
# CONFIG_VLAN_8021Q_GVRP is not set
CONFIG_DECNET=m
CONFIG_DECNET_ROUTER=y
CONFIG_LLC=y
# CONFIG_LLC2 is not set
CONFIG_IPX=m
# CONFIG_IPX_INTERN is not set
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=m
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_IPDDP_DECAP=y
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
CONFIG_WAN_ROUTER=m
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
# CONFIG_NET_SCH_MULTIQ is not set
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
# CONFIG_NET_SCH_DRR is not set
CONFIG_NET_SCH_INGRESS=m

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
# CONFIG_NET_CLS_FLOW is not set
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
# CONFIG_NET_ACT_NAT is not set
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
# CONFIG_NET_ACT_SKBEDIT is not set
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set
CONFIG_DNS_RESOLVER=y
CONFIG_RPS=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NET_TCPPROBE is not set
# CONFIG_NET_DROP_MONITOR is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
CONFIG_IRDA=m

#
# IrDA protocols
#
CONFIG_IRLAN=m
CONFIG_IRNET=m
CONFIG_IRCOMM=m
# CONFIG_IRDA_ULTRA is not set

#
# IrDA options
#
CONFIG_IRDA_CACHE_LAST_LSAP=y
CONFIG_IRDA_FAST_RR=y
# CONFIG_IRDA_DEBUG is not set

#
# Infrared-port device drivers
#

#
# SIR device drivers
#
CONFIG_IRTTY_SIR=m

#
# Dongle support
#
CONFIG_DONGLE=y
CONFIG_ESI_DONGLE=m
CONFIG_ACTISYS_DONGLE=m
CONFIG_TEKRAM_DONGLE=m
CONFIG_TOIM3232_DONGLE=m
CONFIG_LITELINK_DONGLE=m
CONFIG_MA600_DONGLE=m
CONFIG_GIRBIL_DONGLE=m
CONFIG_MCP2120_DONGLE=m
CONFIG_OLD_BELKIN_DONGLE=m
CONFIG_ACT200L_DONGLE=m
# CONFIG_KINGSUN_DONGLE is not set
# CONFIG_KSDAZZLE_DONGLE is not set
# CONFIG_KS959_DONGLE is not set

#
# FIR device drivers
#
CONFIG_USB_IRDA=m
CONFIG_SIGMATEL_FIR=m
CONFIG_NSC_FIR=m
CONFIG_WINBOND_FIR=m
CONFIG_SMC_IRCC_FIR=m
CONFIG_ALI_FIR=m
CONFIG_VLSI_FIR=m
CONFIG_VIA_FIR=m
CONFIG_MCS_FIR=m
CONFIG_BT=m
CONFIG_BT_L2CAP=m
CONFIG_BT_SCO=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_HIDP=m

#
# Bluetooth device drivers
#
# CONFIG_BT_HCIBTUSB is not set
# CONFIG_BT_HCIBTSDIO is not set
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
# CONFIG_BT_HCIUART_ATH3K is not set
# CONFIG_BT_HCIUART_LL is not set
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIDTL1=m
CONFIG_BT_HCIBT3C=m
CONFIG_BT_HCIBLUECARD=m
CONFIG_BT_HCIBTUART=m
CONFIG_BT_HCIVHCI=m
# CONFIG_BT_MRVL is not set
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
# CONFIG_CFG80211 is not set
# CONFIG_LIB80211 is not set

#
# CFG80211 needs to be enabled for MAC80211
#

#
# Some wireless drivers require a rate control algorithm
#
# CONFIG_WIMAX is not set
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
# CONFIG_CAIF is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
# CONFIG_DEVTMPFS is not set
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_MTD is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
CONFIG_PARPORT_PC_PCMCIA=m
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
# CONFIG_PARIDE is not set
CONFIG_BLK_CPQ_DA=y
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=m
CONFIG_BLK_DEV_UMEM=m
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
# CONFIG_BLK_DEV_DRBD is not set
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_SX8=m
CONFIG_BLK_DEV_UB=m
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
CONFIG_ATA_OVER_ETH=m
# CONFIG_BLK_DEV_HD is not set
CONFIG_MISC_DEVICES=y
# CONFIG_AD525X_DPOT is not set
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
CONFIG_SGI_IOC4=m
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
# CONFIG_ICS932S401 is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_CS5535_MFGPT is not set
# CONFIG_HP_ILO is not set
# CONFIG_ISL29003 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_SENSORS_BH1780 is not set
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
# CONFIG_VMWARE_BALLOON is not set
# CONFIG_BMP085 is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_AT24 is not set
# CONFIG_EEPROM_LEGACY is not set
# CONFIG_EEPROM_MAX6875 is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_CB710_CORE is not set
# CONFIG_IWMC3200TOP is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=m
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=m
# CONFIG_SCSI_FC_TGT_ATTRS is not set
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
# CONFIG_ISCSI_BOOT_SYSFS is not set
# CONFIG_SCSI_CXGB3_ISCSI is not set
# CONFIG_SCSI_BNX2_ISCSI is not set
# CONFIG_BE2ISCSI is not set
CONFIG_BLK_DEV_3W_XXXX_RAID=m
# CONFIG_SCSI_HPSA is not set
CONFIG_SCSI_3W_9XXX=m
# CONFIG_SCSI_3W_SAS is not set
CONFIG_SCSI_ACARD=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=y
CONFIG_AIC7XXX_CMDS_PER_DEVICE=4
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=4
CONFIG_AIC79XX_RESET_DELAY_MS=15000
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
CONFIG_SCSI_ARCMSR=m
# CONFIG_SCSI_ARCMSR_AER is not set
CONFIG_MEGARAID_NEWGEN=y
CONFIG_MEGARAID_MM=m
CONFIG_MEGARAID_MAILBOX=m
CONFIG_MEGARAID_LEGACY=m
CONFIG_MEGARAID_SAS=m
# CONFIG_SCSI_MPT2SAS is not set
CONFIG_SCSI_HPTIOP=m
CONFIG_SCSI_BUSLOGIC=m
# CONFIG_VMWARE_PVSCSI is not set
# CONFIG_LIBFC is not set
# CONFIG_LIBFCOE is not set
# CONFIG_FCOE is not set
# CONFIG_FCOE_FNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
CONFIG_SCSI_GDTH=m
CONFIG_SCSI_IPS=m
CONFIG_SCSI_INITIO=m
CONFIG_SCSI_INIA100=m
CONFIG_SCSI_PPA=m
CONFIG_SCSI_IMM=m
# CONFIG_SCSI_IZIP_EPP16 is not set
# CONFIG_SCSI_IZIP_SLOW_CTR is not set
CONFIG_SCSI_STEX=m
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_SYM53C8XX_MMIO=y
# CONFIG_SCSI_IPR is not set
CONFIG_SCSI_QLOGIC_1280=m
CONFIG_SCSI_QLA_FC=m
CONFIG_SCSI_QLA_ISCSI=m
CONFIG_SCSI_LPFC=m
# CONFIG_SCSI_LPFC_DEBUG_FS is not set
CONFIG_SCSI_DC395x=m
CONFIG_SCSI_DC390T=m
# CONFIG_SCSI_PMCRAID is not set
# CONFIG_SCSI_PM8001 is not set
CONFIG_SCSI_SRP=m
# CONFIG_SCSI_BFA_FC is not set
# CONFIG_SCSI_LOWLEVEL_PCMCIA is not set
# CONFIG_SCSI_DH is not set
# CONFIG_SCSI_OSD_INITIATOR is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y

#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=y
# CONFIG_SATA_AHCI_PLATFORM is not set
CONFIG_SATA_INIC162X=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y

#
# SFF controllers with custom DMA interface
#
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
CONFIG_SATA_SX4=m
CONFIG_ATA_BMDMA=y

#
# SATA SFF controllers with BMDMA
#
CONFIG_ATA_PIIX=y
CONFIG_SATA_MV=m
CONFIG_SATA_NV=y
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_SVW=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m

#
# PATA SFF controllers with BMDMA
#
CONFIG_PATA_ALI=m
CONFIG_PATA_AMD=y
CONFIG_PATA_ARTOP=m
CONFIG_PATA_ATIIXP=m
# CONFIG_PATA_ATP867X is not set
CONFIG_PATA_CMD64X=m
CONFIG_PATA_CS5520=m
CONFIG_PATA_CS5530=m
CONFIG_PATA_CYPRESS=m
CONFIG_PATA_EFAR=m
CONFIG_PATA_HPT366=m
CONFIG_PATA_HPT37X=m
CONFIG_PATA_HPT3X2N=m
CONFIG_PATA_HPT3X3=m
# CONFIG_PATA_HPT3X3_DMA is not set
CONFIG_PATA_IT8213=m
CONFIG_PATA_IT821X=m
CONFIG_PATA_JMICRON=m
CONFIG_PATA_MARVELL=m
CONFIG_PATA_NETCELL=m
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87415 is not set
CONFIG_PATA_OLDPIIX=y
CONFIG_PATA_OPTIDMA=m
CONFIG_PATA_PDC2027X=m
CONFIG_PATA_PDC_OLD=m
CONFIG_PATA_RADISYS=m
# CONFIG_PATA_RDC is not set
CONFIG_PATA_SC1200=m
# CONFIG_PATA_SCH is not set
CONFIG_PATA_SERVERWORKS=m
CONFIG_PATA_SIL680=m
CONFIG_PATA_SIS=m
# CONFIG_PATA_TOSHIBA is not set
CONFIG_PATA_TRIFLEX=m
CONFIG_PATA_VIA=m
CONFIG_PATA_WINBOND=m

#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
CONFIG_PATA_MPIIX=m
CONFIG_PATA_NS87410=m
CONFIG_PATA_OPTI=m
CONFIG_PATA_PCMCIA=m
CONFIG_PATA_RZ1000=m

#
# Generic fallback / legacy drivers
#
# CONFIG_PATA_ACPI is not set
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_LEGACY is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
# CONFIG_MULTICORE_RAID456 is not set
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
# CONFIG_DM_LOG_USERSPACE is not set
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
# CONFIG_DM_MULTIPATH_QL is not set
# CONFIG_DM_MULTIPATH_ST is not set
# CONFIG_DM_DELAY is not set
# CONFIG_DM_UEVENT is not set
CONFIG_FUSION=y
CONFIG_FUSION_SPI=m
CONFIG_FUSION_FC=m
CONFIG_FUSION_SAS=m
CONFIG_FUSION_MAX_SGE=40
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LAN=m
# CONFIG_FUSION_LOGGING is not set

#
# IEEE 1394 (FireWire) support
#

#
# You can enable one or both FireWire driver stacks.
#

#
# The newer stack is recommended.
#
# CONFIG_FIREWIRE is not set
CONFIG_IEEE1394=m
CONFIG_IEEE1394_OHCI1394=m
CONFIG_IEEE1394_PCILYNX=m
CONFIG_IEEE1394_SBP2=m
# CONFIG_IEEE1394_SBP2_PHYS_DMA is not set
CONFIG_IEEE1394_ETH1394_ROM_ENTRY=y
CONFIG_IEEE1394_ETH1394=m
CONFIG_IEEE1394_RAWIO=m
CONFIG_IEEE1394_VIDEO1394=m
CONFIG_IEEE1394_DV1394=m
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
# CONFIG_FIREWIRE_NOSY is not set
CONFIG_I2O=m
# CONFIG_I2O_LCT_NOTIFY_ON_CHANGES is not set
CONFIG_I2O_EXT_ADAPTEC=y
CONFIG_I2O_EXT_ADAPTEC_DMA64=y
# CONFIG_I2O_CONFIG is not set
CONFIG_I2O_BUS=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_IFB=m
CONFIG_DUMMY=m
CONFIG_BONDING=m
# CONFIG_MACVLAN is not set
CONFIG_EQUALIZER=m
CONFIG_TUN=m
# CONFIG_VETH is not set
CONFIG_NET_SB1000=m
# CONFIG_ARCNET is not set
CONFIG_PHYLIB=y

#
# MII PHY device drivers
#
CONFIG_MARVELL_PHY=m
CONFIG_DAVICOM_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_LXT_PHY=m
CONFIG_CICADA_PHY=m
CONFIG_VITESSE_PHY=m
CONFIG_SMSC_PHY=m
CONFIG_BROADCOM_PHY=m
# CONFIG_ICPLUS_PHY is not set
# CONFIG_REALTEK_PHY is not set
# CONFIG_NATIONAL_PHY is not set
# CONFIG_STE10XP is not set
# CONFIG_LSI_ET1011C_PHY is not set
# CONFIG_MICREL_PHY is not set
# CONFIG_FIXED_PHY is not set
# CONFIG_MDIO_BITBANG is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_CASSINI=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=y
CONFIG_TYPHOON=m
# CONFIG_ETHOC is not set
# CONFIG_DNET is not set
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_DE2104X_DSL=0
CONFIG_TULIP=m
# CONFIG_TULIP_MWI is not set
CONFIG_TULIP_MMIO=y
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_ULI526X=m
CONFIG_PCMCIA_XIRCOM=m
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
# CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
# CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
CONFIG_AMD8111_ETH=m
CONFIG_ADAPTEC_STARFIRE=m
# CONFIG_KSZ884X_PCI is not set
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
CONFIG_FORCEDETH=y
CONFIG_E100=y
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=m
CONFIG_8139CP=m
CONFIG_8139TOO=y
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_R6040 is not set
CONFIG_SIS900=m
CONFIG_EPIC100=m
# CONFIG_SMSC9420 is not set
CONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not set
# CONFIG_TLAN is not set
# CONFIG_KS8851_MLL is not set
CONFIG_VIA_RHINE=m
CONFIG_VIA_RHINE_MMIO=y
CONFIG_SC92031=m
CONFIG_NET_POCKET=y
CONFIG_ATP=m
CONFIG_DE600=m
CONFIG_DE620=m
# CONFIG_ATL2 is not set
CONFIG_NETDEV_1000=y
CONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not set
CONFIG_DL2K=m
CONFIG_E1000=y
CONFIG_E1000E=y
# CONFIG_IP1000 is not set
CONFIG_IGB=y
# CONFIG_IGBVF is not set
CONFIG_NS83820=m
CONFIG_HAMACHI=m
CONFIG_YELLOWFIN=m
CONFIG_R8169=m
CONFIG_R8169_VLAN=y
# CONFIG_SIS190 is not set
CONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not set
CONFIG_SKY2=m
# CONFIG_SKY2_DEBUG is not set
CONFIG_VIA_VELOCITY=m
CONFIG_TIGON3=y
CONFIG_BNX2=m
# CONFIG_CNIC is not set
CONFIG_QLA3XXX=m
CONFIG_ATL1=m
# CONFIG_ATL1E is not set
# CONFIG_ATL1C is not set
# CONFIG_JME is not set
CONFIG_NETDEV_10000=y
CONFIG_MDIO=m
CONFIG_CHELSIO_T1=m
CONFIG_CHELSIO_T1_1G=y
CONFIG_CHELSIO_T3_DEPENDS=y
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4_DEPENDS=y
# CONFIG_CHELSIO_T4 is not set
CONFIG_CHELSIO_T4VF_DEPENDS=y
# CONFIG_CHELSIO_T4VF is not set
# CONFIG_ENIC is not set
# CONFIG_IXGBE is not set
CONFIG_IXGB=m
CONFIG_S2IO=m
CONFIG_MYRI10GE=m
# CONFIG_NIU is not set
# CONFIG_MLX4_EN is not set
# CONFIG_MLX4_CORE is not set
# CONFIG_TEHUTI is not set
# CONFIG_BNX2X is not set
# CONFIG_QLCNIC is not set
# CONFIG_QLGE is not set
# CONFIG_SFC is not set
# CONFIG_BE2NET is not set
CONFIG_TR=y
CONFIG_IBMOL=m
CONFIG_3C359=m
# CONFIG_TMS380TR is not set
CONFIG_WLAN=y
# CONFIG_PCMCIA_RAYCS is not set
# CONFIG_AIRO is not set
# CONFIG_ATMEL is not set
# CONFIG_AIRO_CS is not set
# CONFIG_PCMCIA_WL3501 is not set
# CONFIG_PRISM54 is not set
# CONFIG_USB_ZD1201 is not set
# CONFIG_HOSTAP is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#

#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_AX8817X=m
CONFIG_USB_NET_CDCETHER=m
# CONFIG_USB_NET_CDC_EEM is not set
CONFIG_USB_NET_DM9601=m
# CONFIG_USB_NET_SMSC75XX is not set
# CONFIG_USB_NET_SMSC95XX is not set
CONFIG_USB_NET_GL620A=m
CONFIG_USB_NET_NET1080=m
CONFIG_USB_NET_PLUSB=m
CONFIG_USB_NET_MCS7830=m
CONFIG_USB_NET_RNDIS_HOST=m
CONFIG_USB_NET_CDC_SUBSET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_KC2190=y
CONFIG_USB_NET_ZAURUS=m
# CONFIG_USB_HSO is not set
# CONFIG_USB_NET_INT51X1 is not set
# CONFIG_USB_IPHETH is not set
# CONFIG_USB_SIERRA_NET is not set
CONFIG_NET_PCMCIA=y
CONFIG_PCMCIA_3C589=m
CONFIG_PCMCIA_3C574=m
CONFIG_PCMCIA_FMVJ18X=m
CONFIG_PCMCIA_PCNET=m
CONFIG_PCMCIA_NMCLAN=m
CONFIG_PCMCIA_SMC91C92=m
CONFIG_PCMCIA_XIRC2PS=m
CONFIG_PCMCIA_AXNET=m
# CONFIG_PCMCIA_IBMTR is not set
# CONFIG_WAN is not set
CONFIG_ATM_DRIVERS=y
# CONFIG_ATM_DUMMY is not set
CONFIG_ATM_TCP=m
CONFIG_ATM_LANAI=m
CONFIG_ATM_ENI=m
# CONFIG_ATM_ENI_DEBUG is not set
# CONFIG_ATM_ENI_TUNE_BURST is not set
CONFIG_ATM_FIRESTREAM=m
# CONFIG_ATM_ZATM is not set
# CONFIG_ATM_NICSTAR is not set
CONFIG_ATM_IDT77252=m
# CONFIG_ATM_IDT77252_DEBUG is not set
# CONFIG_ATM_IDT77252_RCV_ALL is not set
CONFIG_ATM_IDT77252_USE_SUNI=y
CONFIG_ATM_AMBASSADOR=m
# CONFIG_ATM_AMBASSADOR_DEBUG is not set
CONFIG_ATM_HORIZON=m
# CONFIG_ATM_HORIZON_DEBUG is not set
# CONFIG_ATM_IA is not set
# CONFIG_ATM_FORE200E is not set
CONFIG_ATM_HE=m
# CONFIG_ATM_HE_USE_SUNI is not set
# CONFIG_ATM_SOLOS is not set

#
# CAIF transport drivers
#
CONFIG_FDDI=y
# CONFIG_DEFXX is not set
CONFIG_SKFP=m
# CONFIG_HIPPI is not set
CONFIG_PLIP=m
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
# CONFIG_PPP_BSDCOMP is not set
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_PPPOATM=m
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLHC=m
CONFIG_SLIP_SMART=y
# CONFIG_SLIP_MODE_SLIP6 is not set
CONFIG_NET_FC=y
CONFIG_NETCONSOLE=y
# CONFIG_NETCONSOLE_DYNAMIC is not set
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_VMXNET3 is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=y
CONFIG_INPUT_SPARSEKMAP=m

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
CONFIG_KEYBOARD_STOWAWAY=m
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_SENTELIC is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_SERIAL=m
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
CONFIG_JOYSTICK_A3D=m
CONFIG_JOYSTICK_ADI=m
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=m
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDJOY=m
# CONFIG_JOYSTICK_ZHENHUA is not set
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
CONFIG_JOYSTICK_JOYDUMP=m
# CONFIG_JOYSTICK_XPAD is not set
# CONFIG_INPUT_TABLET is not set
CONFIG_INPUT_TOUCHSCREEN=y
# CONFIG_TOUCHSCREEN_AD7879 is not set
# CONFIG_TOUCHSCREEN_DYNAPRO is not set
# CONFIG_TOUCHSCREEN_HAMPSHIRE is not set
# CONFIG_TOUCHSCREEN_EETI is not set
# CONFIG_TOUCHSCREEN_FUJITSU is not set
CONFIG_TOUCHSCREEN_GUNZE=m
CONFIG_TOUCHSCREEN_ELO=m
# CONFIG_TOUCHSCREEN_WACOM_W8001 is not set
# CONFIG_TOUCHSCREEN_MCS5000 is not set
CONFIG_TOUCHSCREEN_MTOUCH=m
# CONFIG_TOUCHSCREEN_INEXIO is not set
CONFIG_TOUCHSCREEN_MK712=m
CONFIG_TOUCHSCREEN_PENMOUNT=m
# CONFIG_TOUCHSCREEN_QT602240 is not set
CONFIG_TOUCHSCREEN_TOUCHRIGHT=m
CONFIG_TOUCHSCREEN_TOUCHWIN=m
# CONFIG_TOUCHSCREEN_WM97XX is not set
# CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
# CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
# CONFIG_TOUCHSCREEN_TSC2007 is not set
# CONFIG_TOUCHSCREEN_TPS6507X is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_APANEL is not set
CONFIG_INPUT_ATLAS_BTNS=m
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
CONFIG_INPUT_UINPUT=m
# CONFIG_INPUT_WINBOND_CIR is not set
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_ADXL34X is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
# CONFIG_SERIO_ALTERA_PS2 is not set
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_DEVKMEM=y
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_COMPUTONE is not set
# CONFIG_ROCKETPORT is not set
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_MOXA_INTELLIO is not set
# CONFIG_MOXA_SMARTIO is not set
# CONFIG_ISI is not set
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
CONFIG_N_HDLC=m
# CONFIG_N_GSM is not set
# CONFIG_RISCOM8 is not set
# CONFIG_SPECIALIX is not set
# CONFIG_STALDRV is not set
# CONFIG_NOZOMI is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CS=m
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
CONFIG_SERIAL_8250_RSA=y

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MFD_HSU is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=m
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
CONFIG_PRINTER=m
CONFIG_LP_CONSOLE=y
CONFIG_PPDEV=m
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
# CONFIG_HW_RANDOM_TIMERIOMEM is not set
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=y
CONFIG_NVRAM=y
CONFIG_R3964=m
# CONFIG_APPLICOM is not set

#
# PCMCIA character devices
#
# CONFIG_SYNCLINK_CS is not set
CONFIG_CARDMAN_4000=m
CONFIG_CARDMAN_4040=m
# CONFIG_IPWIRELESS is not set
CONFIG_MWAVE=m
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=m
CONFIG_TCG_TIS=m
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
# CONFIG_RAMOOPS is not set
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
# CONFIG_I2C_MUX is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_SMBUS=m
CONFIG_I2C_ALGOBIT=m

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
# CONFIG_I2C_ISCH is not set
CONFIG_I2C_PIIX4=y
CONFIG_I2C_NFORCE2=y
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m

#
# ACPI drivers
#
# CONFIG_I2C_SCMI is not set

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_XILINX is not set

#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set

#
# Other I2C/SMBus bus drivers
#
CONFIG_I2C_STUB=m
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_SPI is not set

#
# PPS support
#
# CONFIG_PPS is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
CONFIG_W1=m
CONFIG_W1_CON=y

#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=m
CONFIG_W1_MASTER_DS2490=m
CONFIG_W1_MASTER_DS2482=m

#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=m
CONFIG_W1_SLAVE_SMEM=m
# CONFIG_W1_SLAVE_DS2431 is not set
CONFIG_W1_SLAVE_DS2433=m
CONFIG_W1_SLAVE_DS2433_CRC=y
# CONFIG_W1_SLAVE_DS2760 is not set
# CONFIG_W1_SLAVE_BQ27000 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_BATTERY_MAX17040 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
# CONFIG_SENSORS_ADT7411 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7475 is not set
# CONFIG_SENSORS_ASC7621 is not set
CONFIG_SENSORS_K8TEMP=m
# CONFIG_SENSORS_K10TEMP is not set
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS1621=m
# CONFIG_SENSORS_I5K_AMB is not set
CONFIG_SENSORS_F71805F=m
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_G760A is not set
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
# CONFIG_SENSORS_CORETEMP is not set
# CONFIG_SENSORS_PKGTEMP is not set
# CONFIG_SENSORS_IBMAEM is not set
# CONFIG_SENSORS_IBMPEX is not set
CONFIG_SENSORS_IT87=m
# CONFIG_SENSORS_JC42 is not set
CONFIG_SENSORS_LM63=m
# CONFIG_SENSORS_LM73 is not set
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4245 is not set
# CONFIG_SENSORS_LM95241 is not set
CONFIG_SENSORS_MAX1619=m
# CONFIG_SENSORS_MAX6650 is not set
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_SIS5595=m
# CONFIG_SENSORS_SMM665 is not set
# CONFIG_SENSORS_DME1737 is not set
# CONFIG_SENSORS_EMC1403 is not set
# CONFIG_SENSORS_EMC2103 is not set
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_AMC6821 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_TMP102 is not set
# CONFIG_SENSORS_TMP401 is not set
# CONFIG_SENSORS_TMP421 is not set
# CONFIG_SENSORS_VIA_CPUTEMP is not set
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83L785TS=m
# CONFIG_SENSORS_W83L786NG is not set
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_HDAPS=m
# CONFIG_SENSORS_LIS3_I2C is not set
# CONFIG_SENSORS_APPLESMC is not set

#
# ACPI drivers
#
# CONFIG_SENSORS_ATK0110 is not set
# CONFIG_SENSORS_LIS3LV02D is not set
CONFIG_THERMAL=y
# CONFIG_THERMAL_HWMON is not set
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
# CONFIG_F71808E_WDT is not set
# CONFIG_SC520_WDT is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
CONFIG_IBMASR=m
# CONFIG_WAFER_WDT is not set
CONFIG_I6300ESB_WDT=m
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y
# CONFIG_IT8712F_WDT is not set
# CONFIG_IT87_WDT is not set
# CONFIG_HP_WATCHDOG is not set
# CONFIG_SC1200_WDT is not set
CONFIG_PC87413_WDT=m
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_SMSC_SCH311X_WDT is not set
# CONFIG_SMSC37B787_WDT is not set
CONFIG_W83627HF_WDT=m
CONFIG_W83697HF_WDT=m
# CONFIG_W83697UG_WDT is not set
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
# CONFIG_SBC_EPX_C3_WATCHDOG is not set

#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m

#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
# CONFIG_SSB_B43_PCI_BRIDGE is not set
CONFIG_SSB_PCMCIAHOST_POSSIBLE=y
# CONFIG_SSB_PCMCIAHOST is not set
CONFIG_SSB_SDIOHOST_POSSIBLE=y
# CONFIG_SSB_SDIOHOST is not set
# CONFIG_SSB_DEBUG is not set
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
CONFIG_MFD_SUPPORT=y
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_88PM860X is not set
CONFIG_MFD_SM501=m
# CONFIG_HTC_PASIC3 is not set
# CONFIG_TPS6507X is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_MFD_STMPE is not set
# CONFIG_MFD_TC35892 is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_WM8994 is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_LPC_SCH is not set
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_REGULATOR is not set
# CONFIG_MEDIA_SUPPORT is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_SIS=y
CONFIG_AGP_VIA=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
# CONFIG_VGA_SWITCHEROO is not set
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m
CONFIG_DRM_TDFX=m
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
CONFIG_DRM_I810=m
CONFIG_DRM_I830=m
CONFIG_DRM_I915=m
# CONFIG_DRM_I915_KMS is not set
CONFIG_DRM_MGA=m
# CONFIG_DRM_SIS is not set
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_DDC=m
# CONFIG_FB_BOOT_VESA_SUPPORT is not set
CONFIG_FB_CFB_FILLRECT=m
CONFIG_FB_CFB_COPYAREA=m
CONFIG_FB_CFB_IMAGEBLIT=m
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
CONFIG_FB_SVGALIB=m
# CONFIG_FB_MACMODES is not set
CONFIG_FB_BACKLIGHT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
# CONFIG_FB_VESA is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
CONFIG_FB_NVIDIA=m
CONFIG_FB_NVIDIA_I2C=y
# CONFIG_FB_NVIDIA_DEBUG is not set
CONFIG_FB_NVIDIA_BACKLIGHT=y
CONFIG_FB_RIVA=m
# CONFIG_FB_RIVA_I2C is not set
# CONFIG_FB_RIVA_DEBUG is not set
CONFIG_FB_RIVA_BACKLIGHT=y
# CONFIG_FB_LE80578 is not set
CONFIG_FB_MATROX=m
CONFIG_FB_MATROX_MILLENIUM=y
CONFIG_FB_MATROX_MYSTIQUE=y
CONFIG_FB_MATROX_G=y
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_MATROX_MAVEN=m
# CONFIG_FB_RADEON is not set
CONFIG_FB_ATY128=m
CONFIG_FB_ATY128_BACKLIGHT=y
CONFIG_FB_ATY=m
CONFIG_FB_ATY_CT=y
CONFIG_FB_ATY_GENERIC_LCD=y
CONFIG_FB_ATY_GX=y
CONFIG_FB_ATY_BACKLIGHT=y
CONFIG_FB_S3=m
CONFIG_FB_SAVAGE=m
CONFIG_FB_SAVAGE_I2C=y
CONFIG_FB_SAVAGE_ACCEL=y
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
CONFIG_FB_NEOMAGIC=m
CONFIG_FB_KYRO=m
CONFIG_FB_3DFX=m
CONFIG_FB_3DFX_ACCEL=y
CONFIG_FB_3DFX_I2C=y
CONFIG_FB_VOODOO1=m
# CONFIG_FB_VT8623 is not set
CONFIG_FB_TRIDENT=m
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_GEODE is not set
CONFIG_FB_SM501=m
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
# CONFIG_LCD_PLATFORM is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=y
CONFIG_BACKLIGHT_PROGEAR=m
# CONFIG_BACKLIGHT_MBP_NVIDIA is not set
# CONFIG_BACKLIGHT_SAHARA is not set
# CONFIG_BACKLIGHT_ADP8860 is not set

#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=m
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_DYNAMIC_MINORS=y
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_OPL3_LIB_SEQ=m
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTS64=m
# CONFIG_SND_SERIAL_U16550 is not set
CONFIG_SND_MPU401=m
CONFIG_SND_PORTMAN2X4=m
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
CONFIG_SND_SB_COMMON=m
CONFIG_SND_PCI=y
CONFIG_SND_AD1889=m
CONFIG_SND_ALS300=m
CONFIG_SND_ALS4000=m
CONFIG_SND_ALI5451=m
# CONFIG_SND_ASIHPI is not set
CONFIG_SND_ATIIXP=m
CONFIG_SND_ATIIXP_MODEM=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
# CONFIG_SND_AW2 is not set
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CA0106=m
CONFIG_SND_CMIPCI=m
# CONFIG_SND_OXYGEN is not set
CONFIG_SND_CS4281=m
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_CTXFI is not set
CONFIG_SND_DARLA20=m
CONFIG_SND_GINA20=m
CONFIG_SND_LAYLA20=m
CONFIG_SND_DARLA24=m
CONFIG_SND_GINA24=m
CONFIG_SND_LAYLA24=m
CONFIG_SND_MONA=m
CONFIG_SND_MIA=m
CONFIG_SND_ECHO3G=m
CONFIG_SND_INDIGO=m
CONFIG_SND_INDIGOIO=m
CONFIG_SND_INDIGODJ=m
# CONFIG_SND_INDIGOIOX is not set
# CONFIG_SND_INDIGODJX is not set
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
# CONFIG_SND_ES1968_INPUT is not set
CONFIG_SND_FM801=m
CONFIG_SND_HDA_INTEL=m
# CONFIG_SND_HDA_HWDEP is not set
# CONFIG_SND_HDA_INPUT_BEEP is not set
# CONFIG_SND_HDA_INPUT_JACK is not set
# CONFIG_SND_HDA_PATCH_LOADER is not set
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_ATIHDMI=y
CONFIG_SND_HDA_CODEC_NVHDMI=y
CONFIG_SND_HDA_CODEC_INTELHDMI=y
CONFIG_SND_HDA_ELD=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CA0110=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
# CONFIG_SND_HDA_POWER_SAVE is not set
CONFIG_SND_HDSP=m
CONFIG_SND_HDSPM=m
# CONFIG_SND_HIFIER is not set
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_KORG1212=m
# CONFIG_SND_LX6464ES is not set
CONFIG_SND_MAESTRO3=m
# CONFIG_SND_MAESTRO3_INPUT is not set
CONFIG_SND_MIXART=m
CONFIG_SND_NM256=m
CONFIG_SND_PCXHR=m
CONFIG_SND_RIPTIDE=m
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_TRIDENT=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
# CONFIG_SND_VIRTUOSO is not set
CONFIG_SND_VX222=m
CONFIG_SND_YMFPCI=m
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=m
# CONFIG_SND_USB_UA101 is not set
CONFIG_SND_USB_USX2Y=m
# CONFIG_SND_USB_CAIAQ is not set
# CONFIG_SND_USB_US122L is not set
CONFIG_SND_PCMCIA=y
# CONFIG_SND_VXPOCKET is not set
# CONFIG_SND_PDAUDIOCF is not set
CONFIG_SND_SOC=m
CONFIG_SND_SOC_I2C_AND_SPI=m
# CONFIG_SND_SOC_ALL_CODECS is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=m
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HIDRAW is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y

#
# Special HID drivers
#
# CONFIG_HID_3M_PCT is not set
CONFIG_HID_A4TECH=y
# CONFIG_HID_ACRUX_FF is not set
CONFIG_HID_APPLE=y
CONFIG_HID_BELKIN=y
# CONFIG_HID_CANDO is not set
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
# CONFIG_HID_PRODIKEYS is not set
CONFIG_HID_CYPRESS=y
CONFIG_HID_DRAGONRISE=y
# CONFIG_DRAGONRISE_FF is not set
# CONFIG_HID_EGALAX is not set
# CONFIG_HID_ELECOM is not set
CONFIG_HID_EZKEY=y
CONFIG_HID_KYE=y
CONFIG_HID_GYRATION=y
CONFIG_HID_TWINHAN=y
CONFIG_HID_KENSINGTON=y
CONFIG_HID_LOGITECH=y
CONFIG_LOGITECH_FF=y
# CONFIG_LOGIRUMBLEPAD2_FF is not set
# CONFIG_LOGIG940_FF is not set
# CONFIG_HID_MAGICMOUSE is not set
CONFIG_HID_MICROSOFT=y
# CONFIG_HID_MOSART is not set
CONFIG_HID_MONTEREY=y
CONFIG_HID_NTRIG=y
CONFIG_HID_ORTEK=y
CONFIG_HID_PANTHERLORD=y
CONFIG_PANTHERLORD_FF=y
CONFIG_HID_PETALYNX=y
# CONFIG_HID_PICOLCD is not set
# CONFIG_HID_QUANTA is not set
# CONFIG_HID_ROCCAT is not set
# CONFIG_HID_ROCCAT_KONE is not set
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SONY=y
# CONFIG_HID_STANTUM is not set
CONFIG_HID_SUNPLUS=y
CONFIG_HID_GREENASIA=y
# CONFIG_GREENASIA_FF is not set
CONFIG_HID_SMARTJOYPLUS=y
# CONFIG_SMARTJOYPLUS_FF is not set
CONFIG_HID_TOPSEED=y
CONFIG_HID_THRUSTMASTER=y
CONFIG_THRUSTMASTER_FF=y
# CONFIG_HID_WACOM is not set
CONFIG_HID_ZEROPLUS=y
CONFIG_ZEROPLUS_FF=y
# CONFIG_HID_ZYDACRON is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
CONFIG_USB_DEVICE_CLASS=y
# CONFIG_USB_DYNAMIC_MINORS is not set
CONFIG_USB_MON=y
# CONFIG_USB_WUSB is not set
# CONFIG_USB_WUSB_CBAF is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
# CONFIG_USB_XHCI_HCD is not set
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_USB_OXU210HP_HCD is not set
CONFIG_USB_ISP116X_HCD=m
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_U132_HCD=m
CONFIG_USB_SL811_HCD=m
CONFIG_USB_SL811_CS=m
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_HWA_HCD is not set

#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_DATAFAB=m
CONFIG_USB_STORAGE_FREECOM=m
CONFIG_USB_STORAGE_ISD200=m
CONFIG_USB_STORAGE_USBAT=m
CONFIG_USB_STORAGE_SDDR09=m
CONFIG_USB_STORAGE_SDDR55=m
CONFIG_USB_STORAGE_JUMPSHOT=m
CONFIG_USB_STORAGE_ALAUDA=m
# CONFIG_USB_STORAGE_ONETOUCH is not set
CONFIG_USB_STORAGE_KARMA=m
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
CONFIG_USB_LIBUSUAL=y

#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m

#
# USB port drivers
#
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=m
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
# CONFIG_USB_SERIAL_CH341 is not set
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
# CONFIG_USB_SERIAL_CP210X is not set
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
# CONFIG_USB_SERIAL_IUU is not set
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KEYSPAN_MPR=y
CONFIG_USB_SERIAL_KEYSPAN_USA28=y
CONFIG_USB_SERIAL_KEYSPAN_USA28X=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y
CONFIG_USB_SERIAL_KEYSPAN_USA19=y
CONFIG_USB_SERIAL_KEYSPAN_USA18X=y
CONFIG_USB_SERIAL_KEYSPAN_USA19W=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QI=y
CONFIG_USB_SERIAL_KEYSPAN_USA49W=y
CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_MOS7720=m
# CONFIG_USB_SERIAL_MOS7715_PARPORT is not set
CONFIG_USB_SERIAL_MOS7840=m
# CONFIG_USB_SERIAL_MOTOROLA is not set
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_QCAUX is not set
# CONFIG_USB_SERIAL_QUALCOMM is not set
# CONFIG_USB_SERIAL_SPCP8X5 is not set
CONFIG_USB_SERIAL_HP4X=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SAFE_PADDED=y
# CONFIG_USB_SERIAL_SIEMENS_MPI is not set
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
# CONFIG_USB_SERIAL_SYMBOL is not set
CONFIG_USB_SERIAL_TI=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
# CONFIG_USB_SERIAL_OPTICON is not set
# CONFIG_USB_SERIAL_VIVOPAY_SERIAL is not set
# CONFIG_USB_SERIAL_ZIO is not set
# CONFIG_USB_SERIAL_SSU100 is not set
CONFIG_USB_SERIAL_DEBUG=m

#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_ADUTUX=m
# CONFIG_USB_SEVSEG is not set
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
CONFIG_USB_IDMOUSE=m
CONFIG_USB_FTDI_ELAN=m
CONFIG_USB_APPLEDISPLAY=m
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
CONFIG_USB_TRANCEVIBRATOR=m
CONFIG_USB_IOWARRIOR=m
CONFIG_USB_TEST=m
# CONFIG_USB_ISIGHTFW is not set
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m
# CONFIG_USB_GADGET is not set

#
# OTG and related infrastructure
#
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_UWB is not set
CONFIG_MMC=m
# CONFIG_MMC_DEBUG is not set
# CONFIG_MMC_UNSAFE_RESUME is not set

#
# MMC/SD/SDIO Card Drivers
#
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_BOUNCE=y
# CONFIG_SDIO_UART is not set
# CONFIG_MMC_TEST is not set

#
# MMC/SD/SDIO Host Controller Drivers
#
CONFIG_MMC_SDHCI=m
# CONFIG_MMC_SDHCI_PCI is not set
# CONFIG_MMC_SDHCI_PLTFM is not set
CONFIG_MMC_WBSD=m
CONFIG_MMC_TIFM_SD=m
# CONFIG_MMC_SDRICOH_CS is not set
# CONFIG_MMC_CB710 is not set
# CONFIG_MMC_VIA_SDMMC is not set
# CONFIG_MEMSTICK is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

#
# LED drivers
#
# CONFIG_LEDS_ALIX2 is not set
# CONFIG_LEDS_PCA9532 is not set
# CONFIG_LEDS_LP3944 is not set
# CONFIG_LEDS_CLEVO_MAIL is not set
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_BD2802 is not set
CONFIG_LEDS_TRIGGERS=y

#
# LED Triggers
#
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_HEARTBEAT=m
# CONFIG_LEDS_TRIGGER_BACKLIGHT is not set
# CONFIG_LEDS_TRIGGER_DEFAULT_ON is not set

#
# iptables trigger is under Netfilter config (LED target)
#
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_EDAC=y

#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_DECODE_MCE=y
CONFIG_EDAC_MM_EDAC=m
# CONFIG_EDAC_AMD64 is not set
CONFIG_EDAC_E752X=m
# CONFIG_EDAC_I82975X is not set
# CONFIG_EDAC_I3000 is not set
# CONFIG_EDAC_I3200 is not set
# CONFIG_EDAC_X38 is not set
# CONFIG_EDAC_I5400 is not set
# CONFIG_EDAC_I7CORE is not set
# CONFIG_EDAC_I5000 is not set
# CONFIG_EDAC_I5100 is not set
CONFIG_RTC_LIB=m
CONFIG_RTC_CLASS=m

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
CONFIG_RTC_DRV_DS1307=m
# CONFIG_RTC_DRV_DS1374 is not set
CONFIG_RTC_DRV_DS1672=m
# CONFIG_RTC_DRV_DS3232 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
# CONFIG_RTC_DRV_ISL12022 is not set
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8563=m
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_BQ32K is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
# CONFIG_RTC_DRV_RX8581 is not set
# CONFIG_RTC_DRV_RX8025 is not set

#
# SPI RTC drivers
#

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=m
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_DS1742=m
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_MSM6242 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_RP5C01 is not set
CONFIG_RTC_DRV_V3020=m

#
# on-CPU RTC drivers
#
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
# CONFIG_STAGING is not set
CONFIG_X86_PLATFORM_DEVICES=y
# CONFIG_ACER_WMI is not set
CONFIG_ASUS_LAPTOP=m
# CONFIG_DELL_LAPTOP is not set
# CONFIG_FUJITSU_LAPTOP is not set
CONFIG_MSI_LAPTOP=m
# CONFIG_PANASONIC_LAPTOP is not set
# CONFIG_COMPAL_LAPTOP is not set
CONFIG_SONY_LAPTOP=m
# CONFIG_SONYPI_COMPAT is not set
# CONFIG_IDEAPAD_ACPI is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_INTEL_MENLOW is not set
# CONFIG_EEEPC_LAPTOP is not set
# CONFIG_ACPI_WMI is not set
CONFIG_ACPI_ASUS=m
# CONFIG_TOPSTAR_LAPTOP is not set
CONFIG_ACPI_TOSHIBA=m
# CONFIG_TOSHIBA_BT_RFKILL is not set
# CONFIG_ACPI_CMPC is not set
# CONFIG_INTEL_IPS is not set

#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
CONFIG_DMIID=y
# CONFIG_ISCSI_IBFT_FIND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT2_FS_XIP=y
CONFIG_EXT3_FS=y
# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
# CONFIG_EXT4_FS is not set
CONFIG_FS_XIP=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=m
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
# CONFIG_JFS_DEBUG is not set
# CONFIG_JFS_STATISTICS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
# CONFIG_XFS_RT is not set
# CONFIG_XFS_DEBUG is not set
CONFIG_GFS2_FS=m
# CONFIG_GFS2_FS_LOCKING_DLM is not set
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
CONFIG_OCFS2_FS_STATS=y
# CONFIG_OCFS2_DEBUG_MASKLOG is not set
# CONFIG_OCFS2_DEBUG_FS is not set
# CONFIG_BTRFS_FS is not set
# CONFIG_NILFS2_FS is not set
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_FANOTIFY is not set
CONFIG_QUOTA=y
# CONFIG_QUOTA_NETLINK_INTERFACE is not set
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
# CONFIG_CUSE is not set
CONFIG_GENERIC_ACL=y

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="ascii"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
CONFIG_AFFS_FS=m
# CONFIG_ECRYPT_FS is not set
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
CONFIG_BEFS_FS=m
# CONFIG_BEFS_DEBUG is not set
CONFIG_BFS_FS=m
CONFIG_EFS_FS=m
# CONFIG_LOGFS is not set
CONFIG_CRAMFS=m
# CONFIG_SQUASHFS is not set
CONFIG_VXFS_FS=m
CONFIG_MINIX_FS=m
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
CONFIG_QNX4FS_FS=m
CONFIG_ROMFS_FS=m
CONFIG_ROMFS_BACKED_BY_BLOCK=y
CONFIG_ROMFS_ON_BLOCK=y
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
# CONFIG_UFS_FS_WRITE is not set
# CONFIG_UFS_DEBUG is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_RPCSEC_GSS_SPKM3=m
# CONFIG_SMB_FS is not set
# CONFIG_CEPH_FS is not set
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
CONFIG_CIFS_WEAK_PW_HASH=y
# CONFIG_CIFS_UPCALL is not set
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DFS_UPCALL is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
CONFIG_NCP_FS=m
CONFIG_NCPFS_PACKET_SIGNING=y
CONFIG_NCPFS_IOCTL_LOCKING=y
CONFIG_NCPFS_STRONG=y
CONFIG_NCPFS_NFS_NS=y
CONFIG_NCPFS_OS2_NS=y
CONFIG_NCPFS_SMALLDOS=y
CONFIG_NCPFS_NLS=y
CONFIG_NCPFS_EXTRAS=y
CONFIG_CODA_FS=m
# CONFIG_AFS_FS is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
CONFIG_DLM_DEBUG=y

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_LOCKUP_DETECTOR is not set
# CONFIG_HARDLOCKUP_DETECTOR is not set
CONFIG_DETECT_HUNG_TASK=y
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_LKDTM is not set
# CONFIG_CPU_NOTIFIER_ERROR_INJECT is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_SYSCTL_SYSCALL_CHECK=y
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
# CONFIG_FUNCTION_TRACER is not set
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
CONFIG_ENABLE_DEFAULT_TRACERS=y
# CONFIG_FTRACE_SYSCALLS is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
# CONFIG_STACK_TRACER is not set
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_KPROBE_EVENT is not set
# CONFIG_MMIOTRACE is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_ATOMIC64_SELFTEST is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
CONFIG_DEBUG_RODATA_TEST=y
# CONFIG_DEBUG_NX_TEST is not set
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
# CONFIG_X86_DECODER_SELFTEST is not set
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
CONFIG_OPTIMIZE_INLINING=y

#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
# CONFIG_SECURITY_NETWORK_XFRM is not set
# CONFIG_SECURITY_PATH is not set
CONFIG_LSM_MMAP_MIN_ADDR=65536
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
# CONFIG_SECURITY_SMACK is not set
# CONFIG_SECURITY_TOMOYO is not set
# CONFIG_SECURITY_APPARMOR is not set
# CONFIG_IMA is not set
CONFIG_DEFAULT_SECURITY_SELINUX=y
# CONFIG_DEFAULT_SECURITY_DAC is not set
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
# CONFIG_ASYNC_RAID6_TEST is not set
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_MANAGER_TESTS=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
# CONFIG_CRYPTO_PCRYPT is not set
CONFIG_CRYPTO_WORKQUEUE=y
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=m
# CONFIG_CRYPTO_TEST is not set

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
CONFIG_CRYPTO_CBC=m
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
# CONFIG_CRYPTO_XTS is not set

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
# CONFIG_CRYPTO_VMAC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CRC32C_INTEL is not set
# CONFIG_CRYPTO_GHASH is not set
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
# CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL is not set

#
# Ciphers
#
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_X86_64=m
# CONFIG_CRYPTO_AES_NI_INTEL is not set
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
# CONFIG_CRYPTO_SEED is not set
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
# CONFIG_CRYPTO_ZLIB is not set
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
CONFIG_CRYPTO_HW=y
# CONFIG_CRYPTO_DEV_PADLOCK is not set
# CONFIG_CRYPTO_DEV_HIFN_795X is not set
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
# CONFIG_VHOST_NET is not set
# CONFIG_VIRTIO_PCI is not set
# CONFIG_VIRTIO_BALLOON is not set
CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_DECOMPRESS=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_NLATTR=y

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Wed, Aug 25, 2010 at 01:00:06PM +0200, Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > > You might use the debug patch below for diagnostics.
> > 
> > Thanks, will try that and report back.
> 
> Here's a more detailed description of the regression introduced by:
> 
>   4a31beb: perf, x86: Fix handle_irq return values
>   8e3e42b: perf, x86: Try to handle unknown nmis with an enabled PMU
> 
> Booting into the debug kernel the system boots up fine - no NMI 
> messages, as expected.
> 
> Then when i start 'perf top' for the first time i get the NMI message 
> with this debug output:
> 
>  cpu #15, nmi #160, marked #0, handled = 1, time = 333392635730, delta = 11238255
>  cpu #15, nmi #161, marked #0, handled = 1, time = 333403779380, delta = 11143650
>  cpu #15, nmi #162, marked #0, handled = 1, time = 333415418497, delta = 11639117
>  cpu #15, nmi #163, marked #0, handled = 1, time = 333415467084, delta = 48587
>  cpu #15, nmi #164, marked #0, handled = 1, time = 333415501531, delta = 34447
>  cpu #15, nmi #165, marked #0, handled = 1, time = 333459918106, delta = 44416575
>  cpu #15, nmi #166, marked #0, handled = 0, time = 333459923167, delta = 1666
>  cpu #15, nmi #151, marked #0, handled = 1, time = 332978597882, delta = 11447002
>  cpu #15, nmi #152, marked #0, handled = 1, time = 332978657151, delta = 59269
>  cpu #15, nmi #153, marked #0, handled = 1, time = 332978667847, delta = 10696
>  cpu #15, nmi #154, marked #0, handled = 1, time = 333023125757, delta = 44457910
>  cpu #15, nmi #155, marked #0, handled = 1, time = 333291980833, delta = 268855076
>  cpu #15, nmi #156, marked #0, handled = 1, time = 333325663125, delta = 33682292
>  cpu #15, nmi #157, marked #0, handled = 1, time = 333348216481, delta = 22553356
>  cpu #15, nmi #158, marked #0, handled = 1, time = 333370168887, delta = 21952406
>  cpu #15, nmi #159, marked #0, handled = 1, time = 333381397475, delta = 11228588
>  Uhhuh. NMI received for unknown reason 00 on CPU 15.
>  Do you have a strange power saving mode enabled?
>  Dazed and confused, but trying to continue

So I found a Nehalem box that can reliably reproduce Ingo's problem using
something as simple 'perf top'.  But like above, I am noticing the
samething, an extra NMI(PMI??) that comes out of nowhere.

Looking at the data above the delta between nmis is very small compared to
the other nmis.  It almost suggests that this is an extra PMI.
Considering there is already two cpu errata discussing extra PMIs under
certain configurations, I wouldn't be surprised if this was a third.

Cheers,
Don

> 
> When i start perf top for a second time, no messages are printed at all. 
> The reason is that on one of the CPUs NMIs are 'stuck':
> 
>  NMI: 78164 67099 6342 [*] 65677 66119 63796 65395 63995 65012 64151 65082 
>       63483 64948 62926 65608 62630
> 
> CPU#2 is stuck at 6342.
> 
> The NMIs work fine on other CPUs and perf top works (sans the missing 
> samples from CPU#2@@), and the NMIs keep ticking.
> 
> The CPU is:
> 
>  processor	: 2
>  vendor_id	: GenuineIntel
>  cpu family	: 6
>  model		: 26
>  model name	: Intel(R) Xeon(R) CPU           X55600 @ 2.80GHz
>  stepping	: 5
>  cpu MHz	: 2794.000
>  cache size	: 8192 KB
>  physical id	: 0
>  siblings	: 8
>  core id	: 1
>  cpu cores	: 4
>  apicid		: 2
>  initial apicid	: 2
>  fpu		: yes
>  fpu_exception	: yes
>  cpuid level	: 11
>  wp		: yes
>  flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
>  bogomips	: 5599.98
>  clflush size	: 64
>  cache_alignment: 64
>  address sizes	: 40 bits physical, 48 bits virtual
>  power management:
> 
> The PMU init is:
> 
>  Performance Events: PEBS fmt1+, Nehalem events, Intel PMU driver.
>  ... version:                3
>  ... bit width:              48
>  ... generic registers:      4
>  ... value mask:             0000ffffffffffff
>  ... max period:             000000007fffffff
>  ... fixed-purpose events:   3
>  ... event mask:             000000070000000f
> 
> I've attached the config as well.
> 
> Thanks,
> 
> 	Ingo

> #
> # Automatically generated make config: don't edit
> # Linux kernel version: 2.6.36-rc2
> # Thu Aug 26 07:41:31 2010
> #
> CONFIG_64BIT=y
> # CONFIG_X86_32 is not set
> CONFIG_X86_64=y
> CONFIG_X86=y
> CONFIG_INSTRUCTION_DECODER=y
> CONFIG_OUTPUT_FORMAT="elf64-x86-64"
> CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
> CONFIG_GENERIC_CMOS_UPDATE=y
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_LATENCYTOP_SUPPORT=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_NEED_DMA_MAP_STATE=y
> CONFIG_NEED_SG_DMA_LENGTH=y
> CONFIG_GENERIC_ISA_DMA=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_BUG=y
> CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
> CONFIG_GENERIC_HWEIGHT=y
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> # CONFIG_RWSEM_GENERIC_SPINLOCK is not set
> CONFIG_RWSEM_XCHGADD_ALGORITHM=y
> CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
> CONFIG_GENERIC_CALIBRATE_DELAY=y
> CONFIG_GENERIC_TIME_VSYSCALL=y
> CONFIG_ARCH_HAS_CPU_RELAX=y
> CONFIG_ARCH_HAS_DEFAULT_IDLE=y
> CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
> CONFIG_HAVE_SETUP_PER_CPU_AREA=y
> CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
> CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
> CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
> CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_SUSPEND_POSSIBLE=y
> CONFIG_ZONE_DMA32=y
> CONFIG_ARCH_POPULATES_NODE_MAP=y
> CONFIG_AUDIT_ARCH=y
> CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> CONFIG_HAVE_EARLY_RES=y
> CONFIG_GENERIC_HARDIRQS=y
> CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
> CONFIG_GENERIC_IRQ_PROBE=y
> CONFIG_GENERIC_PENDING_IRQ=y
> CONFIG_USE_GENERIC_SMP_HELPERS=y
> CONFIG_X86_64_SMP=y
> CONFIG_X86_HT=y
> CONFIG_X86_TRAMPOLINE=y
> CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
> # CONFIG_KTIME_SCALAR is not set
> CONFIG_ARCH_CPU_PROBE_RELEASE=y
> CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
> CONFIG_CONSTRUCTORS=y
> 
> #
> # General setup
> #
> CONFIG_EXPERIMENTAL=y
> CONFIG_LOCK_KERNEL=y
> CONFIG_INIT_ENV_ARG_LIMIT=32
> CONFIG_CROSS_COMPILE=""
> CONFIG_LOCALVERSION=""
> CONFIG_LOCALVERSION_AUTO=y
> CONFIG_HAVE_KERNEL_GZIP=y
> CONFIG_HAVE_KERNEL_BZIP2=y
> CONFIG_HAVE_KERNEL_LZMA=y
> CONFIG_HAVE_KERNEL_LZO=y
> CONFIG_KERNEL_GZIP=y
> # CONFIG_KERNEL_BZIP2 is not set
> # CONFIG_KERNEL_LZMA is not set
> # CONFIG_KERNEL_LZO is not set
> CONFIG_SWAP=y
> CONFIG_SYSVIPC=y
> CONFIG_SYSVIPC_SYSCTL=y
> CONFIG_POSIX_MQUEUE=y
> CONFIG_POSIX_MQUEUE_SYSCTL=y
> CONFIG_BSD_PROCESS_ACCT=y
> # CONFIG_BSD_PROCESS_ACCT_V3 is not set
> CONFIG_TASKSTATS=y
> CONFIG_TASK_DELAY_ACCT=y
> # CONFIG_TASK_XACCT is not set
> CONFIG_AUDIT=y
> CONFIG_AUDITSYSCALL=y
> CONFIG_AUDIT_WATCH=y
> CONFIG_AUDIT_TREE=y
> 
> #
> # RCU Subsystem
> #
> CONFIG_TREE_RCU=y
> # CONFIG_PREEMPT_RCU is not set
> # CONFIG_RCU_TRACE is not set
> CONFIG_RCU_FANOUT=64
> # CONFIG_RCU_FANOUT_EXACT is not set
> # CONFIG_TREE_RCU_TRACE is not set
> # CONFIG_IKCONFIG is not set
> CONFIG_LOG_BUF_SHIFT=20
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> # CONFIG_CGROUPS is not set
> CONFIG_SYSFS_DEPRECATED=y
> CONFIG_SYSFS_DEPRECATED_V2=y
> CONFIG_RELAY=y
> CONFIG_NAMESPACES=y
> # CONFIG_UTS_NS is not set
> # CONFIG_IPC_NS is not set
> # CONFIG_USER_NS is not set
> # CONFIG_PID_NS is not set
> # CONFIG_NET_NS is not set
> CONFIG_BLK_DEV_INITRD=y
> CONFIG_INITRAMFS_SOURCE=""
> CONFIG_RD_GZIP=y
> CONFIG_RD_BZIP2=y
> CONFIG_RD_LZMA=y
> CONFIG_RD_LZO=y
> CONFIG_CC_OPTIMIZE_FOR_SIZE=y
> CONFIG_SYSCTL=y
> CONFIG_ANON_INODES=y
> # CONFIG_EMBEDDED is not set
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> CONFIG_KALLSYMS_ALL=y
> CONFIG_KALLSYMS_EXTRA_PASS=y
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_PCSPKR_PLATFORM=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_AIO=y
> CONFIG_HAVE_PERF_EVENTS=y
> 
> #
> # Kernel Performance Events And Counters
> #
> CONFIG_PERF_EVENTS=y
> CONFIG_PERF_COUNTERS=y
> # CONFIG_DEBUG_PERF_USE_VMALLOC is not set
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_PCI_QUIRKS=y
> CONFIG_COMPAT_BRK=y
> CONFIG_SLAB=y
> # CONFIG_SLUB is not set
> CONFIG_PROFILING=y
> CONFIG_TRACEPOINTS=y
> CONFIG_OPROFILE=m
> # CONFIG_OPROFILE_EVENT_MULTIPLEX is not set
> CONFIG_HAVE_OPROFILE=y
> CONFIG_KPROBES=y
> CONFIG_OPTPROBES=y
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
> CONFIG_KRETPROBES=y
> CONFIG_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_IOREMAP_PROT=y
> CONFIG_HAVE_KPROBES=y
> CONFIG_HAVE_KRETPROBES=y
> CONFIG_HAVE_OPTPROBES=y
> CONFIG_HAVE_ARCH_TRACEHOOK=y
> CONFIG_HAVE_DMA_ATTRS=y
> CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
> CONFIG_HAVE_DMA_API_DEBUG=y
> CONFIG_HAVE_HW_BREAKPOINT=y
> CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
> CONFIG_HAVE_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_PERF_EVENTS_NMI=y
> 
> #
> # GCOV-based kernel profiling
> #
> # CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> # CONFIG_MODULE_FORCE_LOAD is not set
> CONFIG_MODULE_UNLOAD=y
> # CONFIG_MODULE_FORCE_UNLOAD is not set
> CONFIG_MODVERSIONS=y
> CONFIG_MODULE_SRCVERSION_ALL=y
> CONFIG_STOP_MACHINE=y
> CONFIG_BLOCK=y
> # CONFIG_BLK_DEV_BSG is not set
> # CONFIG_BLK_DEV_INTEGRITY is not set
> CONFIG_BLOCK_COMPAT=y
> 
> #
> # IO Schedulers
> #
> CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> # CONFIG_DEFAULT_DEADLINE is not set
> CONFIG_DEFAULT_CFQ=y
> # CONFIG_DEFAULT_NOOP is not set
> CONFIG_DEFAULT_IOSCHED="cfq"
> CONFIG_PREEMPT_NOTIFIERS=y
> # CONFIG_INLINE_SPIN_TRYLOCK is not set
> # CONFIG_INLINE_SPIN_TRYLOCK_BH is not set
> # CONFIG_INLINE_SPIN_LOCK is not set
> # CONFIG_INLINE_SPIN_LOCK_BH is not set
> # CONFIG_INLINE_SPIN_LOCK_IRQ is not set
> # CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
> CONFIG_INLINE_SPIN_UNLOCK=y
> # CONFIG_INLINE_SPIN_UNLOCK_BH is not set
> CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
> # CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
> # CONFIG_INLINE_READ_TRYLOCK is not set
> # CONFIG_INLINE_READ_LOCK is not set
> # CONFIG_INLINE_READ_LOCK_BH is not set
> # CONFIG_INLINE_READ_LOCK_IRQ is not set
> # CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
> CONFIG_INLINE_READ_UNLOCK=y
> # CONFIG_INLINE_READ_UNLOCK_BH is not set
> CONFIG_INLINE_READ_UNLOCK_IRQ=y
> # CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
> # CONFIG_INLINE_WRITE_TRYLOCK is not set
> # CONFIG_INLINE_WRITE_LOCK is not set
> # CONFIG_INLINE_WRITE_LOCK_BH is not set
> # CONFIG_INLINE_WRITE_LOCK_IRQ is not set
> # CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
> CONFIG_INLINE_WRITE_UNLOCK=y
> # CONFIG_INLINE_WRITE_UNLOCK_BH is not set
> CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
> # CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
> CONFIG_MUTEX_SPIN_ON_OWNER=y
> CONFIG_FREEZER=y
> 
> #
> # Processor type and features
> #
> # CONFIG_NO_HZ is not set
> # CONFIG_HIGH_RES_TIMERS is not set
> CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
> CONFIG_SMP=y
> # CONFIG_SPARSE_IRQ is not set
> CONFIG_X86_MPPARSE=y
> CONFIG_X86_EXTENDED_PLATFORM=y
> # CONFIG_X86_VSMP is not set
> CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> # CONFIG_PARAVIRT_GUEST is not set
> CONFIG_NO_BOOTMEM=y
> # CONFIG_MEMTEST is not set
> # CONFIG_MK8 is not set
> # CONFIG_MPSC is not set
> # CONFIG_MCORE2 is not set
> # CONFIG_MATOM is not set
> CONFIG_GENERIC_CPU=y
> CONFIG_X86_CPU=y
> CONFIG_X86_INTERNODE_CACHE_SHIFT=7
> CONFIG_X86_CMPXCHG=y
> CONFIG_X86_L1_CACHE_SHIFT=6
> CONFIG_X86_XADD=y
> CONFIG_X86_WP_WORKS_OK=y
> CONFIG_X86_TSC=y
> CONFIG_X86_CMPXCHG64=y
> CONFIG_X86_CMOV=y
> CONFIG_X86_MINIMUM_CPU_FAMILY=64
> CONFIG_X86_DEBUGCTLMSR=y
> CONFIG_CPU_SUP_INTEL=y
> CONFIG_CPU_SUP_AMD=y
> CONFIG_CPU_SUP_CENTAUR=y
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> CONFIG_DMI=y
> CONFIG_GART_IOMMU=y
> CONFIG_CALGARY_IOMMU=y
> CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
> # CONFIG_AMD_IOMMU is not set
> CONFIG_SWIOTLB=y
> CONFIG_IOMMU_HELPER=y
> # CONFIG_IOMMU_API is not set
> # CONFIG_MAXSMP is not set
> CONFIG_NR_CPUS=255
> CONFIG_SCHED_SMT=y
> CONFIG_SCHED_MC=y
> CONFIG_PREEMPT_NONE=y
> # CONFIG_PREEMPT_VOLUNTARY is not set
> # CONFIG_PREEMPT is not set
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
> CONFIG_X86_MCE=y
> CONFIG_X86_MCE_INTEL=y
> CONFIG_X86_MCE_AMD=y
> CONFIG_X86_MCE_THRESHOLD=y
> # CONFIG_X86_MCE_INJECT is not set
> CONFIG_X86_THERMAL_VECTOR=y
> # CONFIG_I8K is not set
> CONFIG_MICROCODE=m
> CONFIG_MICROCODE_INTEL=y
> # CONFIG_MICROCODE_AMD is not set
> CONFIG_MICROCODE_OLD_INTERFACE=y
> CONFIG_X86_MSR=y
> CONFIG_X86_CPUID=y
> CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
> CONFIG_DIRECT_GBPAGES=y
> CONFIG_NUMA=y
> CONFIG_K8_NUMA=y
> CONFIG_X86_64_ACPI_NUMA=y
> CONFIG_NODES_SPAN_OTHER_NODES=y
> # CONFIG_NUMA_EMU is not set
> CONFIG_NODES_SHIFT=6
> CONFIG_ARCH_PROC_KCORE_TEXT=y
> CONFIG_ARCH_SPARSEMEM_DEFAULT=y
> CONFIG_ARCH_SPARSEMEM_ENABLE=y
> CONFIG_ARCH_SELECT_MEMORY_MODEL=y
> CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
> CONFIG_SELECT_MEMORY_MODEL=y
> CONFIG_SPARSEMEM_MANUAL=y
> CONFIG_SPARSEMEM=y
> CONFIG_NEED_MULTIPLE_NODES=y
> CONFIG_HAVE_MEMORY_PRESENT=y
> CONFIG_SPARSEMEM_EXTREME=y
> CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
> CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
> CONFIG_SPARSEMEM_VMEMMAP=y
> # CONFIG_MEMORY_HOTPLUG is not set
> CONFIG_PAGEFLAGS_EXTENDED=y
> CONFIG_SPLIT_PTLOCK_CPUS=4
> # CONFIG_COMPACTION is not set
> CONFIG_MIGRATION=y
> CONFIG_PHYS_ADDR_T_64BIT=y
> CONFIG_ZONE_DMA_FLAG=1
> CONFIG_BOUNCE=y
> CONFIG_VIRT_TO_BUS=y
> CONFIG_MMU_NOTIFIER=y
> # CONFIG_KSM is not set
> CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
> CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
> # CONFIG_MEMORY_FAILURE is not set
> CONFIG_X86_CHECK_BIOS_CORRUPTION=y
> CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
> CONFIG_X86_LOW_RESERVE=64
> CONFIG_MTRR=y
> # CONFIG_MTRR_SANITIZER is not set
> CONFIG_X86_PAT=y
> CONFIG_ARCH_USES_PG_UNCACHED=y
> # CONFIG_EFI is not set
> # CONFIG_SECCOMP is not set
> # CONFIG_CC_STACKPROTECTOR is not set
> # CONFIG_HZ_100 is not set
> CONFIG_HZ_250=y
> # CONFIG_HZ_300 is not set
> # CONFIG_HZ_1000 is not set
> CONFIG_HZ=250
> # CONFIG_SCHED_HRTICK is not set
> CONFIG_KEXEC=y
> # CONFIG_CRASH_DUMP is not set
> CONFIG_PHYSICAL_START=0x1000000
> # CONFIG_RELOCATABLE is not set
> CONFIG_PHYSICAL_ALIGN=0x1000000
> CONFIG_HOTPLUG_CPU=y
> CONFIG_COMPAT_VDSO=y
> # CONFIG_CMDLINE_BOOL is not set
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID=y
> CONFIG_USE_PERCPU_NUMA_NODE_ID=y
> 
> #
> # Power management and ACPI options
> #
> CONFIG_PM=y
> # CONFIG_PM_DEBUG is not set
> CONFIG_PM_SLEEP_SMP=y
> CONFIG_PM_SLEEP=y
> CONFIG_SUSPEND_NVS=y
> CONFIG_SUSPEND=y
> CONFIG_SUSPEND_FREEZER=y
> # CONFIG_HIBERNATION is not set
> # CONFIG_PM_RUNTIME is not set
> CONFIG_PM_OPS=y
> CONFIG_ACPI=y
> CONFIG_ACPI_SLEEP=y
> CONFIG_ACPI_PROCFS=y
> CONFIG_ACPI_PROCFS_POWER=y
> # CONFIG_ACPI_POWER_METER is not set
> CONFIG_ACPI_SYSFS_POWER=y
> # CONFIG_ACPI_EC_DEBUGFS is not set
> CONFIG_ACPI_PROC_EVENT=y
> CONFIG_ACPI_AC=m
> CONFIG_ACPI_BATTERY=m
> CONFIG_ACPI_BUTTON=m
> CONFIG_ACPI_VIDEO=m
> CONFIG_ACPI_FAN=y
> CONFIG_ACPI_DOCK=y
> CONFIG_ACPI_PROCESSOR=y
> CONFIG_ACPI_HOTPLUG_CPU=y
> # CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
> CONFIG_ACPI_THERMAL=y
> CONFIG_ACPI_NUMA=y
> # CONFIG_ACPI_CUSTOM_DSDT is not set
> CONFIG_ACPI_BLACKLIST_YEAR=0
> # CONFIG_ACPI_DEBUG is not set
> # CONFIG_ACPI_PCI_SLOT is not set
> CONFIG_X86_PM_TIMER=y
> CONFIG_ACPI_CONTAINER=y
> CONFIG_ACPI_SBS=m
> # CONFIG_ACPI_HED is not set
> # CONFIG_ACPI_APEI is not set
> # CONFIG_SFI is not set
> 
> #
> # CPU Frequency scaling
> #
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_TABLE=y
> CONFIG_CPU_FREQ_DEBUG=y
> CONFIG_CPU_FREQ_STAT=m
> CONFIG_CPU_FREQ_STAT_DETAILS=y
> # CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
> CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
> # CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
> # CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> CONFIG_CPU_FREQ_GOV_POWERSAVE=m
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> CONFIG_CPU_FREQ_GOV_ONDEMAND=m
> CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m
> 
> #
> # CPUFreq processor drivers
> #
> # CONFIG_X86_PCC_CPUFREQ is not set
> CONFIG_X86_ACPI_CPUFREQ=y
> CONFIG_X86_POWERNOW_K8=y
> # CONFIG_X86_SPEEDSTEP_CENTRINO is not set
> # CONFIG_X86_P4_CLOCKMOD is not set
> 
> #
> # shared options
> #
> # CONFIG_X86_SPEEDSTEP_LIB is not set
> CONFIG_CPU_IDLE=y
> CONFIG_CPU_IDLE_GOV_LADDER=y
> # CONFIG_INTEL_IDLE is not set
> 
> #
> # Memory power savings
> #
> # CONFIG_I7300_IDLE is not set
> 
> #
> # Bus options (PCI etc.)
> #
> CONFIG_PCI=y
> CONFIG_PCI_DIRECT=y
> CONFIG_PCI_MMCONFIG=y
> CONFIG_PCI_DOMAINS=y
> # CONFIG_PCI_CNB20LE_QUIRK is not set
> CONFIG_PCIEPORTBUS=y
> CONFIG_HOTPLUG_PCI_PCIE=m
> CONFIG_PCIEAER=y
> # CONFIG_PCIE_ECRC is not set
> # CONFIG_PCIEAER_INJECT is not set
> CONFIG_PCIEASPM=y
> # CONFIG_PCIEASPM_DEBUG is not set
> CONFIG_ARCH_SUPPORTS_MSI=y
> # CONFIG_PCI_MSI is not set
> # CONFIG_PCI_DEBUG is not set
> # CONFIG_PCI_STUB is not set
> CONFIG_HT_IRQ=y
> # CONFIG_PCI_IOV is not set
> CONFIG_PCI_IOAPIC=y
> CONFIG_ISA_DMA_API=y
> CONFIG_K8_NB=y
> CONFIG_PCCARD=y
> CONFIG_PCMCIA=y
> CONFIG_PCMCIA_LOAD_CIS=y
> CONFIG_CARDBUS=y
> 
> #
> # PC-card bridges
> #
> CONFIG_YENTA=y
> CONFIG_YENTA_O2=y
> CONFIG_YENTA_RICOH=y
> CONFIG_YENTA_TI=y
> CONFIG_YENTA_ENE_TUNE=y
> CONFIG_YENTA_TOSHIBA=y
> CONFIG_PD6729=m
> CONFIG_I82092=m
> CONFIG_PCCARD_NONSTATIC=y
> CONFIG_HOTPLUG_PCI=y
> CONFIG_HOTPLUG_PCI_FAKE=m
> CONFIG_HOTPLUG_PCI_ACPI=m
> CONFIG_HOTPLUG_PCI_ACPI_IBM=m
> # CONFIG_HOTPLUG_PCI_CPCI is not set
> CONFIG_HOTPLUG_PCI_SHPC=m
> 
> #
> # Executable file formats / Emulations
> #
> CONFIG_BINFMT_ELF=y
> CONFIG_COMPAT_BINFMT_ELF=y
> # CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
> # CONFIG_HAVE_AOUT is not set
> CONFIG_BINFMT_MISC=y
> CONFIG_IA32_EMULATION=y
> # CONFIG_IA32_AOUT is not set
> CONFIG_COMPAT=y
> CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
> CONFIG_SYSVIPC_COMPAT=y
> CONFIG_NET=y
> 
> #
> # Networking options
> #
> CONFIG_PACKET=y
> CONFIG_UNIX=y
> CONFIG_XFRM=y
> CONFIG_XFRM_USER=y
> # CONFIG_XFRM_SUB_POLICY is not set
> CONFIG_XFRM_MIGRATE=y
> # CONFIG_XFRM_STATISTICS is not set
> CONFIG_XFRM_IPCOMP=m
> CONFIG_NET_KEY=m
> CONFIG_NET_KEY_MIGRATE=y
> CONFIG_INET=y
> CONFIG_IP_MULTICAST=y
> CONFIG_IP_ADVANCED_ROUTER=y
> CONFIG_ASK_IP_FIB_HASH=y
> # CONFIG_IP_FIB_TRIE is not set
> CONFIG_IP_FIB_HASH=y
> CONFIG_IP_MULTIPLE_TABLES=y
> CONFIG_IP_ROUTE_MULTIPATH=y
> CONFIG_IP_ROUTE_VERBOSE=y
> # CONFIG_IP_PNP is not set
> CONFIG_NET_IPIP=m
> CONFIG_NET_IPGRE=m
> CONFIG_NET_IPGRE_BROADCAST=y
> CONFIG_IP_MROUTE=y
> # CONFIG_IP_MROUTE_MULTIPLE_TABLES is not set
> CONFIG_IP_PIMSM_V1=y
> CONFIG_IP_PIMSM_V2=y
> # CONFIG_ARPD is not set
> CONFIG_SYN_COOKIES=y
> CONFIG_INET_AH=m
> CONFIG_INET_ESP=m
> CONFIG_INET_IPCOMP=m
> CONFIG_INET_XFRM_TUNNEL=m
> CONFIG_INET_TUNNEL=m
> CONFIG_INET_XFRM_MODE_TRANSPORT=m
> CONFIG_INET_XFRM_MODE_TUNNEL=m
> CONFIG_INET_XFRM_MODE_BEET=m
> CONFIG_INET_LRO=y
> CONFIG_INET_DIAG=m
> CONFIG_INET_TCP_DIAG=m
> CONFIG_TCP_CONG_ADVANCED=y
> CONFIG_TCP_CONG_BIC=y
> CONFIG_TCP_CONG_CUBIC=m
> CONFIG_TCP_CONG_WESTWOOD=m
> CONFIG_TCP_CONG_HTCP=m
> CONFIG_TCP_CONG_HSTCP=m
> CONFIG_TCP_CONG_HYBLA=m
> CONFIG_TCP_CONG_VEGAS=m
> CONFIG_TCP_CONG_SCALABLE=m
> CONFIG_TCP_CONG_LP=m
> CONFIG_TCP_CONG_VENO=m
> # CONFIG_TCP_CONG_YEAH is not set
> # CONFIG_TCP_CONG_ILLINOIS is not set
> CONFIG_DEFAULT_BIC=y
> # CONFIG_DEFAULT_RENO is not set
> CONFIG_DEFAULT_TCP_CONG="bic"
> CONFIG_TCP_MD5SIG=y
> CONFIG_IPV6=m
> CONFIG_IPV6_PRIVACY=y
> CONFIG_IPV6_ROUTER_PREF=y
> CONFIG_IPV6_ROUTE_INFO=y
> # CONFIG_IPV6_OPTIMISTIC_DAD is not set
> CONFIG_INET6_AH=m
> CONFIG_INET6_ESP=m
> CONFIG_INET6_IPCOMP=m
> # CONFIG_IPV6_MIP6 is not set
> CONFIG_INET6_XFRM_TUNNEL=m
> CONFIG_INET6_TUNNEL=m
> CONFIG_INET6_XFRM_MODE_TRANSPORT=m
> CONFIG_INET6_XFRM_MODE_TUNNEL=m
> CONFIG_INET6_XFRM_MODE_BEET=m
> CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
> CONFIG_IPV6_SIT=m
> # CONFIG_IPV6_SIT_6RD is not set
> CONFIG_IPV6_NDISC_NODETYPE=y
> CONFIG_IPV6_TUNNEL=m
> # CONFIG_IPV6_MULTIPLE_TABLES is not set
> # CONFIG_IPV6_MROUTE is not set
> CONFIG_NETLABEL=y
> CONFIG_NETWORK_SECMARK=y
> # CONFIG_NETWORK_PHY_TIMESTAMPING is not set
> CONFIG_NETFILTER=y
> CONFIG_NETFILTER_DEBUG=y
> CONFIG_NETFILTER_ADVANCED=y
> CONFIG_BRIDGE_NETFILTER=y
> 
> #
> # Core Netfilter Configuration
> #
> CONFIG_NETFILTER_NETLINK=m
> CONFIG_NETFILTER_NETLINK_QUEUE=m
> CONFIG_NETFILTER_NETLINK_LOG=m
> CONFIG_NF_CONNTRACK=y
> CONFIG_NF_CONNTRACK_MARK=y
> CONFIG_NF_CONNTRACK_SECMARK=y
> CONFIG_NF_CONNTRACK_EVENTS=y
> CONFIG_NF_CT_PROTO_DCCP=m
> CONFIG_NF_CT_PROTO_GRE=m
> CONFIG_NF_CT_PROTO_SCTP=m
> # CONFIG_NF_CT_PROTO_UDPLITE is not set
> CONFIG_NF_CONNTRACK_AMANDA=m
> CONFIG_NF_CONNTRACK_FTP=m
> CONFIG_NF_CONNTRACK_H323=m
> CONFIG_NF_CONNTRACK_IRC=m
> CONFIG_NF_CONNTRACK_NETBIOS_NS=m
> CONFIG_NF_CONNTRACK_PPTP=m
> CONFIG_NF_CONNTRACK_SANE=m
> CONFIG_NF_CONNTRACK_SIP=m
> CONFIG_NF_CONNTRACK_TFTP=m
> # CONFIG_NF_CT_NETLINK is not set
> # CONFIG_NETFILTER_TPROXY is not set
> CONFIG_NETFILTER_XTABLES=m
> 
> #
> # Xtables combined modules
> #
> CONFIG_NETFILTER_XT_MARK=m
> CONFIG_NETFILTER_XT_CONNMARK=m
> 
> #
> # Xtables targets
> #
> # CONFIG_NETFILTER_XT_TARGET_CHECKSUM is not set
> CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
> CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
> CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
> # CONFIG_NETFILTER_XT_TARGET_CT is not set
> CONFIG_NETFILTER_XT_TARGET_DSCP=m
> CONFIG_NETFILTER_XT_TARGET_HL=m
> # CONFIG_NETFILTER_XT_TARGET_IDLETIMER is not set
> # CONFIG_NETFILTER_XT_TARGET_LED is not set
> CONFIG_NETFILTER_XT_TARGET_MARK=m
> CONFIG_NETFILTER_XT_TARGET_NFLOG=m
> CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
> CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
> # CONFIG_NETFILTER_XT_TARGET_RATEEST is not set
> # CONFIG_NETFILTER_XT_TARGET_TEE is not set
> # CONFIG_NETFILTER_XT_TARGET_TRACE is not set
> CONFIG_NETFILTER_XT_TARGET_SECMARK=m
> CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
> # CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP is not set
> 
> #
> # Xtables matches
> #
> # CONFIG_NETFILTER_XT_MATCH_CLUSTER is not set
> CONFIG_NETFILTER_XT_MATCH_COMMENT=m
> CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
> # CONFIG_NETFILTER_XT_MATCH_CONNLIMIT is not set
> CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
> CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
> # CONFIG_NETFILTER_XT_MATCH_CPU is not set
> CONFIG_NETFILTER_XT_MATCH_DCCP=m
> CONFIG_NETFILTER_XT_MATCH_DSCP=m
> CONFIG_NETFILTER_XT_MATCH_ESP=m
> CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_HELPER=m
> CONFIG_NETFILTER_XT_MATCH_HL=m
> # CONFIG_NETFILTER_XT_MATCH_IPRANGE is not set
> # CONFIG_NETFILTER_XT_MATCH_IPVS is not set
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m
> CONFIG_NETFILTER_XT_MATCH_LIMIT=m
> CONFIG_NETFILTER_XT_MATCH_MAC=m
> CONFIG_NETFILTER_XT_MATCH_MARK=m
> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
> # CONFIG_NETFILTER_XT_MATCH_OSF is not set
> # CONFIG_NETFILTER_XT_MATCH_OWNER is not set
> CONFIG_NETFILTER_XT_MATCH_POLICY=m
> CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
> CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
> CONFIG_NETFILTER_XT_MATCH_QUOTA=m
> # CONFIG_NETFILTER_XT_MATCH_RATEEST is not set
> CONFIG_NETFILTER_XT_MATCH_REALM=m
> # CONFIG_NETFILTER_XT_MATCH_RECENT is not set
> CONFIG_NETFILTER_XT_MATCH_SCTP=m
> CONFIG_NETFILTER_XT_MATCH_STATE=m
> CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
> CONFIG_NETFILTER_XT_MATCH_STRING=m
> CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
> # CONFIG_NETFILTER_XT_MATCH_TIME is not set
> # CONFIG_NETFILTER_XT_MATCH_U32 is not set
> CONFIG_IP_VS=m
> # CONFIG_IP_VS_IPV6 is not set
> # CONFIG_IP_VS_DEBUG is not set
> CONFIG_IP_VS_TAB_BITS=12
> 
> #
> # IPVS transport protocol load balancing support
> #
> CONFIG_IP_VS_PROTO_TCP=y
> CONFIG_IP_VS_PROTO_UDP=y
> CONFIG_IP_VS_PROTO_AH_ESP=y
> CONFIG_IP_VS_PROTO_ESP=y
> CONFIG_IP_VS_PROTO_AH=y
> # CONFIG_IP_VS_PROTO_SCTP is not set
> 
> #
> # IPVS scheduler
> #
> CONFIG_IP_VS_RR=m
> CONFIG_IP_VS_WRR=m
> CONFIG_IP_VS_LC=m
> CONFIG_IP_VS_WLC=m
> CONFIG_IP_VS_LBLC=m
> CONFIG_IP_VS_LBLCR=m
> CONFIG_IP_VS_DH=m
> CONFIG_IP_VS_SH=m
> CONFIG_IP_VS_SED=m
> CONFIG_IP_VS_NQ=m
> 
> #
> # IPVS application helper
> #
> CONFIG_IP_VS_FTP=m
> 
> #
> # IP: Netfilter Configuration
> #
> CONFIG_NF_DEFRAG_IPV4=m
> CONFIG_NF_CONNTRACK_IPV4=m
> CONFIG_NF_CONNTRACK_PROC_COMPAT=y
> CONFIG_IP_NF_QUEUE=m
> CONFIG_IP_NF_IPTABLES=m
> CONFIG_IP_NF_MATCH_ADDRTYPE=m
> CONFIG_IP_NF_MATCH_AH=m
> CONFIG_IP_NF_MATCH_ECN=m
> CONFIG_IP_NF_MATCH_TTL=m
> CONFIG_IP_NF_FILTER=m
> CONFIG_IP_NF_TARGET_REJECT=m
> CONFIG_IP_NF_TARGET_LOG=m
> CONFIG_IP_NF_TARGET_ULOG=m
> CONFIG_NF_NAT=m
> CONFIG_NF_NAT_NEEDED=y
> CONFIG_IP_NF_TARGET_MASQUERADE=m
> CONFIG_IP_NF_TARGET_NETMAP=m
> CONFIG_IP_NF_TARGET_REDIRECT=m
> CONFIG_NF_NAT_SNMP_BASIC=m
> CONFIG_NF_NAT_PROTO_DCCP=m
> CONFIG_NF_NAT_PROTO_GRE=m
> CONFIG_NF_NAT_PROTO_SCTP=m
> CONFIG_NF_NAT_FTP=m
> CONFIG_NF_NAT_IRC=m
> CONFIG_NF_NAT_TFTP=m
> CONFIG_NF_NAT_AMANDA=m
> CONFIG_NF_NAT_PPTP=m
> CONFIG_NF_NAT_H323=m
> CONFIG_NF_NAT_SIP=m
> CONFIG_IP_NF_MANGLE=m
> CONFIG_IP_NF_TARGET_CLUSTERIP=m
> CONFIG_IP_NF_TARGET_ECN=m
> CONFIG_IP_NF_TARGET_TTL=m
> CONFIG_IP_NF_RAW=m
> # CONFIG_IP_NF_SECURITY is not set
> CONFIG_IP_NF_ARPTABLES=m
> CONFIG_IP_NF_ARPFILTER=m
> CONFIG_IP_NF_ARP_MANGLE=m
> 
> #
> # IPv6: Netfilter Configuration
> #
> CONFIG_NF_CONNTRACK_IPV6=m
> CONFIG_IP6_NF_QUEUE=m
> CONFIG_IP6_NF_IPTABLES=m
> CONFIG_IP6_NF_MATCH_AH=m
> CONFIG_IP6_NF_MATCH_EUI64=m
> CONFIG_IP6_NF_MATCH_FRAG=m
> CONFIG_IP6_NF_MATCH_OPTS=m
> CONFIG_IP6_NF_MATCH_HL=m
> CONFIG_IP6_NF_MATCH_IPV6HEADER=m
> CONFIG_IP6_NF_MATCH_MH=m
> CONFIG_IP6_NF_MATCH_RT=m
> CONFIG_IP6_NF_TARGET_HL=m
> CONFIG_IP6_NF_TARGET_LOG=m
> CONFIG_IP6_NF_FILTER=m
> CONFIG_IP6_NF_TARGET_REJECT=m
> CONFIG_IP6_NF_MANGLE=m
> CONFIG_IP6_NF_RAW=m
> # CONFIG_IP6_NF_SECURITY is not set
> 
> #
> # DECnet: Netfilter Configuration
> #
> # CONFIG_DECNET_NF_GRABULATOR is not set
> CONFIG_BRIDGE_NF_EBTABLES=m
> CONFIG_BRIDGE_EBT_BROUTE=m
> CONFIG_BRIDGE_EBT_T_FILTER=m
> CONFIG_BRIDGE_EBT_T_NAT=m
> CONFIG_BRIDGE_EBT_802_3=m
> CONFIG_BRIDGE_EBT_AMONG=m
> CONFIG_BRIDGE_EBT_ARP=m
> CONFIG_BRIDGE_EBT_IP=m
> # CONFIG_BRIDGE_EBT_IP6 is not set
> CONFIG_BRIDGE_EBT_LIMIT=m
> CONFIG_BRIDGE_EBT_MARK=m
> CONFIG_BRIDGE_EBT_PKTTYPE=m
> CONFIG_BRIDGE_EBT_STP=m
> CONFIG_BRIDGE_EBT_VLAN=m
> CONFIG_BRIDGE_EBT_ARPREPLY=m
> CONFIG_BRIDGE_EBT_DNAT=m
> CONFIG_BRIDGE_EBT_MARK_T=m
> CONFIG_BRIDGE_EBT_REDIRECT=m
> CONFIG_BRIDGE_EBT_SNAT=m
> CONFIG_BRIDGE_EBT_LOG=m
> CONFIG_BRIDGE_EBT_ULOG=m
> # CONFIG_BRIDGE_EBT_NFLOG is not set
> CONFIG_IP_DCCP=m
> CONFIG_INET_DCCP_DIAG=m
> 
> #
> # DCCP CCIDs Configuration (EXPERIMENTAL)
> #
> # CONFIG_IP_DCCP_CCID2_DEBUG is not set
> CONFIG_IP_DCCP_CCID3=y
> # CONFIG_IP_DCCP_CCID3_DEBUG is not set
> CONFIG_IP_DCCP_CCID3_RTO=100
> CONFIG_IP_DCCP_TFRC_LIB=y
> 
> #
> # DCCP Kernel Hacking
> #
> # CONFIG_IP_DCCP_DEBUG is not set
> # CONFIG_NET_DCCPPROBE is not set
> CONFIG_IP_SCTP=m
> # CONFIG_NET_SCTPPROBE is not set
> # CONFIG_SCTP_DBG_MSG is not set
> # CONFIG_SCTP_DBG_OBJCNT is not set
> # CONFIG_SCTP_HMAC_NONE is not set
> # CONFIG_SCTP_HMAC_SHA1 is not set
> CONFIG_SCTP_HMAC_MD5=y
> # CONFIG_RDS is not set
> CONFIG_TIPC=m
> # CONFIG_TIPC_ADVANCED is not set
> # CONFIG_TIPC_DEBUG is not set
> CONFIG_ATM=m
> CONFIG_ATM_CLIP=m
> # CONFIG_ATM_CLIP_NO_ICMP is not set
> CONFIG_ATM_LANE=m
> # CONFIG_ATM_MPOA is not set
> CONFIG_ATM_BR2684=m
> # CONFIG_ATM_BR2684_IPFILTER is not set
> # CONFIG_L2TP is not set
> CONFIG_STP=m
> CONFIG_BRIDGE=m
> CONFIG_BRIDGE_IGMP_SNOOPING=y
> # CONFIG_NET_DSA is not set
> CONFIG_VLAN_8021Q=m
> # CONFIG_VLAN_8021Q_GVRP is not set
> CONFIG_DECNET=m
> CONFIG_DECNET_ROUTER=y
> CONFIG_LLC=y
> # CONFIG_LLC2 is not set
> CONFIG_IPX=m
> # CONFIG_IPX_INTERN is not set
> CONFIG_ATALK=m
> CONFIG_DEV_APPLETALK=m
> CONFIG_IPDDP=m
> CONFIG_IPDDP_ENCAP=y
> CONFIG_IPDDP_DECAP=y
> # CONFIG_X25 is not set
> # CONFIG_LAPB is not set
> # CONFIG_ECONET is not set
> CONFIG_WAN_ROUTER=m
> # CONFIG_PHONET is not set
> # CONFIG_IEEE802154 is not set
> CONFIG_NET_SCHED=y
> 
> #
> # Queueing/Scheduling
> #
> CONFIG_NET_SCH_CBQ=m
> CONFIG_NET_SCH_HTB=m
> CONFIG_NET_SCH_HFSC=m
> CONFIG_NET_SCH_ATM=m
> CONFIG_NET_SCH_PRIO=m
> # CONFIG_NET_SCH_MULTIQ is not set
> CONFIG_NET_SCH_RED=m
> CONFIG_NET_SCH_SFQ=m
> CONFIG_NET_SCH_TEQL=m
> CONFIG_NET_SCH_TBF=m
> CONFIG_NET_SCH_GRED=m
> CONFIG_NET_SCH_DSMARK=m
> CONFIG_NET_SCH_NETEM=m
> # CONFIG_NET_SCH_DRR is not set
> CONFIG_NET_SCH_INGRESS=m
> 
> #
> # Classification
> #
> CONFIG_NET_CLS=y
> CONFIG_NET_CLS_BASIC=m
> CONFIG_NET_CLS_TCINDEX=m
> CONFIG_NET_CLS_ROUTE4=m
> CONFIG_NET_CLS_ROUTE=y
> CONFIG_NET_CLS_FW=m
> CONFIG_NET_CLS_U32=m
> CONFIG_CLS_U32_PERF=y
> CONFIG_CLS_U32_MARK=y
> CONFIG_NET_CLS_RSVP=m
> CONFIG_NET_CLS_RSVP6=m
> # CONFIG_NET_CLS_FLOW is not set
> CONFIG_NET_EMATCH=y
> CONFIG_NET_EMATCH_STACK=32
> CONFIG_NET_EMATCH_CMP=m
> CONFIG_NET_EMATCH_NBYTE=m
> CONFIG_NET_EMATCH_U32=m
> CONFIG_NET_EMATCH_META=m
> CONFIG_NET_EMATCH_TEXT=m
> CONFIG_NET_CLS_ACT=y
> CONFIG_NET_ACT_POLICE=m
> CONFIG_NET_ACT_GACT=m
> CONFIG_GACT_PROB=y
> CONFIG_NET_ACT_MIRRED=m
> CONFIG_NET_ACT_IPT=m
> # CONFIG_NET_ACT_NAT is not set
> CONFIG_NET_ACT_PEDIT=m
> CONFIG_NET_ACT_SIMP=m
> # CONFIG_NET_ACT_SKBEDIT is not set
> CONFIG_NET_CLS_IND=y
> CONFIG_NET_SCH_FIFO=y
> # CONFIG_DCB is not set
> CONFIG_DNS_RESOLVER=y
> CONFIG_RPS=y
> 
> #
> # Network testing
> #
> CONFIG_NET_PKTGEN=m
> # CONFIG_NET_TCPPROBE is not set
> # CONFIG_NET_DROP_MONITOR is not set
> # CONFIG_HAMRADIO is not set
> # CONFIG_CAN is not set
> CONFIG_IRDA=m
> 
> #
> # IrDA protocols
> #
> CONFIG_IRLAN=m
> CONFIG_IRNET=m
> CONFIG_IRCOMM=m
> # CONFIG_IRDA_ULTRA is not set
> 
> #
> # IrDA options
> #
> CONFIG_IRDA_CACHE_LAST_LSAP=y
> CONFIG_IRDA_FAST_RR=y
> # CONFIG_IRDA_DEBUG is not set
> 
> #
> # Infrared-port device drivers
> #
> 
> #
> # SIR device drivers
> #
> CONFIG_IRTTY_SIR=m
> 
> #
> # Dongle support
> #
> CONFIG_DONGLE=y
> CONFIG_ESI_DONGLE=m
> CONFIG_ACTISYS_DONGLE=m
> CONFIG_TEKRAM_DONGLE=m
> CONFIG_TOIM3232_DONGLE=m
> CONFIG_LITELINK_DONGLE=m
> CONFIG_MA600_DONGLE=m
> CONFIG_GIRBIL_DONGLE=m
> CONFIG_MCP2120_DONGLE=m
> CONFIG_OLD_BELKIN_DONGLE=m
> CONFIG_ACT200L_DONGLE=m
> # CONFIG_KINGSUN_DONGLE is not set
> # CONFIG_KSDAZZLE_DONGLE is not set
> # CONFIG_KS959_DONGLE is not set
> 
> #
> # FIR device drivers
> #
> CONFIG_USB_IRDA=m
> CONFIG_SIGMATEL_FIR=m
> CONFIG_NSC_FIR=m
> CONFIG_WINBOND_FIR=m
> CONFIG_SMC_IRCC_FIR=m
> CONFIG_ALI_FIR=m
> CONFIG_VLSI_FIR=m
> CONFIG_VIA_FIR=m
> CONFIG_MCS_FIR=m
> CONFIG_BT=m
> CONFIG_BT_L2CAP=m
> CONFIG_BT_SCO=m
> CONFIG_BT_RFCOMM=m
> CONFIG_BT_RFCOMM_TTY=y
> CONFIG_BT_BNEP=m
> CONFIG_BT_BNEP_MC_FILTER=y
> CONFIG_BT_BNEP_PROTO_FILTER=y
> CONFIG_BT_HIDP=m
> 
> #
> # Bluetooth device drivers
> #
> # CONFIG_BT_HCIBTUSB is not set
> # CONFIG_BT_HCIBTSDIO is not set
> CONFIG_BT_HCIUART=m
> CONFIG_BT_HCIUART_H4=y
> CONFIG_BT_HCIUART_BCSP=y
> # CONFIG_BT_HCIUART_ATH3K is not set
> # CONFIG_BT_HCIUART_LL is not set
> CONFIG_BT_HCIBCM203X=m
> CONFIG_BT_HCIBPA10X=m
> CONFIG_BT_HCIBFUSB=m
> CONFIG_BT_HCIDTL1=m
> CONFIG_BT_HCIBT3C=m
> CONFIG_BT_HCIBLUECARD=m
> CONFIG_BT_HCIBTUART=m
> CONFIG_BT_HCIVHCI=m
> # CONFIG_BT_MRVL is not set
> # CONFIG_AF_RXRPC is not set
> CONFIG_FIB_RULES=y
> CONFIG_WIRELESS=y
> # CONFIG_CFG80211 is not set
> # CONFIG_LIB80211 is not set
> 
> #
> # CFG80211 needs to be enabled for MAC80211
> #
> 
> #
> # Some wireless drivers require a rate control algorithm
> #
> # CONFIG_WIMAX is not set
> CONFIG_RFKILL=m
> CONFIG_RFKILL_LEDS=y
> CONFIG_RFKILL_INPUT=y
> # CONFIG_CAIF is not set
> 
> #
> # Device Drivers
> #
> 
> #
> # Generic Driver Options
> #
> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> # CONFIG_DEVTMPFS is not set
> CONFIG_STANDALONE=y
> CONFIG_PREVENT_FIRMWARE_BUILD=y
> CONFIG_FW_LOADER=y
> CONFIG_FIRMWARE_IN_KERNEL=y
> CONFIG_EXTRA_FIRMWARE=""
> # CONFIG_DEBUG_DRIVER is not set
> # CONFIG_DEBUG_DEVRES is not set
> # CONFIG_SYS_HYPERVISOR is not set
> CONFIG_CONNECTOR=y
> CONFIG_PROC_EVENTS=y
> # CONFIG_MTD is not set
> CONFIG_PARPORT=m
> CONFIG_PARPORT_PC=m
> CONFIG_PARPORT_SERIAL=m
> # CONFIG_PARPORT_PC_FIFO is not set
> # CONFIG_PARPORT_PC_SUPERIO is not set
> CONFIG_PARPORT_PC_PCMCIA=m
> # CONFIG_PARPORT_GSC is not set
> # CONFIG_PARPORT_AX88796 is not set
> CONFIG_PARPORT_1284=y
> CONFIG_PARPORT_NOT_PC=y
> CONFIG_PNP=y
> CONFIG_PNP_DEBUG_MESSAGES=y
> 
> #
> # Protocols
> #
> CONFIG_PNPACPI=y
> CONFIG_BLK_DEV=y
> CONFIG_BLK_DEV_FD=m
> # CONFIG_PARIDE is not set
> CONFIG_BLK_CPQ_DA=y
> CONFIG_BLK_CPQ_CISS_DA=m
> CONFIG_CISS_SCSI_TAPE=y
> CONFIG_BLK_DEV_DAC960=m
> CONFIG_BLK_DEV_UMEM=m
> # CONFIG_BLK_DEV_COW_COMMON is not set
> CONFIG_BLK_DEV_LOOP=m
> CONFIG_BLK_DEV_CRYPTOLOOP=m
> # CONFIG_BLK_DEV_DRBD is not set
> CONFIG_BLK_DEV_NBD=m
> CONFIG_BLK_DEV_SX8=m
> CONFIG_BLK_DEV_UB=m
> CONFIG_BLK_DEV_RAM=y
> CONFIG_BLK_DEV_RAM_COUNT=16
> CONFIG_BLK_DEV_RAM_SIZE=16384
> # CONFIG_BLK_DEV_XIP is not set
> CONFIG_CDROM_PKTCDVD=m
> CONFIG_CDROM_PKTCDVD_BUFFERS=8
> # CONFIG_CDROM_PKTCDVD_WCACHE is not set
> CONFIG_ATA_OVER_ETH=m
> # CONFIG_BLK_DEV_HD is not set
> CONFIG_MISC_DEVICES=y
> # CONFIG_AD525X_DPOT is not set
> # CONFIG_IBM_ASM is not set
> # CONFIG_PHANTOM is not set
> CONFIG_SGI_IOC4=m
> CONFIG_TIFM_CORE=m
> CONFIG_TIFM_7XX1=m
> # CONFIG_ICS932S401 is not set
> # CONFIG_ENCLOSURE_SERVICES is not set
> # CONFIG_CS5535_MFGPT is not set
> # CONFIG_HP_ILO is not set
> # CONFIG_ISL29003 is not set
> # CONFIG_SENSORS_TSL2550 is not set
> # CONFIG_SENSORS_BH1780 is not set
> # CONFIG_HMC6352 is not set
> # CONFIG_DS1682 is not set
> # CONFIG_VMWARE_BALLOON is not set
> # CONFIG_BMP085 is not set
> # CONFIG_C2PORT is not set
> 
> #
> # EEPROM support
> #
> # CONFIG_EEPROM_AT24 is not set
> # CONFIG_EEPROM_LEGACY is not set
> # CONFIG_EEPROM_MAX6875 is not set
> # CONFIG_EEPROM_93CX6 is not set
> # CONFIG_CB710_CORE is not set
> # CONFIG_IWMC3200TOP is not set
> CONFIG_HAVE_IDE=y
> # CONFIG_IDE is not set
> 
> #
> # SCSI device support
> #
> CONFIG_SCSI_MOD=y
> CONFIG_RAID_ATTRS=m
> CONFIG_SCSI=y
> CONFIG_SCSI_DMA=y
> CONFIG_SCSI_TGT=m
> CONFIG_SCSI_NETLINK=y
> CONFIG_SCSI_PROC_FS=y
> 
> #
> # SCSI support type (disk, tape, CD-ROM)
> #
> CONFIG_BLK_DEV_SD=y
> CONFIG_CHR_DEV_ST=m
> CONFIG_CHR_DEV_OSST=m
> CONFIG_BLK_DEV_SR=m
> CONFIG_BLK_DEV_SR_VENDOR=y
> CONFIG_CHR_DEV_SG=m
> CONFIG_CHR_DEV_SCH=m
> CONFIG_SCSI_MULTI_LUN=y
> CONFIG_SCSI_CONSTANTS=y
> CONFIG_SCSI_LOGGING=y
> # CONFIG_SCSI_SCAN_ASYNC is not set
> CONFIG_SCSI_WAIT_SCAN=m
> 
> #
> # SCSI Transports
> #
> CONFIG_SCSI_SPI_ATTRS=y
> CONFIG_SCSI_FC_ATTRS=m
> # CONFIG_SCSI_FC_TGT_ATTRS is not set
> CONFIG_SCSI_ISCSI_ATTRS=m
> CONFIG_SCSI_SAS_ATTRS=m
> # CONFIG_SCSI_SAS_LIBSAS is not set
> # CONFIG_SCSI_SRP_ATTRS is not set
> CONFIG_SCSI_LOWLEVEL=y
> CONFIG_ISCSI_TCP=m
> # CONFIG_ISCSI_BOOT_SYSFS is not set
> # CONFIG_SCSI_CXGB3_ISCSI is not set
> # CONFIG_SCSI_BNX2_ISCSI is not set
> # CONFIG_BE2ISCSI is not set
> CONFIG_BLK_DEV_3W_XXXX_RAID=m
> # CONFIG_SCSI_HPSA is not set
> CONFIG_SCSI_3W_9XXX=m
> # CONFIG_SCSI_3W_SAS is not set
> CONFIG_SCSI_ACARD=m
> CONFIG_SCSI_AACRAID=m
> CONFIG_SCSI_AIC7XXX=y
> CONFIG_AIC7XXX_CMDS_PER_DEVICE=4
> CONFIG_AIC7XXX_RESET_DELAY_MS=15000
> # CONFIG_AIC7XXX_DEBUG_ENABLE is not set
> CONFIG_AIC7XXX_DEBUG_MASK=0
> # CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
> CONFIG_SCSI_AIC7XXX_OLD=m
> CONFIG_SCSI_AIC79XX=m
> CONFIG_AIC79XX_CMDS_PER_DEVICE=4
> CONFIG_AIC79XX_RESET_DELAY_MS=15000
> # CONFIG_AIC79XX_DEBUG_ENABLE is not set
> CONFIG_AIC79XX_DEBUG_MASK=0
> # CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
> # CONFIG_SCSI_AIC94XX is not set
> # CONFIG_SCSI_MVSAS is not set
> # CONFIG_SCSI_DPT_I2O is not set
> # CONFIG_SCSI_ADVANSYS is not set
> CONFIG_SCSI_ARCMSR=m
> # CONFIG_SCSI_ARCMSR_AER is not set
> CONFIG_MEGARAID_NEWGEN=y
> CONFIG_MEGARAID_MM=m
> CONFIG_MEGARAID_MAILBOX=m
> CONFIG_MEGARAID_LEGACY=m
> CONFIG_MEGARAID_SAS=m
> # CONFIG_SCSI_MPT2SAS is not set
> CONFIG_SCSI_HPTIOP=m
> CONFIG_SCSI_BUSLOGIC=m
> # CONFIG_VMWARE_PVSCSI is not set
> # CONFIG_LIBFC is not set
> # CONFIG_LIBFCOE is not set
> # CONFIG_FCOE is not set
> # CONFIG_FCOE_FNIC is not set
> # CONFIG_SCSI_DMX3191D is not set
> # CONFIG_SCSI_EATA is not set
> # CONFIG_SCSI_FUTURE_DOMAIN is not set
> CONFIG_SCSI_GDTH=m
> CONFIG_SCSI_IPS=m
> CONFIG_SCSI_INITIO=m
> CONFIG_SCSI_INIA100=m
> CONFIG_SCSI_PPA=m
> CONFIG_SCSI_IMM=m
> # CONFIG_SCSI_IZIP_EPP16 is not set
> # CONFIG_SCSI_IZIP_SLOW_CTR is not set
> CONFIG_SCSI_STEX=m
> CONFIG_SCSI_SYM53C8XX_2=m
> CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
> CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
> CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
> CONFIG_SCSI_SYM53C8XX_MMIO=y
> # CONFIG_SCSI_IPR is not set
> CONFIG_SCSI_QLOGIC_1280=m
> CONFIG_SCSI_QLA_FC=m
> CONFIG_SCSI_QLA_ISCSI=m
> CONFIG_SCSI_LPFC=m
> # CONFIG_SCSI_LPFC_DEBUG_FS is not set
> CONFIG_SCSI_DC395x=m
> CONFIG_SCSI_DC390T=m
> # CONFIG_SCSI_PMCRAID is not set
> # CONFIG_SCSI_PM8001 is not set
> CONFIG_SCSI_SRP=m
> # CONFIG_SCSI_BFA_FC is not set
> # CONFIG_SCSI_LOWLEVEL_PCMCIA is not set
> # CONFIG_SCSI_DH is not set
> # CONFIG_SCSI_OSD_INITIATOR is not set
> CONFIG_ATA=y
> # CONFIG_ATA_NONSTANDARD is not set
> CONFIG_ATA_VERBOSE_ERROR=y
> CONFIG_ATA_ACPI=y
> CONFIG_SATA_PMP=y
> 
> #
> # Controllers with non-SFF native interface
> #
> CONFIG_SATA_AHCI=y
> # CONFIG_SATA_AHCI_PLATFORM is not set
> CONFIG_SATA_INIC162X=m
> CONFIG_SATA_SIL24=m
> CONFIG_ATA_SFF=y
> 
> #
> # SFF controllers with custom DMA interface
> #
> CONFIG_PDC_ADMA=m
> CONFIG_SATA_QSTOR=m
> CONFIG_SATA_SX4=m
> CONFIG_ATA_BMDMA=y
> 
> #
> # SATA SFF controllers with BMDMA
> #
> CONFIG_ATA_PIIX=y
> CONFIG_SATA_MV=m
> CONFIG_SATA_NV=y
> CONFIG_SATA_PROMISE=m
> CONFIG_SATA_SIL=m
> CONFIG_SATA_SIS=m
> CONFIG_SATA_SVW=m
> CONFIG_SATA_ULI=m
> CONFIG_SATA_VIA=m
> CONFIG_SATA_VITESSE=m
> 
> #
> # PATA SFF controllers with BMDMA
> #
> CONFIG_PATA_ALI=m
> CONFIG_PATA_AMD=y
> CONFIG_PATA_ARTOP=m
> CONFIG_PATA_ATIIXP=m
> # CONFIG_PATA_ATP867X is not set
> CONFIG_PATA_CMD64X=m
> CONFIG_PATA_CS5520=m
> CONFIG_PATA_CS5530=m
> CONFIG_PATA_CYPRESS=m
> CONFIG_PATA_EFAR=m
> CONFIG_PATA_HPT366=m
> CONFIG_PATA_HPT37X=m
> CONFIG_PATA_HPT3X2N=m
> CONFIG_PATA_HPT3X3=m
> # CONFIG_PATA_HPT3X3_DMA is not set
> CONFIG_PATA_IT8213=m
> CONFIG_PATA_IT821X=m
> CONFIG_PATA_JMICRON=m
> CONFIG_PATA_MARVELL=m
> CONFIG_PATA_NETCELL=m
> # CONFIG_PATA_NINJA32 is not set
> # CONFIG_PATA_NS87415 is not set
> CONFIG_PATA_OLDPIIX=y
> CONFIG_PATA_OPTIDMA=m
> CONFIG_PATA_PDC2027X=m
> CONFIG_PATA_PDC_OLD=m
> CONFIG_PATA_RADISYS=m
> # CONFIG_PATA_RDC is not set
> CONFIG_PATA_SC1200=m
> # CONFIG_PATA_SCH is not set
> CONFIG_PATA_SERVERWORKS=m
> CONFIG_PATA_SIL680=m
> CONFIG_PATA_SIS=m
> # CONFIG_PATA_TOSHIBA is not set
> CONFIG_PATA_TRIFLEX=m
> CONFIG_PATA_VIA=m
> CONFIG_PATA_WINBOND=m
> 
> #
> # PIO-only SFF controllers
> #
> # CONFIG_PATA_CMD640_PCI is not set
> CONFIG_PATA_MPIIX=m
> CONFIG_PATA_NS87410=m
> CONFIG_PATA_OPTI=m
> CONFIG_PATA_PCMCIA=m
> CONFIG_PATA_RZ1000=m
> 
> #
> # Generic fallback / legacy drivers
> #
> # CONFIG_PATA_ACPI is not set
> CONFIG_ATA_GENERIC=m
> # CONFIG_PATA_LEGACY is not set
> CONFIG_MD=y
> CONFIG_BLK_DEV_MD=y
> CONFIG_MD_AUTODETECT=y
> CONFIG_MD_LINEAR=m
> CONFIG_MD_RAID0=m
> CONFIG_MD_RAID1=m
> CONFIG_MD_RAID10=m
> CONFIG_MD_RAID456=m
> # CONFIG_MULTICORE_RAID456 is not set
> CONFIG_MD_MULTIPATH=m
> CONFIG_MD_FAULTY=m
> CONFIG_BLK_DEV_DM=m
> # CONFIG_DM_DEBUG is not set
> CONFIG_DM_CRYPT=m
> CONFIG_DM_SNAPSHOT=m
> CONFIG_DM_MIRROR=m
> # CONFIG_DM_LOG_USERSPACE is not set
> CONFIG_DM_ZERO=m
> CONFIG_DM_MULTIPATH=m
> # CONFIG_DM_MULTIPATH_QL is not set
> # CONFIG_DM_MULTIPATH_ST is not set
> # CONFIG_DM_DELAY is not set
> # CONFIG_DM_UEVENT is not set
> CONFIG_FUSION=y
> CONFIG_FUSION_SPI=m
> CONFIG_FUSION_FC=m
> CONFIG_FUSION_SAS=m
> CONFIG_FUSION_MAX_SGE=40
> CONFIG_FUSION_CTL=m
> CONFIG_FUSION_LAN=m
> # CONFIG_FUSION_LOGGING is not set
> 
> #
> # IEEE 1394 (FireWire) support
> #
> 
> #
> # You can enable one or both FireWire driver stacks.
> #
> 
> #
> # The newer stack is recommended.
> #
> # CONFIG_FIREWIRE is not set
> CONFIG_IEEE1394=m
> CONFIG_IEEE1394_OHCI1394=m
> CONFIG_IEEE1394_PCILYNX=m
> CONFIG_IEEE1394_SBP2=m
> # CONFIG_IEEE1394_SBP2_PHYS_DMA is not set
> CONFIG_IEEE1394_ETH1394_ROM_ENTRY=y
> CONFIG_IEEE1394_ETH1394=m
> CONFIG_IEEE1394_RAWIO=m
> CONFIG_IEEE1394_VIDEO1394=m
> CONFIG_IEEE1394_DV1394=m
> # CONFIG_IEEE1394_VERBOSEDEBUG is not set
> # CONFIG_FIREWIRE_NOSY is not set
> CONFIG_I2O=m
> # CONFIG_I2O_LCT_NOTIFY_ON_CHANGES is not set
> CONFIG_I2O_EXT_ADAPTEC=y
> CONFIG_I2O_EXT_ADAPTEC_DMA64=y
> # CONFIG_I2O_CONFIG is not set
> CONFIG_I2O_BUS=m
> CONFIG_I2O_BLOCK=m
> CONFIG_I2O_SCSI=m
> CONFIG_I2O_PROC=m
> # CONFIG_MACINTOSH_DRIVERS is not set
> CONFIG_NETDEVICES=y
> CONFIG_IFB=m
> CONFIG_DUMMY=m
> CONFIG_BONDING=m
> # CONFIG_MACVLAN is not set
> CONFIG_EQUALIZER=m
> CONFIG_TUN=m
> # CONFIG_VETH is not set
> CONFIG_NET_SB1000=m
> # CONFIG_ARCNET is not set
> CONFIG_PHYLIB=y
> 
> #
> # MII PHY device drivers
> #
> CONFIG_MARVELL_PHY=m
> CONFIG_DAVICOM_PHY=m
> CONFIG_QSEMI_PHY=m
> CONFIG_LXT_PHY=m
> CONFIG_CICADA_PHY=m
> CONFIG_VITESSE_PHY=m
> CONFIG_SMSC_PHY=m
> CONFIG_BROADCOM_PHY=m
> # CONFIG_ICPLUS_PHY is not set
> # CONFIG_REALTEK_PHY is not set
> # CONFIG_NATIONAL_PHY is not set
> # CONFIG_STE10XP is not set
> # CONFIG_LSI_ET1011C_PHY is not set
> # CONFIG_MICREL_PHY is not set
> # CONFIG_FIXED_PHY is not set
> # CONFIG_MDIO_BITBANG is not set
> CONFIG_NET_ETHERNET=y
> CONFIG_MII=y
> CONFIG_HAPPYMEAL=m
> CONFIG_SUNGEM=m
> CONFIG_CASSINI=m
> CONFIG_NET_VENDOR_3COM=y
> CONFIG_VORTEX=y
> CONFIG_TYPHOON=m
> # CONFIG_ETHOC is not set
> # CONFIG_DNET is not set
> CONFIG_NET_TULIP=y
> CONFIG_DE2104X=m
> CONFIG_DE2104X_DSL=0
> CONFIG_TULIP=m
> # CONFIG_TULIP_MWI is not set
> CONFIG_TULIP_MMIO=y
> # CONFIG_TULIP_NAPI is not set
> CONFIG_DE4X5=m
> CONFIG_WINBOND_840=m
> CONFIG_DM9102=m
> CONFIG_ULI526X=m
> CONFIG_PCMCIA_XIRCOM=m
> # CONFIG_HP100 is not set
> # CONFIG_IBM_NEW_EMAC_ZMII is not set
> # CONFIG_IBM_NEW_EMAC_RGMII is not set
> # CONFIG_IBM_NEW_EMAC_TAH is not set
> # CONFIG_IBM_NEW_EMAC_EMAC4 is not set
> # CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
> # CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
> # CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
> CONFIG_NET_PCI=y
> CONFIG_PCNET32=m
> CONFIG_AMD8111_ETH=m
> CONFIG_ADAPTEC_STARFIRE=m
> # CONFIG_KSZ884X_PCI is not set
> CONFIG_B44=m
> CONFIG_B44_PCI_AUTOSELECT=y
> CONFIG_B44_PCICORE_AUTOSELECT=y
> CONFIG_B44_PCI=y
> CONFIG_FORCEDETH=y
> CONFIG_E100=y
> CONFIG_FEALNX=m
> CONFIG_NATSEMI=m
> CONFIG_NE2K_PCI=m
> CONFIG_8139CP=m
> CONFIG_8139TOO=y
> # CONFIG_8139TOO_PIO is not set
> # CONFIG_8139TOO_TUNE_TWISTER is not set
> CONFIG_8139TOO_8129=y
> # CONFIG_8139_OLD_RX_RESET is not set
> # CONFIG_R6040 is not set
> CONFIG_SIS900=m
> CONFIG_EPIC100=m
> # CONFIG_SMSC9420 is not set
> CONFIG_SUNDANCE=m
> # CONFIG_SUNDANCE_MMIO is not set
> # CONFIG_TLAN is not set
> # CONFIG_KS8851_MLL is not set
> CONFIG_VIA_RHINE=m
> CONFIG_VIA_RHINE_MMIO=y
> CONFIG_SC92031=m
> CONFIG_NET_POCKET=y
> CONFIG_ATP=m
> CONFIG_DE600=m
> CONFIG_DE620=m
> # CONFIG_ATL2 is not set
> CONFIG_NETDEV_1000=y
> CONFIG_ACENIC=m
> # CONFIG_ACENIC_OMIT_TIGON_I is not set
> CONFIG_DL2K=m
> CONFIG_E1000=y
> CONFIG_E1000E=y
> # CONFIG_IP1000 is not set
> CONFIG_IGB=y
> # CONFIG_IGBVF is not set
> CONFIG_NS83820=m
> CONFIG_HAMACHI=m
> CONFIG_YELLOWFIN=m
> CONFIG_R8169=m
> CONFIG_R8169_VLAN=y
> # CONFIG_SIS190 is not set
> CONFIG_SKGE=m
> # CONFIG_SKGE_DEBUG is not set
> CONFIG_SKY2=m
> # CONFIG_SKY2_DEBUG is not set
> CONFIG_VIA_VELOCITY=m
> CONFIG_TIGON3=y
> CONFIG_BNX2=m
> # CONFIG_CNIC is not set
> CONFIG_QLA3XXX=m
> CONFIG_ATL1=m
> # CONFIG_ATL1E is not set
> # CONFIG_ATL1C is not set
> # CONFIG_JME is not set
> CONFIG_NETDEV_10000=y
> CONFIG_MDIO=m
> CONFIG_CHELSIO_T1=m
> CONFIG_CHELSIO_T1_1G=y
> CONFIG_CHELSIO_T3_DEPENDS=y
> CONFIG_CHELSIO_T3=m
> CONFIG_CHELSIO_T4_DEPENDS=y
> # CONFIG_CHELSIO_T4 is not set
> CONFIG_CHELSIO_T4VF_DEPENDS=y
> # CONFIG_CHELSIO_T4VF is not set
> # CONFIG_ENIC is not set
> # CONFIG_IXGBE is not set
> CONFIG_IXGB=m
> CONFIG_S2IO=m
> CONFIG_MYRI10GE=m
> # CONFIG_NIU is not set
> # CONFIG_MLX4_EN is not set
> # CONFIG_MLX4_CORE is not set
> # CONFIG_TEHUTI is not set
> # CONFIG_BNX2X is not set
> # CONFIG_QLCNIC is not set
> # CONFIG_QLGE is not set
> # CONFIG_SFC is not set
> # CONFIG_BE2NET is not set
> CONFIG_TR=y
> CONFIG_IBMOL=m
> CONFIG_3C359=m
> # CONFIG_TMS380TR is not set
> CONFIG_WLAN=y
> # CONFIG_PCMCIA_RAYCS is not set
> # CONFIG_AIRO is not set
> # CONFIG_ATMEL is not set
> # CONFIG_AIRO_CS is not set
> # CONFIG_PCMCIA_WL3501 is not set
> # CONFIG_PRISM54 is not set
> # CONFIG_USB_ZD1201 is not set
> # CONFIG_HOSTAP is not set
> 
> #
> # Enable WiMAX (Networking options) to see the WiMAX drivers
> #
> 
> #
> # USB Network Adapters
> #
> CONFIG_USB_CATC=m
> CONFIG_USB_KAWETH=m
> CONFIG_USB_PEGASUS=m
> CONFIG_USB_RTL8150=m
> CONFIG_USB_USBNET=m
> CONFIG_USB_NET_AX8817X=m
> CONFIG_USB_NET_CDCETHER=m
> # CONFIG_USB_NET_CDC_EEM is not set
> CONFIG_USB_NET_DM9601=m
> # CONFIG_USB_NET_SMSC75XX is not set
> # CONFIG_USB_NET_SMSC95XX is not set
> CONFIG_USB_NET_GL620A=m
> CONFIG_USB_NET_NET1080=m
> CONFIG_USB_NET_PLUSB=m
> CONFIG_USB_NET_MCS7830=m
> CONFIG_USB_NET_RNDIS_HOST=m
> CONFIG_USB_NET_CDC_SUBSET=m
> CONFIG_USB_ALI_M5632=y
> CONFIG_USB_AN2720=y
> CONFIG_USB_BELKIN=y
> CONFIG_USB_ARMLINUX=y
> CONFIG_USB_EPSON2888=y
> CONFIG_USB_KC2190=y
> CONFIG_USB_NET_ZAURUS=m
> # CONFIG_USB_HSO is not set
> # CONFIG_USB_NET_INT51X1 is not set
> # CONFIG_USB_IPHETH is not set
> # CONFIG_USB_SIERRA_NET is not set
> CONFIG_NET_PCMCIA=y
> CONFIG_PCMCIA_3C589=m
> CONFIG_PCMCIA_3C574=m
> CONFIG_PCMCIA_FMVJ18X=m
> CONFIG_PCMCIA_PCNET=m
> CONFIG_PCMCIA_NMCLAN=m
> CONFIG_PCMCIA_SMC91C92=m
> CONFIG_PCMCIA_XIRC2PS=m
> CONFIG_PCMCIA_AXNET=m
> # CONFIG_PCMCIA_IBMTR is not set
> # CONFIG_WAN is not set
> CONFIG_ATM_DRIVERS=y
> # CONFIG_ATM_DUMMY is not set
> CONFIG_ATM_TCP=m
> CONFIG_ATM_LANAI=m
> CONFIG_ATM_ENI=m
> # CONFIG_ATM_ENI_DEBUG is not set
> # CONFIG_ATM_ENI_TUNE_BURST is not set
> CONFIG_ATM_FIRESTREAM=m
> # CONFIG_ATM_ZATM is not set
> # CONFIG_ATM_NICSTAR is not set
> CONFIG_ATM_IDT77252=m
> # CONFIG_ATM_IDT77252_DEBUG is not set
> # CONFIG_ATM_IDT77252_RCV_ALL is not set
> CONFIG_ATM_IDT77252_USE_SUNI=y
> CONFIG_ATM_AMBASSADOR=m
> # CONFIG_ATM_AMBASSADOR_DEBUG is not set
> CONFIG_ATM_HORIZON=m
> # CONFIG_ATM_HORIZON_DEBUG is not set
> # CONFIG_ATM_IA is not set
> # CONFIG_ATM_FORE200E is not set
> CONFIG_ATM_HE=m
> # CONFIG_ATM_HE_USE_SUNI is not set
> # CONFIG_ATM_SOLOS is not set
> 
> #
> # CAIF transport drivers
> #
> CONFIG_FDDI=y
> # CONFIG_DEFXX is not set
> CONFIG_SKFP=m
> # CONFIG_HIPPI is not set
> CONFIG_PLIP=m
> CONFIG_PPP=m
> CONFIG_PPP_MULTILINK=y
> CONFIG_PPP_FILTER=y
> CONFIG_PPP_ASYNC=m
> CONFIG_PPP_SYNC_TTY=m
> CONFIG_PPP_DEFLATE=m
> # CONFIG_PPP_BSDCOMP is not set
> CONFIG_PPP_MPPE=m
> CONFIG_PPPOE=m
> CONFIG_PPPOATM=m
> CONFIG_SLIP=m
> CONFIG_SLIP_COMPRESSED=y
> CONFIG_SLHC=m
> CONFIG_SLIP_SMART=y
> # CONFIG_SLIP_MODE_SLIP6 is not set
> CONFIG_NET_FC=y
> CONFIG_NETCONSOLE=y
> # CONFIG_NETCONSOLE_DYNAMIC is not set
> CONFIG_NETPOLL=y
> CONFIG_NETPOLL_TRAP=y
> CONFIG_NET_POLL_CONTROLLER=y
> # CONFIG_VMXNET3 is not set
> # CONFIG_ISDN is not set
> # CONFIG_PHONE is not set
> 
> #
> # Input device support
> #
> CONFIG_INPUT=y
> CONFIG_INPUT_FF_MEMLESS=y
> CONFIG_INPUT_POLLDEV=y
> CONFIG_INPUT_SPARSEKMAP=m
> 
> #
> # Userland interfaces
> #
> CONFIG_INPUT_MOUSEDEV=y
> # CONFIG_INPUT_MOUSEDEV_PSAUX is not set
> CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
> CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
> CONFIG_INPUT_JOYDEV=m
> CONFIG_INPUT_EVDEV=y
> # CONFIG_INPUT_EVBUG is not set
> 
> #
> # Input Device Drivers
> #
> CONFIG_INPUT_KEYBOARD=y
> # CONFIG_KEYBOARD_ADP5588 is not set
> CONFIG_KEYBOARD_ATKBD=y
> # CONFIG_KEYBOARD_QT2160 is not set
> # CONFIG_KEYBOARD_LKKBD is not set
> # CONFIG_KEYBOARD_TCA6416 is not set
> # CONFIG_KEYBOARD_LM8323 is not set
> # CONFIG_KEYBOARD_MAX7359 is not set
> # CONFIG_KEYBOARD_MCS is not set
> # CONFIG_KEYBOARD_NEWTON is not set
> # CONFIG_KEYBOARD_OPENCORES is not set
> CONFIG_KEYBOARD_STOWAWAY=m
> # CONFIG_KEYBOARD_SUNKBD is not set
> # CONFIG_KEYBOARD_XTKBD is not set
> CONFIG_INPUT_MOUSE=y
> CONFIG_MOUSE_PS2=y
> CONFIG_MOUSE_PS2_ALPS=y
> CONFIG_MOUSE_PS2_LOGIPS2PP=y
> CONFIG_MOUSE_PS2_SYNAPTICS=y
> CONFIG_MOUSE_PS2_LIFEBOOK=y
> CONFIG_MOUSE_PS2_TRACKPOINT=y
> # CONFIG_MOUSE_PS2_ELANTECH is not set
> # CONFIG_MOUSE_PS2_SENTELIC is not set
> # CONFIG_MOUSE_PS2_TOUCHKIT is not set
> CONFIG_MOUSE_SERIAL=m
> # CONFIG_MOUSE_APPLETOUCH is not set
> # CONFIG_MOUSE_BCM5974 is not set
> CONFIG_MOUSE_VSXXXAA=m
> # CONFIG_MOUSE_SYNAPTICS_I2C is not set
> CONFIG_INPUT_JOYSTICK=y
> CONFIG_JOYSTICK_ANALOG=m
> CONFIG_JOYSTICK_A3D=m
> CONFIG_JOYSTICK_ADI=m
> CONFIG_JOYSTICK_COBRA=m
> CONFIG_JOYSTICK_GF2K=m
> CONFIG_JOYSTICK_GRIP=m
> CONFIG_JOYSTICK_GRIP_MP=m
> CONFIG_JOYSTICK_GUILLEMOT=m
> CONFIG_JOYSTICK_INTERACT=m
> CONFIG_JOYSTICK_SIDEWINDER=m
> CONFIG_JOYSTICK_TMDC=m
> CONFIG_JOYSTICK_IFORCE=m
> CONFIG_JOYSTICK_IFORCE_USB=y
> CONFIG_JOYSTICK_IFORCE_232=y
> CONFIG_JOYSTICK_WARRIOR=m
> CONFIG_JOYSTICK_MAGELLAN=m
> CONFIG_JOYSTICK_SPACEORB=m
> CONFIG_JOYSTICK_SPACEBALL=m
> CONFIG_JOYSTICK_STINGER=m
> CONFIG_JOYSTICK_TWIDJOY=m
> # CONFIG_JOYSTICK_ZHENHUA is not set
> CONFIG_JOYSTICK_DB9=m
> CONFIG_JOYSTICK_GAMECON=m
> CONFIG_JOYSTICK_TURBOGRAFX=m
> CONFIG_JOYSTICK_JOYDUMP=m
> # CONFIG_JOYSTICK_XPAD is not set
> # CONFIG_INPUT_TABLET is not set
> CONFIG_INPUT_TOUCHSCREEN=y
> # CONFIG_TOUCHSCREEN_AD7879 is not set
> # CONFIG_TOUCHSCREEN_DYNAPRO is not set
> # CONFIG_TOUCHSCREEN_HAMPSHIRE is not set
> # CONFIG_TOUCHSCREEN_EETI is not set
> # CONFIG_TOUCHSCREEN_FUJITSU is not set
> CONFIG_TOUCHSCREEN_GUNZE=m
> CONFIG_TOUCHSCREEN_ELO=m
> # CONFIG_TOUCHSCREEN_WACOM_W8001 is not set
> # CONFIG_TOUCHSCREEN_MCS5000 is not set
> CONFIG_TOUCHSCREEN_MTOUCH=m
> # CONFIG_TOUCHSCREEN_INEXIO is not set
> CONFIG_TOUCHSCREEN_MK712=m
> CONFIG_TOUCHSCREEN_PENMOUNT=m
> # CONFIG_TOUCHSCREEN_QT602240 is not set
> CONFIG_TOUCHSCREEN_TOUCHRIGHT=m
> CONFIG_TOUCHSCREEN_TOUCHWIN=m
> # CONFIG_TOUCHSCREEN_WM97XX is not set
> # CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
> # CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
> # CONFIG_TOUCHSCREEN_TSC2007 is not set
> # CONFIG_TOUCHSCREEN_TPS6507X is not set
> CONFIG_INPUT_MISC=y
> # CONFIG_INPUT_AD714X is not set
> CONFIG_INPUT_PCSPKR=m
> # CONFIG_INPUT_APANEL is not set
> CONFIG_INPUT_ATLAS_BTNS=m
> # CONFIG_INPUT_ATI_REMOTE is not set
> # CONFIG_INPUT_ATI_REMOTE2 is not set
> # CONFIG_INPUT_KEYSPAN_REMOTE is not set
> # CONFIG_INPUT_POWERMATE is not set
> # CONFIG_INPUT_YEALINK is not set
> # CONFIG_INPUT_CM109 is not set
> CONFIG_INPUT_UINPUT=m
> # CONFIG_INPUT_WINBOND_CIR is not set
> # CONFIG_INPUT_PCF8574 is not set
> # CONFIG_INPUT_ADXL34X is not set
> 
> #
> # Hardware I/O ports
> #
> CONFIG_SERIO=y
> CONFIG_SERIO_I8042=y
> CONFIG_SERIO_SERPORT=y
> # CONFIG_SERIO_CT82C710 is not set
> # CONFIG_SERIO_PARKBD is not set
> # CONFIG_SERIO_PCIPS2 is not set
> CONFIG_SERIO_LIBPS2=y
> CONFIG_SERIO_RAW=m
> # CONFIG_SERIO_ALTERA_PS2 is not set
> CONFIG_GAMEPORT=m
> CONFIG_GAMEPORT_NS558=m
> CONFIG_GAMEPORT_L4=m
> CONFIG_GAMEPORT_EMU10K1=m
> CONFIG_GAMEPORT_FM801=m
> 
> #
> # Character devices
> #
> CONFIG_VT=y
> CONFIG_CONSOLE_TRANSLATIONS=y
> CONFIG_VT_CONSOLE=y
> CONFIG_HW_CONSOLE=y
> CONFIG_VT_HW_CONSOLE_BINDING=y
> CONFIG_DEVKMEM=y
> CONFIG_SERIAL_NONSTANDARD=y
> # CONFIG_COMPUTONE is not set
> # CONFIG_ROCKETPORT is not set
> CONFIG_CYCLADES=m
> # CONFIG_CYZ_INTR is not set
> # CONFIG_DIGIEPCA is not set
> # CONFIG_MOXA_INTELLIO is not set
> # CONFIG_MOXA_SMARTIO is not set
> # CONFIG_ISI is not set
> CONFIG_SYNCLINK=m
> CONFIG_SYNCLINKMP=m
> CONFIG_SYNCLINK_GT=m
> CONFIG_N_HDLC=m
> # CONFIG_N_GSM is not set
> # CONFIG_RISCOM8 is not set
> # CONFIG_SPECIALIX is not set
> # CONFIG_STALDRV is not set
> # CONFIG_NOZOMI is not set
> 
> #
> # Serial drivers
> #
> CONFIG_SERIAL_8250=y
> CONFIG_SERIAL_8250_CONSOLE=y
> CONFIG_FIX_EARLYCON_MEM=y
> CONFIG_SERIAL_8250_PCI=y
> CONFIG_SERIAL_8250_PNP=y
> CONFIG_SERIAL_8250_CS=m
> CONFIG_SERIAL_8250_NR_UARTS=32
> CONFIG_SERIAL_8250_RUNTIME_UARTS=4
> CONFIG_SERIAL_8250_EXTENDED=y
> CONFIG_SERIAL_8250_MANY_PORTS=y
> CONFIG_SERIAL_8250_SHARE_IRQ=y
> CONFIG_SERIAL_8250_DETECT_IRQ=y
> CONFIG_SERIAL_8250_RSA=y
> 
> #
> # Non-8250 serial port support
> #
> # CONFIG_SERIAL_MFD_HSU is not set
> CONFIG_SERIAL_CORE=y
> CONFIG_SERIAL_CORE_CONSOLE=y
> CONFIG_SERIAL_JSM=m
> # CONFIG_SERIAL_TIMBERDALE is not set
> # CONFIG_SERIAL_ALTERA_JTAGUART is not set
> # CONFIG_SERIAL_ALTERA_UART is not set
> CONFIG_UNIX98_PTYS=y
> # CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
> # CONFIG_LEGACY_PTYS is not set
> CONFIG_PRINTER=m
> CONFIG_LP_CONSOLE=y
> CONFIG_PPDEV=m
> CONFIG_IPMI_HANDLER=m
> # CONFIG_IPMI_PANIC_EVENT is not set
> CONFIG_IPMI_DEVICE_INTERFACE=m
> CONFIG_IPMI_SI=m
> CONFIG_IPMI_WATCHDOG=m
> CONFIG_IPMI_POWEROFF=m
> CONFIG_HW_RANDOM=y
> # CONFIG_HW_RANDOM_TIMERIOMEM is not set
> CONFIG_HW_RANDOM_INTEL=m
> CONFIG_HW_RANDOM_AMD=m
> CONFIG_HW_RANDOM_VIA=y
> CONFIG_NVRAM=y
> CONFIG_R3964=m
> # CONFIG_APPLICOM is not set
> 
> #
> # PCMCIA character devices
> #
> # CONFIG_SYNCLINK_CS is not set
> CONFIG_CARDMAN_4000=m
> CONFIG_CARDMAN_4040=m
> # CONFIG_IPWIRELESS is not set
> CONFIG_MWAVE=m
> # CONFIG_RAW_DRIVER is not set
> # CONFIG_HPET is not set
> CONFIG_HANGCHECK_TIMER=m
> CONFIG_TCG_TPM=m
> CONFIG_TCG_TIS=m
> CONFIG_TCG_NSC=m
> CONFIG_TCG_ATMEL=m
> CONFIG_TCG_INFINEON=m
> # CONFIG_TELCLOCK is not set
> CONFIG_DEVPORT=y
> # CONFIG_RAMOOPS is not set
> CONFIG_I2C=y
> CONFIG_I2C_BOARDINFO=y
> CONFIG_I2C_COMPAT=y
> CONFIG_I2C_CHARDEV=m
> # CONFIG_I2C_MUX is not set
> CONFIG_I2C_HELPER_AUTO=y
> CONFIG_I2C_SMBUS=m
> CONFIG_I2C_ALGOBIT=m
> 
> #
> # I2C Hardware Bus support
> #
> 
> #
> # PC SMBus host controller drivers
> #
> # CONFIG_I2C_ALI1535 is not set
> # CONFIG_I2C_ALI1563 is not set
> # CONFIG_I2C_ALI15X3 is not set
> CONFIG_I2C_AMD756=m
> CONFIG_I2C_AMD8111=m
> CONFIG_I2C_I801=m
> # CONFIG_I2C_ISCH is not set
> CONFIG_I2C_PIIX4=y
> CONFIG_I2C_NFORCE2=y
> # CONFIG_I2C_SIS5595 is not set
> # CONFIG_I2C_SIS630 is not set
> CONFIG_I2C_SIS96X=m
> CONFIG_I2C_VIA=m
> CONFIG_I2C_VIAPRO=m
> 
> #
> # ACPI drivers
> #
> # CONFIG_I2C_SCMI is not set
> 
> #
> # I2C system bus drivers (mostly embedded / system-on-chip)
> #
> # CONFIG_I2C_OCORES is not set
> # CONFIG_I2C_PCA_PLATFORM is not set
> # CONFIG_I2C_SIMTEC is not set
> # CONFIG_I2C_XILINX is not set
> 
> #
> # External I2C/SMBus adapter drivers
> #
> CONFIG_I2C_PARPORT=m
> CONFIG_I2C_PARPORT_LIGHT=m
> # CONFIG_I2C_TAOS_EVM is not set
> # CONFIG_I2C_TINY_USB is not set
> 
> #
> # Other I2C/SMBus bus drivers
> #
> CONFIG_I2C_STUB=m
> # CONFIG_I2C_DEBUG_CORE is not set
> # CONFIG_I2C_DEBUG_ALGO is not set
> # CONFIG_I2C_DEBUG_BUS is not set
> # CONFIG_SPI is not set
> 
> #
> # PPS support
> #
> # CONFIG_PPS is not set
> CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
> # CONFIG_GPIOLIB is not set
> CONFIG_W1=m
> CONFIG_W1_CON=y
> 
> #
> # 1-wire Bus Masters
> #
> CONFIG_W1_MASTER_MATROX=m
> CONFIG_W1_MASTER_DS2490=m
> CONFIG_W1_MASTER_DS2482=m
> 
> #
> # 1-wire Slaves
> #
> CONFIG_W1_SLAVE_THERM=m
> CONFIG_W1_SLAVE_SMEM=m
> # CONFIG_W1_SLAVE_DS2431 is not set
> CONFIG_W1_SLAVE_DS2433=m
> CONFIG_W1_SLAVE_DS2433_CRC=y
> # CONFIG_W1_SLAVE_DS2760 is not set
> # CONFIG_W1_SLAVE_BQ27000 is not set
> CONFIG_POWER_SUPPLY=y
> # CONFIG_POWER_SUPPLY_DEBUG is not set
> # CONFIG_PDA_POWER is not set
> # CONFIG_TEST_POWER is not set
> # CONFIG_BATTERY_DS2760 is not set
> # CONFIG_BATTERY_DS2782 is not set
> # CONFIG_BATTERY_BQ27x00 is not set
> # CONFIG_BATTERY_MAX17040 is not set
> CONFIG_HWMON=y
> CONFIG_HWMON_VID=m
> # CONFIG_HWMON_DEBUG_CHIP is not set
> 
> #
> # Native drivers
> #
> CONFIG_SENSORS_ABITUGURU=m
> # CONFIG_SENSORS_ABITUGURU3 is not set
> # CONFIG_SENSORS_AD7414 is not set
> # CONFIG_SENSORS_AD7418 is not set
> CONFIG_SENSORS_ADM1021=m
> CONFIG_SENSORS_ADM1025=m
> CONFIG_SENSORS_ADM1026=m
> CONFIG_SENSORS_ADM1029=m
> CONFIG_SENSORS_ADM1031=m
> CONFIG_SENSORS_ADM9240=m
> # CONFIG_SENSORS_ADT7411 is not set
> # CONFIG_SENSORS_ADT7462 is not set
> # CONFIG_SENSORS_ADT7470 is not set
> # CONFIG_SENSORS_ADT7475 is not set
> # CONFIG_SENSORS_ASC7621 is not set
> CONFIG_SENSORS_K8TEMP=m
> # CONFIG_SENSORS_K10TEMP is not set
> CONFIG_SENSORS_ASB100=m
> CONFIG_SENSORS_ATXP1=m
> CONFIG_SENSORS_DS1621=m
> # CONFIG_SENSORS_I5K_AMB is not set
> CONFIG_SENSORS_F71805F=m
> # CONFIG_SENSORS_F71882FG is not set
> # CONFIG_SENSORS_F75375S is not set
> # CONFIG_SENSORS_FSCHMD is not set
> # CONFIG_SENSORS_G760A is not set
> CONFIG_SENSORS_GL518SM=m
> CONFIG_SENSORS_GL520SM=m
> # CONFIG_SENSORS_CORETEMP is not set
> # CONFIG_SENSORS_PKGTEMP is not set
> # CONFIG_SENSORS_IBMAEM is not set
> # CONFIG_SENSORS_IBMPEX is not set
> CONFIG_SENSORS_IT87=m
> # CONFIG_SENSORS_JC42 is not set
> CONFIG_SENSORS_LM63=m
> # CONFIG_SENSORS_LM73 is not set
> CONFIG_SENSORS_LM75=m
> CONFIG_SENSORS_LM77=m
> CONFIG_SENSORS_LM78=m
> CONFIG_SENSORS_LM80=m
> CONFIG_SENSORS_LM83=m
> CONFIG_SENSORS_LM85=m
> CONFIG_SENSORS_LM87=m
> CONFIG_SENSORS_LM90=m
> CONFIG_SENSORS_LM92=m
> # CONFIG_SENSORS_LM93 is not set
> # CONFIG_SENSORS_LTC4215 is not set
> # CONFIG_SENSORS_LTC4245 is not set
> # CONFIG_SENSORS_LM95241 is not set
> CONFIG_SENSORS_MAX1619=m
> # CONFIG_SENSORS_MAX6650 is not set
> CONFIG_SENSORS_PC87360=m
> CONFIG_SENSORS_PC87427=m
> CONFIG_SENSORS_PCF8591=m
> CONFIG_SENSORS_SIS5595=m
> # CONFIG_SENSORS_SMM665 is not set
> # CONFIG_SENSORS_DME1737 is not set
> # CONFIG_SENSORS_EMC1403 is not set
> # CONFIG_SENSORS_EMC2103 is not set
> CONFIG_SENSORS_SMSC47M1=m
> CONFIG_SENSORS_SMSC47M192=m
> CONFIG_SENSORS_SMSC47B397=m
> # CONFIG_SENSORS_ADS7828 is not set
> # CONFIG_SENSORS_AMC6821 is not set
> # CONFIG_SENSORS_THMC50 is not set
> # CONFIG_SENSORS_TMP102 is not set
> # CONFIG_SENSORS_TMP401 is not set
> # CONFIG_SENSORS_TMP421 is not set
> # CONFIG_SENSORS_VIA_CPUTEMP is not set
> CONFIG_SENSORS_VIA686A=m
> CONFIG_SENSORS_VT1211=m
> CONFIG_SENSORS_VT8231=m
> CONFIG_SENSORS_W83781D=m
> CONFIG_SENSORS_W83791D=m
> CONFIG_SENSORS_W83792D=m
> CONFIG_SENSORS_W83793=m
> CONFIG_SENSORS_W83L785TS=m
> # CONFIG_SENSORS_W83L786NG is not set
> CONFIG_SENSORS_W83627HF=m
> CONFIG_SENSORS_W83627EHF=m
> CONFIG_SENSORS_HDAPS=m
> # CONFIG_SENSORS_LIS3_I2C is not set
> # CONFIG_SENSORS_APPLESMC is not set
> 
> #
> # ACPI drivers
> #
> # CONFIG_SENSORS_ATK0110 is not set
> # CONFIG_SENSORS_LIS3LV02D is not set
> CONFIG_THERMAL=y
> # CONFIG_THERMAL_HWMON is not set
> CONFIG_WATCHDOG=y
> # CONFIG_WATCHDOG_NOWAYOUT is not set
> 
> #
> # Watchdog Device Drivers
> #
> CONFIG_SOFT_WATCHDOG=m
> # CONFIG_ACQUIRE_WDT is not set
> # CONFIG_ADVANTECH_WDT is not set
> CONFIG_ALIM1535_WDT=m
> CONFIG_ALIM7101_WDT=m
> # CONFIG_F71808E_WDT is not set
> # CONFIG_SC520_WDT is not set
> # CONFIG_EUROTECH_WDT is not set
> # CONFIG_IB700_WDT is not set
> CONFIG_IBMASR=m
> # CONFIG_WAFER_WDT is not set
> CONFIG_I6300ESB_WDT=m
> CONFIG_ITCO_WDT=m
> CONFIG_ITCO_VENDOR_SUPPORT=y
> # CONFIG_IT8712F_WDT is not set
> # CONFIG_IT87_WDT is not set
> # CONFIG_HP_WATCHDOG is not set
> # CONFIG_SC1200_WDT is not set
> CONFIG_PC87413_WDT=m
> # CONFIG_60XX_WDT is not set
> # CONFIG_SBC8360_WDT is not set
> # CONFIG_CPU5_WDT is not set
> # CONFIG_SMSC_SCH311X_WDT is not set
> # CONFIG_SMSC37B787_WDT is not set
> CONFIG_W83627HF_WDT=m
> CONFIG_W83697HF_WDT=m
> # CONFIG_W83697UG_WDT is not set
> CONFIG_W83877F_WDT=m
> CONFIG_W83977F_WDT=m
> CONFIG_MACHZ_WDT=m
> # CONFIG_SBC_EPX_C3_WATCHDOG is not set
> 
> #
> # PCI-based Watchdog Cards
> #
> CONFIG_PCIPCWATCHDOG=m
> CONFIG_WDTPCI=m
> 
> #
> # USB-based Watchdog Cards
> #
> CONFIG_USBPCWATCHDOG=m
> CONFIG_SSB_POSSIBLE=y
> 
> #
> # Sonics Silicon Backplane
> #
> CONFIG_SSB=m
> CONFIG_SSB_SPROM=y
> CONFIG_SSB_PCIHOST_POSSIBLE=y
> CONFIG_SSB_PCIHOST=y
> # CONFIG_SSB_B43_PCI_BRIDGE is not set
> CONFIG_SSB_PCMCIAHOST_POSSIBLE=y
> # CONFIG_SSB_PCMCIAHOST is not set
> CONFIG_SSB_SDIOHOST_POSSIBLE=y
> # CONFIG_SSB_SDIOHOST is not set
> # CONFIG_SSB_DEBUG is not set
> CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
> CONFIG_SSB_DRIVER_PCICORE=y
> CONFIG_MFD_SUPPORT=y
> # CONFIG_MFD_CORE is not set
> # CONFIG_MFD_88PM860X is not set
> CONFIG_MFD_SM501=m
> # CONFIG_HTC_PASIC3 is not set
> # CONFIG_TPS6507X is not set
> # CONFIG_TWL4030_CORE is not set
> # CONFIG_MFD_STMPE is not set
> # CONFIG_MFD_TC35892 is not set
> # CONFIG_MFD_TMIO is not set
> # CONFIG_PMIC_DA903X is not set
> # CONFIG_PMIC_ADP5520 is not set
> # CONFIG_MFD_MAX8925 is not set
> # CONFIG_MFD_MAX8998 is not set
> # CONFIG_MFD_WM8400 is not set
> # CONFIG_MFD_WM831X is not set
> # CONFIG_MFD_WM8350_I2C is not set
> # CONFIG_MFD_WM8994 is not set
> # CONFIG_MFD_PCF50633 is not set
> # CONFIG_ABX500_CORE is not set
> # CONFIG_LPC_SCH is not set
> # CONFIG_MFD_RDC321X is not set
> # CONFIG_MFD_JANZ_CMODIO is not set
> # CONFIG_REGULATOR is not set
> # CONFIG_MEDIA_SUPPORT is not set
> 
> #
> # Graphics support
> #
> CONFIG_AGP=y
> CONFIG_AGP_AMD64=y
> CONFIG_AGP_INTEL=y
> CONFIG_AGP_SIS=y
> CONFIG_AGP_VIA=y
> CONFIG_VGA_ARB=y
> CONFIG_VGA_ARB_MAX_GPUS=16
> # CONFIG_VGA_SWITCHEROO is not set
> CONFIG_DRM=m
> CONFIG_DRM_KMS_HELPER=m
> CONFIG_DRM_TTM=m
> CONFIG_DRM_TDFX=m
> CONFIG_DRM_R128=m
> CONFIG_DRM_RADEON=m
> CONFIG_DRM_I810=m
> CONFIG_DRM_I830=m
> CONFIG_DRM_I915=m
> # CONFIG_DRM_I915_KMS is not set
> CONFIG_DRM_MGA=m
> # CONFIG_DRM_SIS is not set
> CONFIG_DRM_VIA=m
> CONFIG_DRM_SAVAGE=m
> CONFIG_VGASTATE=m
> CONFIG_VIDEO_OUTPUT_CONTROL=m
> CONFIG_FB=y
> # CONFIG_FIRMWARE_EDID is not set
> CONFIG_FB_DDC=m
> # CONFIG_FB_BOOT_VESA_SUPPORT is not set
> CONFIG_FB_CFB_FILLRECT=m
> CONFIG_FB_CFB_COPYAREA=m
> CONFIG_FB_CFB_IMAGEBLIT=m
> # CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
> # CONFIG_FB_SYS_FILLRECT is not set
> # CONFIG_FB_SYS_COPYAREA is not set
> # CONFIG_FB_SYS_IMAGEBLIT is not set
> # CONFIG_FB_FOREIGN_ENDIAN is not set
> # CONFIG_FB_SYS_FOPS is not set
> CONFIG_FB_SVGALIB=m
> # CONFIG_FB_MACMODES is not set
> CONFIG_FB_BACKLIGHT=y
> CONFIG_FB_MODE_HELPERS=y
> CONFIG_FB_TILEBLITTING=y
> 
> #
> # Frame buffer hardware drivers
> #
> # CONFIG_FB_CIRRUS is not set
> # CONFIG_FB_PM2 is not set
> # CONFIG_FB_CYBER2000 is not set
> # CONFIG_FB_ARC is not set
> # CONFIG_FB_ASILIANT is not set
> # CONFIG_FB_IMSTT is not set
> # CONFIG_FB_VGA16 is not set
> # CONFIG_FB_UVESA is not set
> # CONFIG_FB_VESA is not set
> # CONFIG_FB_N411 is not set
> # CONFIG_FB_HGA is not set
> # CONFIG_FB_S1D13XXX is not set
> CONFIG_FB_NVIDIA=m
> CONFIG_FB_NVIDIA_I2C=y
> # CONFIG_FB_NVIDIA_DEBUG is not set
> CONFIG_FB_NVIDIA_BACKLIGHT=y
> CONFIG_FB_RIVA=m
> # CONFIG_FB_RIVA_I2C is not set
> # CONFIG_FB_RIVA_DEBUG is not set
> CONFIG_FB_RIVA_BACKLIGHT=y
> # CONFIG_FB_LE80578 is not set
> CONFIG_FB_MATROX=m
> CONFIG_FB_MATROX_MILLENIUM=y
> CONFIG_FB_MATROX_MYSTIQUE=y
> CONFIG_FB_MATROX_G=y
> CONFIG_FB_MATROX_I2C=m
> CONFIG_FB_MATROX_MAVEN=m
> # CONFIG_FB_RADEON is not set
> CONFIG_FB_ATY128=m
> CONFIG_FB_ATY128_BACKLIGHT=y
> CONFIG_FB_ATY=m
> CONFIG_FB_ATY_CT=y
> CONFIG_FB_ATY_GENERIC_LCD=y
> CONFIG_FB_ATY_GX=y
> CONFIG_FB_ATY_BACKLIGHT=y
> CONFIG_FB_S3=m
> CONFIG_FB_SAVAGE=m
> CONFIG_FB_SAVAGE_I2C=y
> CONFIG_FB_SAVAGE_ACCEL=y
> # CONFIG_FB_SIS is not set
> # CONFIG_FB_VIA is not set
> CONFIG_FB_NEOMAGIC=m
> CONFIG_FB_KYRO=m
> CONFIG_FB_3DFX=m
> CONFIG_FB_3DFX_ACCEL=y
> CONFIG_FB_3DFX_I2C=y
> CONFIG_FB_VOODOO1=m
> # CONFIG_FB_VT8623 is not set
> CONFIG_FB_TRIDENT=m
> # CONFIG_FB_ARK is not set
> # CONFIG_FB_PM3 is not set
> # CONFIG_FB_CARMINE is not set
> # CONFIG_FB_GEODE is not set
> CONFIG_FB_SM501=m
> # CONFIG_FB_VIRTUAL is not set
> # CONFIG_FB_METRONOME is not set
> # CONFIG_FB_MB862XX is not set
> # CONFIG_FB_BROADSHEET is not set
> CONFIG_BACKLIGHT_LCD_SUPPORT=y
> CONFIG_LCD_CLASS_DEVICE=m
> # CONFIG_LCD_PLATFORM is not set
> CONFIG_BACKLIGHT_CLASS_DEVICE=y
> CONFIG_BACKLIGHT_GENERIC=y
> CONFIG_BACKLIGHT_PROGEAR=m
> # CONFIG_BACKLIGHT_MBP_NVIDIA is not set
> # CONFIG_BACKLIGHT_SAHARA is not set
> # CONFIG_BACKLIGHT_ADP8860 is not set
> 
> #
> # Display device support
> #
> # CONFIG_DISPLAY_SUPPORT is not set
> 
> #
> # Console display driver support
> #
> CONFIG_VGA_CONSOLE=y
> CONFIG_VGACON_SOFT_SCROLLBACK=y
> CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
> CONFIG_DUMMY_CONSOLE=y
> CONFIG_FRAMEBUFFER_CONSOLE=m
> # CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
> # CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
> # CONFIG_FONTS is not set
> CONFIG_FONT_8x8=y
> CONFIG_FONT_8x16=y
> CONFIG_LOGO=y
> # CONFIG_LOGO_LINUX_MONO is not set
> # CONFIG_LOGO_LINUX_VGA16 is not set
> CONFIG_LOGO_LINUX_CLUT224=y
> CONFIG_SOUND=m
> CONFIG_SOUND_OSS_CORE=y
> CONFIG_SOUND_OSS_CORE_PRECLAIM=y
> CONFIG_SND=m
> CONFIG_SND_TIMER=m
> CONFIG_SND_PCM=m
> CONFIG_SND_HWDEP=m
> CONFIG_SND_RAWMIDI=m
> CONFIG_SND_JACK=y
> CONFIG_SND_SEQUENCER=m
> CONFIG_SND_SEQ_DUMMY=m
> CONFIG_SND_OSSEMUL=y
> CONFIG_SND_MIXER_OSS=m
> CONFIG_SND_PCM_OSS=m
> CONFIG_SND_PCM_OSS_PLUGINS=y
> CONFIG_SND_SEQUENCER_OSS=y
> CONFIG_SND_DYNAMIC_MINORS=y
> # CONFIG_SND_SUPPORT_OLD_API is not set
> CONFIG_SND_VERBOSE_PROCFS=y
> # CONFIG_SND_VERBOSE_PRINTK is not set
> # CONFIG_SND_DEBUG is not set
> CONFIG_SND_VMASTER=y
> CONFIG_SND_DMA_SGBUF=y
> CONFIG_SND_RAWMIDI_SEQ=m
> CONFIG_SND_OPL3_LIB_SEQ=m
> # CONFIG_SND_OPL4_LIB_SEQ is not set
> # CONFIG_SND_SBAWE_SEQ is not set
> CONFIG_SND_EMU10K1_SEQ=m
> CONFIG_SND_MPU401_UART=m
> CONFIG_SND_OPL3_LIB=m
> CONFIG_SND_VX_LIB=m
> CONFIG_SND_AC97_CODEC=m
> CONFIG_SND_DRIVERS=y
> CONFIG_SND_DUMMY=m
> CONFIG_SND_VIRMIDI=m
> CONFIG_SND_MTS64=m
> # CONFIG_SND_SERIAL_U16550 is not set
> CONFIG_SND_MPU401=m
> CONFIG_SND_PORTMAN2X4=m
> CONFIG_SND_AC97_POWER_SAVE=y
> CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
> CONFIG_SND_SB_COMMON=m
> CONFIG_SND_PCI=y
> CONFIG_SND_AD1889=m
> CONFIG_SND_ALS300=m
> CONFIG_SND_ALS4000=m
> CONFIG_SND_ALI5451=m
> # CONFIG_SND_ASIHPI is not set
> CONFIG_SND_ATIIXP=m
> CONFIG_SND_ATIIXP_MODEM=m
> CONFIG_SND_AU8810=m
> CONFIG_SND_AU8820=m
> CONFIG_SND_AU8830=m
> # CONFIG_SND_AW2 is not set
> CONFIG_SND_AZT3328=m
> CONFIG_SND_BT87X=m
> # CONFIG_SND_BT87X_OVERCLOCK is not set
> CONFIG_SND_CA0106=m
> CONFIG_SND_CMIPCI=m
> # CONFIG_SND_OXYGEN is not set
> CONFIG_SND_CS4281=m
> CONFIG_SND_CS46XX=m
> CONFIG_SND_CS46XX_NEW_DSP=y
> # CONFIG_SND_CS5530 is not set
> # CONFIG_SND_CS5535AUDIO is not set
> # CONFIG_SND_CTXFI is not set
> CONFIG_SND_DARLA20=m
> CONFIG_SND_GINA20=m
> CONFIG_SND_LAYLA20=m
> CONFIG_SND_DARLA24=m
> CONFIG_SND_GINA24=m
> CONFIG_SND_LAYLA24=m
> CONFIG_SND_MONA=m
> CONFIG_SND_MIA=m
> CONFIG_SND_ECHO3G=m
> CONFIG_SND_INDIGO=m
> CONFIG_SND_INDIGOIO=m
> CONFIG_SND_INDIGODJ=m
> # CONFIG_SND_INDIGOIOX is not set
> # CONFIG_SND_INDIGODJX is not set
> CONFIG_SND_EMU10K1=m
> CONFIG_SND_EMU10K1X=m
> CONFIG_SND_ENS1370=m
> CONFIG_SND_ENS1371=m
> CONFIG_SND_ES1938=m
> CONFIG_SND_ES1968=m
> # CONFIG_SND_ES1968_INPUT is not set
> CONFIG_SND_FM801=m
> CONFIG_SND_HDA_INTEL=m
> # CONFIG_SND_HDA_HWDEP is not set
> # CONFIG_SND_HDA_INPUT_BEEP is not set
> # CONFIG_SND_HDA_INPUT_JACK is not set
> # CONFIG_SND_HDA_PATCH_LOADER is not set
> CONFIG_SND_HDA_CODEC_REALTEK=y
> CONFIG_SND_HDA_CODEC_ANALOG=y
> CONFIG_SND_HDA_CODEC_SIGMATEL=y
> CONFIG_SND_HDA_CODEC_VIA=y
> CONFIG_SND_HDA_CODEC_ATIHDMI=y
> CONFIG_SND_HDA_CODEC_NVHDMI=y
> CONFIG_SND_HDA_CODEC_INTELHDMI=y
> CONFIG_SND_HDA_ELD=y
> CONFIG_SND_HDA_CODEC_CIRRUS=y
> CONFIG_SND_HDA_CODEC_CONEXANT=y
> CONFIG_SND_HDA_CODEC_CA0110=y
> CONFIG_SND_HDA_CODEC_CMEDIA=y
> CONFIG_SND_HDA_CODEC_SI3054=y
> CONFIG_SND_HDA_GENERIC=y
> # CONFIG_SND_HDA_POWER_SAVE is not set
> CONFIG_SND_HDSP=m
> CONFIG_SND_HDSPM=m
> # CONFIG_SND_HIFIER is not set
> CONFIG_SND_ICE1712=m
> CONFIG_SND_ICE1724=m
> CONFIG_SND_INTEL8X0=m
> CONFIG_SND_INTEL8X0M=m
> CONFIG_SND_KORG1212=m
> # CONFIG_SND_LX6464ES is not set
> CONFIG_SND_MAESTRO3=m
> # CONFIG_SND_MAESTRO3_INPUT is not set
> CONFIG_SND_MIXART=m
> CONFIG_SND_NM256=m
> CONFIG_SND_PCXHR=m
> CONFIG_SND_RIPTIDE=m
> CONFIG_SND_RME32=m
> CONFIG_SND_RME96=m
> CONFIG_SND_RME9652=m
> CONFIG_SND_SONICVIBES=m
> CONFIG_SND_TRIDENT=m
> CONFIG_SND_VIA82XX=m
> CONFIG_SND_VIA82XX_MODEM=m
> # CONFIG_SND_VIRTUOSO is not set
> CONFIG_SND_VX222=m
> CONFIG_SND_YMFPCI=m
> CONFIG_SND_USB=y
> CONFIG_SND_USB_AUDIO=m
> # CONFIG_SND_USB_UA101 is not set
> CONFIG_SND_USB_USX2Y=m
> # CONFIG_SND_USB_CAIAQ is not set
> # CONFIG_SND_USB_US122L is not set
> CONFIG_SND_PCMCIA=y
> # CONFIG_SND_VXPOCKET is not set
> # CONFIG_SND_PDAUDIOCF is not set
> CONFIG_SND_SOC=m
> CONFIG_SND_SOC_I2C_AND_SPI=m
> # CONFIG_SND_SOC_ALL_CODECS is not set
> # CONFIG_SOUND_PRIME is not set
> CONFIG_AC97_BUS=m
> CONFIG_HID_SUPPORT=y
> CONFIG_HID=y
> # CONFIG_HIDRAW is not set
> 
> #
> # USB Input Devices
> #
> CONFIG_USB_HID=y
> CONFIG_HID_PID=y
> CONFIG_USB_HIDDEV=y
> 
> #
> # Special HID drivers
> #
> # CONFIG_HID_3M_PCT is not set
> CONFIG_HID_A4TECH=y
> # CONFIG_HID_ACRUX_FF is not set
> CONFIG_HID_APPLE=y
> CONFIG_HID_BELKIN=y
> # CONFIG_HID_CANDO is not set
> CONFIG_HID_CHERRY=y
> CONFIG_HID_CHICONY=y
> # CONFIG_HID_PRODIKEYS is not set
> CONFIG_HID_CYPRESS=y
> CONFIG_HID_DRAGONRISE=y
> # CONFIG_DRAGONRISE_FF is not set
> # CONFIG_HID_EGALAX is not set
> # CONFIG_HID_ELECOM is not set
> CONFIG_HID_EZKEY=y
> CONFIG_HID_KYE=y
> CONFIG_HID_GYRATION=y
> CONFIG_HID_TWINHAN=y
> CONFIG_HID_KENSINGTON=y
> CONFIG_HID_LOGITECH=y
> CONFIG_LOGITECH_FF=y
> # CONFIG_LOGIRUMBLEPAD2_FF is not set
> # CONFIG_LOGIG940_FF is not set
> # CONFIG_HID_MAGICMOUSE is not set
> CONFIG_HID_MICROSOFT=y
> # CONFIG_HID_MOSART is not set
> CONFIG_HID_MONTEREY=y
> CONFIG_HID_NTRIG=y
> CONFIG_HID_ORTEK=y
> CONFIG_HID_PANTHERLORD=y
> CONFIG_PANTHERLORD_FF=y
> CONFIG_HID_PETALYNX=y
> # CONFIG_HID_PICOLCD is not set
> # CONFIG_HID_QUANTA is not set
> # CONFIG_HID_ROCCAT is not set
> # CONFIG_HID_ROCCAT_KONE is not set
> CONFIG_HID_SAMSUNG=y
> CONFIG_HID_SONY=y
> # CONFIG_HID_STANTUM is not set
> CONFIG_HID_SUNPLUS=y
> CONFIG_HID_GREENASIA=y
> # CONFIG_GREENASIA_FF is not set
> CONFIG_HID_SMARTJOYPLUS=y
> # CONFIG_SMARTJOYPLUS_FF is not set
> CONFIG_HID_TOPSEED=y
> CONFIG_HID_THRUSTMASTER=y
> CONFIG_THRUSTMASTER_FF=y
> # CONFIG_HID_WACOM is not set
> CONFIG_HID_ZEROPLUS=y
> CONFIG_ZEROPLUS_FF=y
> # CONFIG_HID_ZYDACRON is not set
> CONFIG_USB_SUPPORT=y
> CONFIG_USB_ARCH_HAS_HCD=y
> CONFIG_USB_ARCH_HAS_OHCI=y
> CONFIG_USB_ARCH_HAS_EHCI=y
> CONFIG_USB=y
> # CONFIG_USB_DEBUG is not set
> # CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set
> 
> #
> # Miscellaneous USB options
> #
> CONFIG_USB_DEVICEFS=y
> CONFIG_USB_DEVICE_CLASS=y
> # CONFIG_USB_DYNAMIC_MINORS is not set
> CONFIG_USB_MON=y
> # CONFIG_USB_WUSB is not set
> # CONFIG_USB_WUSB_CBAF is not set
> 
> #
> # USB Host Controller Drivers
> #
> # CONFIG_USB_C67X00_HCD is not set
> # CONFIG_USB_XHCI_HCD is not set
> CONFIG_USB_EHCI_HCD=y
> CONFIG_USB_EHCI_ROOT_HUB_TT=y
> CONFIG_USB_EHCI_TT_NEWSCHED=y
> # CONFIG_USB_OXU210HP_HCD is not set
> CONFIG_USB_ISP116X_HCD=m
> # CONFIG_USB_ISP1760_HCD is not set
> # CONFIG_USB_ISP1362_HCD is not set
> CONFIG_USB_OHCI_HCD=y
> # CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
> # CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
> CONFIG_USB_OHCI_LITTLE_ENDIAN=y
> CONFIG_USB_UHCI_HCD=y
> CONFIG_USB_U132_HCD=m
> CONFIG_USB_SL811_HCD=m
> CONFIG_USB_SL811_CS=m
> # CONFIG_USB_R8A66597_HCD is not set
> # CONFIG_USB_HWA_HCD is not set
> 
> #
> # USB Device Class drivers
> #
> CONFIG_USB_ACM=m
> CONFIG_USB_PRINTER=m
> # CONFIG_USB_WDM is not set
> # CONFIG_USB_TMC is not set
> 
> #
> # NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
> #
> 
> #
> # also be needed; see USB_STORAGE Help for more info
> #
> CONFIG_USB_STORAGE=m
> # CONFIG_USB_STORAGE_DEBUG is not set
> CONFIG_USB_STORAGE_DATAFAB=m
> CONFIG_USB_STORAGE_FREECOM=m
> CONFIG_USB_STORAGE_ISD200=m
> CONFIG_USB_STORAGE_USBAT=m
> CONFIG_USB_STORAGE_SDDR09=m
> CONFIG_USB_STORAGE_SDDR55=m
> CONFIG_USB_STORAGE_JUMPSHOT=m
> CONFIG_USB_STORAGE_ALAUDA=m
> # CONFIG_USB_STORAGE_ONETOUCH is not set
> CONFIG_USB_STORAGE_KARMA=m
> # CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
> CONFIG_USB_LIBUSUAL=y
> 
> #
> # USB Imaging devices
> #
> CONFIG_USB_MDC800=m
> CONFIG_USB_MICROTEK=m
> 
> #
> # USB port drivers
> #
> CONFIG_USB_USS720=m
> CONFIG_USB_SERIAL=m
> CONFIG_USB_EZUSB=y
> CONFIG_USB_SERIAL_GENERIC=y
> CONFIG_USB_SERIAL_AIRCABLE=m
> CONFIG_USB_SERIAL_ARK3116=m
> CONFIG_USB_SERIAL_BELKIN=m
> # CONFIG_USB_SERIAL_CH341 is not set
> CONFIG_USB_SERIAL_WHITEHEAT=m
> CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
> # CONFIG_USB_SERIAL_CP210X is not set
> CONFIG_USB_SERIAL_CYPRESS_M8=m
> CONFIG_USB_SERIAL_EMPEG=m
> CONFIG_USB_SERIAL_FTDI_SIO=m
> CONFIG_USB_SERIAL_FUNSOFT=m
> CONFIG_USB_SERIAL_VISOR=m
> CONFIG_USB_SERIAL_IPAQ=m
> CONFIG_USB_SERIAL_IR=m
> CONFIG_USB_SERIAL_EDGEPORT=m
> CONFIG_USB_SERIAL_EDGEPORT_TI=m
> CONFIG_USB_SERIAL_GARMIN=m
> CONFIG_USB_SERIAL_IPW=m
> # CONFIG_USB_SERIAL_IUU is not set
> CONFIG_USB_SERIAL_KEYSPAN_PDA=m
> CONFIG_USB_SERIAL_KEYSPAN=m
> CONFIG_USB_SERIAL_KEYSPAN_MPR=y
> CONFIG_USB_SERIAL_KEYSPAN_USA28=y
> CONFIG_USB_SERIAL_KEYSPAN_USA28X=y
> CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y
> CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y
> CONFIG_USB_SERIAL_KEYSPAN_USA19=y
> CONFIG_USB_SERIAL_KEYSPAN_USA18X=y
> CONFIG_USB_SERIAL_KEYSPAN_USA19W=y
> CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y
> CONFIG_USB_SERIAL_KEYSPAN_USA19QI=y
> CONFIG_USB_SERIAL_KEYSPAN_USA49W=y
> CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y
> CONFIG_USB_SERIAL_KLSI=m
> CONFIG_USB_SERIAL_KOBIL_SCT=m
> CONFIG_USB_SERIAL_MCT_U232=m
> CONFIG_USB_SERIAL_MOS7720=m
> # CONFIG_USB_SERIAL_MOS7715_PARPORT is not set
> CONFIG_USB_SERIAL_MOS7840=m
> # CONFIG_USB_SERIAL_MOTOROLA is not set
> CONFIG_USB_SERIAL_NAVMAN=m
> CONFIG_USB_SERIAL_PL2303=m
> # CONFIG_USB_SERIAL_OTI6858 is not set
> # CONFIG_USB_SERIAL_QCAUX is not set
> # CONFIG_USB_SERIAL_QUALCOMM is not set
> # CONFIG_USB_SERIAL_SPCP8X5 is not set
> CONFIG_USB_SERIAL_HP4X=m
> CONFIG_USB_SERIAL_SAFE=m
> CONFIG_USB_SERIAL_SAFE_PADDED=y
> # CONFIG_USB_SERIAL_SIEMENS_MPI is not set
> CONFIG_USB_SERIAL_SIERRAWIRELESS=m
> # CONFIG_USB_SERIAL_SYMBOL is not set
> CONFIG_USB_SERIAL_TI=m
> CONFIG_USB_SERIAL_CYBERJACK=m
> CONFIG_USB_SERIAL_XIRCOM=m
> CONFIG_USB_SERIAL_WWAN=m
> CONFIG_USB_SERIAL_OPTION=m
> CONFIG_USB_SERIAL_OMNINET=m
> # CONFIG_USB_SERIAL_OPTICON is not set
> # CONFIG_USB_SERIAL_VIVOPAY_SERIAL is not set
> # CONFIG_USB_SERIAL_ZIO is not set
> # CONFIG_USB_SERIAL_SSU100 is not set
> CONFIG_USB_SERIAL_DEBUG=m
> 
> #
> # USB Miscellaneous drivers
> #
> CONFIG_USB_EMI62=m
> CONFIG_USB_EMI26=m
> CONFIG_USB_ADUTUX=m
> # CONFIG_USB_SEVSEG is not set
> CONFIG_USB_RIO500=m
> CONFIG_USB_LEGOTOWER=m
> CONFIG_USB_LCD=m
> CONFIG_USB_LED=m
> # CONFIG_USB_CYPRESS_CY7C63 is not set
> # CONFIG_USB_CYTHERM is not set
> CONFIG_USB_IDMOUSE=m
> CONFIG_USB_FTDI_ELAN=m
> CONFIG_USB_APPLEDISPLAY=m
> CONFIG_USB_SISUSBVGA=m
> CONFIG_USB_SISUSBVGA_CON=y
> CONFIG_USB_LD=m
> CONFIG_USB_TRANCEVIBRATOR=m
> CONFIG_USB_IOWARRIOR=m
> CONFIG_USB_TEST=m
> # CONFIG_USB_ISIGHTFW is not set
> CONFIG_USB_ATM=m
> CONFIG_USB_SPEEDTOUCH=m
> CONFIG_USB_CXACRU=m
> CONFIG_USB_UEAGLEATM=m
> CONFIG_USB_XUSBATM=m
> # CONFIG_USB_GADGET is not set
> 
> #
> # OTG and related infrastructure
> #
> # CONFIG_NOP_USB_XCEIV is not set
> # CONFIG_UWB is not set
> CONFIG_MMC=m
> # CONFIG_MMC_DEBUG is not set
> # CONFIG_MMC_UNSAFE_RESUME is not set
> 
> #
> # MMC/SD/SDIO Card Drivers
> #
> CONFIG_MMC_BLOCK=m
> CONFIG_MMC_BLOCK_BOUNCE=y
> # CONFIG_SDIO_UART is not set
> # CONFIG_MMC_TEST is not set
> 
> #
> # MMC/SD/SDIO Host Controller Drivers
> #
> CONFIG_MMC_SDHCI=m
> # CONFIG_MMC_SDHCI_PCI is not set
> # CONFIG_MMC_SDHCI_PLTFM is not set
> CONFIG_MMC_WBSD=m
> CONFIG_MMC_TIFM_SD=m
> # CONFIG_MMC_SDRICOH_CS is not set
> # CONFIG_MMC_CB710 is not set
> # CONFIG_MMC_VIA_SDMMC is not set
> # CONFIG_MEMSTICK is not set
> CONFIG_NEW_LEDS=y
> CONFIG_LEDS_CLASS=y
> 
> #
> # LED drivers
> #
> # CONFIG_LEDS_ALIX2 is not set
> # CONFIG_LEDS_PCA9532 is not set
> # CONFIG_LEDS_LP3944 is not set
> # CONFIG_LEDS_CLEVO_MAIL is not set
> # CONFIG_LEDS_PCA955X is not set
> # CONFIG_LEDS_BD2802 is not set
> CONFIG_LEDS_TRIGGERS=y
> 
> #
> # LED Triggers
> #
> CONFIG_LEDS_TRIGGER_TIMER=m
> CONFIG_LEDS_TRIGGER_HEARTBEAT=m
> # CONFIG_LEDS_TRIGGER_BACKLIGHT is not set
> # CONFIG_LEDS_TRIGGER_DEFAULT_ON is not set
> 
> #
> # iptables trigger is under Netfilter config (LED target)
> #
> # CONFIG_ACCESSIBILITY is not set
> # CONFIG_INFINIBAND is not set
> CONFIG_EDAC=y
> 
> #
> # Reporting subsystems
> #
> # CONFIG_EDAC_DEBUG is not set
> CONFIG_EDAC_DECODE_MCE=y
> CONFIG_EDAC_MM_EDAC=m
> # CONFIG_EDAC_AMD64 is not set
> CONFIG_EDAC_E752X=m
> # CONFIG_EDAC_I82975X is not set
> # CONFIG_EDAC_I3000 is not set
> # CONFIG_EDAC_I3200 is not set
> # CONFIG_EDAC_X38 is not set
> # CONFIG_EDAC_I5400 is not set
> # CONFIG_EDAC_I7CORE is not set
> # CONFIG_EDAC_I5000 is not set
> # CONFIG_EDAC_I5100 is not set
> CONFIG_RTC_LIB=m
> CONFIG_RTC_CLASS=m
> 
> #
> # RTC interfaces
> #
> CONFIG_RTC_INTF_SYSFS=y
> CONFIG_RTC_INTF_PROC=y
> CONFIG_RTC_INTF_DEV=y
> # CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
> # CONFIG_RTC_DRV_TEST is not set
> 
> #
> # I2C RTC drivers
> #
> CONFIG_RTC_DRV_DS1307=m
> # CONFIG_RTC_DRV_DS1374 is not set
> CONFIG_RTC_DRV_DS1672=m
> # CONFIG_RTC_DRV_DS3232 is not set
> # CONFIG_RTC_DRV_MAX6900 is not set
> CONFIG_RTC_DRV_RS5C372=m
> CONFIG_RTC_DRV_ISL1208=m
> # CONFIG_RTC_DRV_ISL12022 is not set
> CONFIG_RTC_DRV_X1205=m
> CONFIG_RTC_DRV_PCF8563=m
> # CONFIG_RTC_DRV_PCF8583 is not set
> # CONFIG_RTC_DRV_M41T80 is not set
> # CONFIG_RTC_DRV_BQ32K is not set
> # CONFIG_RTC_DRV_S35390A is not set
> # CONFIG_RTC_DRV_FM3130 is not set
> # CONFIG_RTC_DRV_RX8581 is not set
> # CONFIG_RTC_DRV_RX8025 is not set
> 
> #
> # SPI RTC drivers
> #
> 
> #
> # Platform RTC drivers
> #
> CONFIG_RTC_DRV_CMOS=m
> # CONFIG_RTC_DRV_DS1286 is not set
> # CONFIG_RTC_DRV_DS1511 is not set
> CONFIG_RTC_DRV_DS1553=m
> CONFIG_RTC_DRV_DS1742=m
> # CONFIG_RTC_DRV_STK17TA8 is not set
> # CONFIG_RTC_DRV_M48T86 is not set
> # CONFIG_RTC_DRV_M48T35 is not set
> # CONFIG_RTC_DRV_M48T59 is not set
> # CONFIG_RTC_DRV_MSM6242 is not set
> # CONFIG_RTC_DRV_BQ4802 is not set
> # CONFIG_RTC_DRV_RP5C01 is not set
> CONFIG_RTC_DRV_V3020=m
> 
> #
> # on-CPU RTC drivers
> #
> # CONFIG_DMADEVICES is not set
> # CONFIG_AUXDISPLAY is not set
> # CONFIG_UIO is not set
> # CONFIG_STAGING is not set
> CONFIG_X86_PLATFORM_DEVICES=y
> # CONFIG_ACER_WMI is not set
> CONFIG_ASUS_LAPTOP=m
> # CONFIG_DELL_LAPTOP is not set
> # CONFIG_FUJITSU_LAPTOP is not set
> CONFIG_MSI_LAPTOP=m
> # CONFIG_PANASONIC_LAPTOP is not set
> # CONFIG_COMPAL_LAPTOP is not set
> CONFIG_SONY_LAPTOP=m
> # CONFIG_SONYPI_COMPAT is not set
> # CONFIG_IDEAPAD_ACPI is not set
> # CONFIG_THINKPAD_ACPI is not set
> # CONFIG_INTEL_MENLOW is not set
> # CONFIG_EEEPC_LAPTOP is not set
> # CONFIG_ACPI_WMI is not set
> CONFIG_ACPI_ASUS=m
> # CONFIG_TOPSTAR_LAPTOP is not set
> CONFIG_ACPI_TOSHIBA=m
> # CONFIG_TOSHIBA_BT_RFKILL is not set
> # CONFIG_ACPI_CMPC is not set
> # CONFIG_INTEL_IPS is not set
> 
> #
> # Firmware Drivers
> #
> CONFIG_EDD=m
> # CONFIG_EDD_OFF is not set
> CONFIG_FIRMWARE_MEMMAP=y
> CONFIG_DELL_RBU=m
> CONFIG_DCDBAS=m
> CONFIG_DMIID=y
> # CONFIG_ISCSI_IBFT_FIND is not set
> 
> #
> # File systems
> #
> CONFIG_EXT2_FS=y
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT2_FS_POSIX_ACL=y
> CONFIG_EXT2_FS_SECURITY=y
> CONFIG_EXT2_FS_XIP=y
> CONFIG_EXT3_FS=y
> # CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
> CONFIG_EXT3_FS_XATTR=y
> CONFIG_EXT3_FS_POSIX_ACL=y
> CONFIG_EXT3_FS_SECURITY=y
> # CONFIG_EXT4_FS is not set
> CONFIG_FS_XIP=y
> CONFIG_JBD=y
> # CONFIG_JBD_DEBUG is not set
> CONFIG_JBD2=m
> # CONFIG_JBD2_DEBUG is not set
> CONFIG_FS_MBCACHE=y
> CONFIG_REISERFS_FS=m
> # CONFIG_REISERFS_CHECK is not set
> CONFIG_REISERFS_PROC_INFO=y
> CONFIG_REISERFS_FS_XATTR=y
> CONFIG_REISERFS_FS_POSIX_ACL=y
> CONFIG_REISERFS_FS_SECURITY=y
> CONFIG_JFS_FS=m
> CONFIG_JFS_POSIX_ACL=y
> CONFIG_JFS_SECURITY=y
> # CONFIG_JFS_DEBUG is not set
> # CONFIG_JFS_STATISTICS is not set
> CONFIG_FS_POSIX_ACL=y
> CONFIG_XFS_FS=m
> CONFIG_XFS_QUOTA=y
> CONFIG_XFS_POSIX_ACL=y
> # CONFIG_XFS_RT is not set
> # CONFIG_XFS_DEBUG is not set
> CONFIG_GFS2_FS=m
> # CONFIG_GFS2_FS_LOCKING_DLM is not set
> CONFIG_OCFS2_FS=m
> CONFIG_OCFS2_FS_O2CB=m
> CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
> CONFIG_OCFS2_FS_STATS=y
> # CONFIG_OCFS2_DEBUG_MASKLOG is not set
> # CONFIG_OCFS2_DEBUG_FS is not set
> # CONFIG_BTRFS_FS is not set
> # CONFIG_NILFS2_FS is not set
> CONFIG_FILE_LOCKING=y
> CONFIG_FSNOTIFY=y
> CONFIG_DNOTIFY=y
> CONFIG_INOTIFY_USER=y
> # CONFIG_FANOTIFY is not set
> CONFIG_QUOTA=y
> # CONFIG_QUOTA_NETLINK_INTERFACE is not set
> CONFIG_PRINT_QUOTA_WARNING=y
> # CONFIG_QUOTA_DEBUG is not set
> CONFIG_QUOTA_TREE=y
> # CONFIG_QFMT_V1 is not set
> CONFIG_QFMT_V2=y
> CONFIG_QUOTACTL=y
> CONFIG_QUOTACTL_COMPAT=y
> CONFIG_AUTOFS_FS=m
> CONFIG_AUTOFS4_FS=m
> CONFIG_FUSE_FS=m
> # CONFIG_CUSE is not set
> CONFIG_GENERIC_ACL=y
> 
> #
> # Caches
> #
> # CONFIG_FSCACHE is not set
> 
> #
> # CD-ROM/DVD Filesystems
> #
> CONFIG_ISO9660_FS=y
> CONFIG_JOLIET=y
> CONFIG_ZISOFS=y
> CONFIG_UDF_FS=m
> CONFIG_UDF_NLS=y
> 
> #
> # DOS/FAT/NT Filesystems
> #
> CONFIG_FAT_FS=m
> CONFIG_MSDOS_FS=m
> CONFIG_VFAT_FS=m
> CONFIG_FAT_DEFAULT_CODEPAGE=437
> CONFIG_FAT_DEFAULT_IOCHARSET="ascii"
> # CONFIG_NTFS_FS is not set
> 
> #
> # Pseudo filesystems
> #
> CONFIG_PROC_FS=y
> CONFIG_PROC_KCORE=y
> CONFIG_PROC_SYSCTL=y
> CONFIG_PROC_PAGE_MONITOR=y
> CONFIG_SYSFS=y
> CONFIG_TMPFS=y
> CONFIG_TMPFS_POSIX_ACL=y
> CONFIG_HUGETLBFS=y
> CONFIG_HUGETLB_PAGE=y
> CONFIG_CONFIGFS_FS=m
> CONFIG_MISC_FILESYSTEMS=y
> # CONFIG_ADFS_FS is not set
> CONFIG_AFFS_FS=m
> # CONFIG_ECRYPT_FS is not set
> CONFIG_HFS_FS=m
> CONFIG_HFSPLUS_FS=m
> CONFIG_BEFS_FS=m
> # CONFIG_BEFS_DEBUG is not set
> CONFIG_BFS_FS=m
> CONFIG_EFS_FS=m
> # CONFIG_LOGFS is not set
> CONFIG_CRAMFS=m
> # CONFIG_SQUASHFS is not set
> CONFIG_VXFS_FS=m
> CONFIG_MINIX_FS=m
> # CONFIG_OMFS_FS is not set
> # CONFIG_HPFS_FS is not set
> CONFIG_QNX4FS_FS=m
> CONFIG_ROMFS_FS=m
> CONFIG_ROMFS_BACKED_BY_BLOCK=y
> CONFIG_ROMFS_ON_BLOCK=y
> CONFIG_SYSV_FS=m
> CONFIG_UFS_FS=m
> # CONFIG_UFS_FS_WRITE is not set
> # CONFIG_UFS_DEBUG is not set
> CONFIG_NETWORK_FILESYSTEMS=y
> CONFIG_NFS_FS=m
> CONFIG_NFS_V3=y
> CONFIG_NFS_V3_ACL=y
> CONFIG_NFS_V4=y
> # CONFIG_NFS_V4_1 is not set
> # CONFIG_NFS_USE_LEGACY_DNS is not set
> CONFIG_NFS_USE_KERNEL_DNS=y
> CONFIG_NFSD=m
> CONFIG_NFSD_V2_ACL=y
> CONFIG_NFSD_V3=y
> CONFIG_NFSD_V3_ACL=y
> CONFIG_NFSD_V4=y
> CONFIG_LOCKD=m
> CONFIG_LOCKD_V4=y
> CONFIG_EXPORTFS=m
> CONFIG_NFS_ACL_SUPPORT=m
> CONFIG_NFS_COMMON=y
> CONFIG_SUNRPC=m
> CONFIG_SUNRPC_GSS=m
> CONFIG_RPCSEC_GSS_KRB5=m
> CONFIG_RPCSEC_GSS_SPKM3=m
> # CONFIG_SMB_FS is not set
> # CONFIG_CEPH_FS is not set
> CONFIG_CIFS=m
> # CONFIG_CIFS_STATS is not set
> CONFIG_CIFS_WEAK_PW_HASH=y
> # CONFIG_CIFS_UPCALL is not set
> CONFIG_CIFS_XATTR=y
> CONFIG_CIFS_POSIX=y
> # CONFIG_CIFS_DEBUG2 is not set
> # CONFIG_CIFS_DFS_UPCALL is not set
> # CONFIG_CIFS_EXPERIMENTAL is not set
> CONFIG_NCP_FS=m
> CONFIG_NCPFS_PACKET_SIGNING=y
> CONFIG_NCPFS_IOCTL_LOCKING=y
> CONFIG_NCPFS_STRONG=y
> CONFIG_NCPFS_NFS_NS=y
> CONFIG_NCPFS_OS2_NS=y
> CONFIG_NCPFS_SMALLDOS=y
> CONFIG_NCPFS_NLS=y
> CONFIG_NCPFS_EXTRAS=y
> CONFIG_CODA_FS=m
> # CONFIG_AFS_FS is not set
> 
> #
> # Partition Types
> #
> CONFIG_PARTITION_ADVANCED=y
> # CONFIG_ACORN_PARTITION is not set
> CONFIG_OSF_PARTITION=y
> CONFIG_AMIGA_PARTITION=y
> # CONFIG_ATARI_PARTITION is not set
> CONFIG_MAC_PARTITION=y
> CONFIG_MSDOS_PARTITION=y
> CONFIG_BSD_DISKLABEL=y
> CONFIG_MINIX_SUBPARTITION=y
> CONFIG_SOLARIS_X86_PARTITION=y
> CONFIG_UNIXWARE_DISKLABEL=y
> # CONFIG_LDM_PARTITION is not set
> CONFIG_SGI_PARTITION=y
> # CONFIG_ULTRIX_PARTITION is not set
> CONFIG_SUN_PARTITION=y
> CONFIG_KARMA_PARTITION=y
> CONFIG_EFI_PARTITION=y
> # CONFIG_SYSV68_PARTITION is not set
> CONFIG_NLS=y
> CONFIG_NLS_DEFAULT="utf8"
> CONFIG_NLS_CODEPAGE_437=y
> CONFIG_NLS_CODEPAGE_737=m
> CONFIG_NLS_CODEPAGE_775=m
> CONFIG_NLS_CODEPAGE_850=m
> CONFIG_NLS_CODEPAGE_852=m
> CONFIG_NLS_CODEPAGE_855=m
> CONFIG_NLS_CODEPAGE_857=m
> CONFIG_NLS_CODEPAGE_860=m
> CONFIG_NLS_CODEPAGE_861=m
> CONFIG_NLS_CODEPAGE_862=m
> CONFIG_NLS_CODEPAGE_863=m
> CONFIG_NLS_CODEPAGE_864=m
> CONFIG_NLS_CODEPAGE_865=m
> CONFIG_NLS_CODEPAGE_866=m
> CONFIG_NLS_CODEPAGE_869=m
> CONFIG_NLS_CODEPAGE_936=m
> CONFIG_NLS_CODEPAGE_950=m
> CONFIG_NLS_CODEPAGE_932=m
> CONFIG_NLS_CODEPAGE_949=m
> CONFIG_NLS_CODEPAGE_874=m
> CONFIG_NLS_ISO8859_8=m
> CONFIG_NLS_CODEPAGE_1250=m
> CONFIG_NLS_CODEPAGE_1251=m
> CONFIG_NLS_ASCII=y
> CONFIG_NLS_ISO8859_1=m
> CONFIG_NLS_ISO8859_2=m
> CONFIG_NLS_ISO8859_3=m
> CONFIG_NLS_ISO8859_4=m
> CONFIG_NLS_ISO8859_5=m
> CONFIG_NLS_ISO8859_6=m
> CONFIG_NLS_ISO8859_7=m
> CONFIG_NLS_ISO8859_9=m
> CONFIG_NLS_ISO8859_13=m
> CONFIG_NLS_ISO8859_14=m
> CONFIG_NLS_ISO8859_15=m
> CONFIG_NLS_KOI8_R=m
> CONFIG_NLS_KOI8_U=m
> CONFIG_NLS_UTF8=m
> CONFIG_DLM=m
> CONFIG_DLM_DEBUG=y
> 
> #
> # Kernel hacking
> #
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> # CONFIG_PRINTK_TIME is not set
> # CONFIG_ENABLE_WARN_DEPRECATED is not set
> # CONFIG_ENABLE_MUST_CHECK is not set
> CONFIG_FRAME_WARN=2048
> CONFIG_MAGIC_SYSRQ=y
> # CONFIG_STRIP_ASM_SYMS is not set
> # CONFIG_UNUSED_SYMBOLS is not set
> CONFIG_DEBUG_FS=y
> # CONFIG_HEADERS_CHECK is not set
> CONFIG_DEBUG_KERNEL=y
> # CONFIG_DEBUG_SHIRQ is not set
> # CONFIG_LOCKUP_DETECTOR is not set
> # CONFIG_HARDLOCKUP_DETECTOR is not set
> CONFIG_DETECT_HUNG_TASK=y
> # CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
> CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
> CONFIG_SCHED_DEBUG=y
> # CONFIG_SCHEDSTATS is not set
> # CONFIG_TIMER_STATS is not set
> # CONFIG_DEBUG_OBJECTS is not set
> # CONFIG_DEBUG_SLAB is not set
> # CONFIG_DEBUG_RT_MUTEXES is not set
> # CONFIG_RT_MUTEX_TESTER is not set
> # CONFIG_DEBUG_SPINLOCK is not set
> # CONFIG_DEBUG_MUTEXES is not set
> # CONFIG_DEBUG_LOCK_ALLOC is not set
> # CONFIG_PROVE_LOCKING is not set
> # CONFIG_SPARSE_RCU_POINTER is not set
> # CONFIG_LOCK_STAT is not set
> # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
> # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
> CONFIG_STACKTRACE=y
> # CONFIG_DEBUG_KOBJECT is not set
> CONFIG_DEBUG_BUGVERBOSE=y
> # CONFIG_DEBUG_INFO is not set
> # CONFIG_DEBUG_VM is not set
> # CONFIG_DEBUG_VIRTUAL is not set
> # CONFIG_DEBUG_WRITECOUNT is not set
> CONFIG_DEBUG_MEMORY_INIT=y
> # CONFIG_DEBUG_LIST is not set
> # CONFIG_DEBUG_SG is not set
> # CONFIG_DEBUG_NOTIFIERS is not set
> # CONFIG_DEBUG_CREDENTIALS is not set
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_FRAME_POINTER=y
> # CONFIG_BOOT_PRINTK_DELAY is not set
> # CONFIG_RCU_TORTURE_TEST is not set
> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> # CONFIG_KPROBES_SANITY_TEST is not set
> # CONFIG_BACKTRACE_SELF_TEST is not set
> # CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
> # CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
> # CONFIG_LKDTM is not set
> # CONFIG_CPU_NOTIFIER_ERROR_INJECT is not set
> # CONFIG_FAULT_INJECTION is not set
> # CONFIG_LATENCYTOP is not set
> CONFIG_SYSCTL_SYSCALL_CHECK=y
> # CONFIG_DEBUG_PAGEALLOC is not set
> CONFIG_USER_STACKTRACE_SUPPORT=y
> CONFIG_NOP_TRACER=y
> CONFIG_HAVE_FUNCTION_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
> CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
> CONFIG_HAVE_DYNAMIC_FTRACE=y
> CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
> CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
> CONFIG_RING_BUFFER=y
> CONFIG_EVENT_TRACING=y
> CONFIG_CONTEXT_SWITCH_TRACER=y
> CONFIG_RING_BUFFER_ALLOW_SWAP=y
> CONFIG_TRACING=y
> CONFIG_TRACING_SUPPORT=y
> CONFIG_FTRACE=y
> # CONFIG_FUNCTION_TRACER is not set
> # CONFIG_IRQSOFF_TRACER is not set
> # CONFIG_SCHED_TRACER is not set
> CONFIG_ENABLE_DEFAULT_TRACERS=y
> # CONFIG_FTRACE_SYSCALLS is not set
> CONFIG_BRANCH_PROFILE_NONE=y
> # CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
> # CONFIG_PROFILE_ALL_BRANCHES is not set
> # CONFIG_STACK_TRACER is not set
> # CONFIG_BLK_DEV_IO_TRACE is not set
> # CONFIG_KPROBE_EVENT is not set
> # CONFIG_MMIOTRACE is not set
> # CONFIG_RING_BUFFER_BENCHMARK is not set
> # CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
> # CONFIG_DYNAMIC_DEBUG is not set
> # CONFIG_DMA_API_DEBUG is not set
> # CONFIG_ATOMIC64_SELFTEST is not set
> # CONFIG_SAMPLES is not set
> CONFIG_HAVE_ARCH_KGDB=y
> # CONFIG_KGDB is not set
> CONFIG_HAVE_ARCH_KMEMCHECK=y
> # CONFIG_STRICT_DEVMEM is not set
> CONFIG_X86_VERBOSE_BOOTUP=y
> CONFIG_EARLY_PRINTK=y
> # CONFIG_EARLY_PRINTK_DBGP is not set
> # CONFIG_DEBUG_STACKOVERFLOW is not set
> # CONFIG_DEBUG_STACK_USAGE is not set
> # CONFIG_DEBUG_PER_CPU_MAPS is not set
> # CONFIG_X86_PTDUMP is not set
> CONFIG_DEBUG_RODATA=y
> CONFIG_DEBUG_RODATA_TEST=y
> # CONFIG_DEBUG_NX_TEST is not set
> # CONFIG_IOMMU_DEBUG is not set
> # CONFIG_IOMMU_STRESS is not set
> CONFIG_HAVE_MMIOTRACE_SUPPORT=y
> # CONFIG_X86_DECODER_SELFTEST is not set
> CONFIG_IO_DELAY_TYPE_0X80=0
> CONFIG_IO_DELAY_TYPE_0XED=1
> CONFIG_IO_DELAY_TYPE_UDELAY=2
> CONFIG_IO_DELAY_TYPE_NONE=3
> CONFIG_IO_DELAY_0X80=y
> # CONFIG_IO_DELAY_0XED is not set
> # CONFIG_IO_DELAY_UDELAY is not set
> # CONFIG_IO_DELAY_NONE is not set
> CONFIG_DEFAULT_IO_DELAY_TYPE=0
> # CONFIG_DEBUG_BOOT_PARAMS is not set
> # CONFIG_CPA_DEBUG is not set
> CONFIG_OPTIMIZE_INLINING=y
> 
> #
> # Security options
> #
> CONFIG_KEYS=y
> CONFIG_KEYS_DEBUG_PROC_KEYS=y
> CONFIG_SECURITY=y
> CONFIG_SECURITYFS=y
> CONFIG_SECURITY_NETWORK=y
> # CONFIG_SECURITY_NETWORK_XFRM is not set
> # CONFIG_SECURITY_PATH is not set
> CONFIG_LSM_MMAP_MIN_ADDR=65536
> CONFIG_SECURITY_SELINUX=y
> CONFIG_SECURITY_SELINUX_BOOTPARAM=y
> CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
> CONFIG_SECURITY_SELINUX_DISABLE=y
> CONFIG_SECURITY_SELINUX_DEVELOP=y
> CONFIG_SECURITY_SELINUX_AVC_STATS=y
> CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
> # CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
> # CONFIG_SECURITY_SMACK is not set
> # CONFIG_SECURITY_TOMOYO is not set
> # CONFIG_SECURITY_APPARMOR is not set
> # CONFIG_IMA is not set
> CONFIG_DEFAULT_SECURITY_SELINUX=y
> # CONFIG_DEFAULT_SECURITY_DAC is not set
> CONFIG_DEFAULT_SECURITY="selinux"
> CONFIG_XOR_BLOCKS=m
> CONFIG_ASYNC_CORE=m
> CONFIG_ASYNC_MEMCPY=m
> CONFIG_ASYNC_XOR=m
> CONFIG_ASYNC_PQ=m
> CONFIG_ASYNC_RAID6_RECOV=m
> # CONFIG_ASYNC_RAID6_TEST is not set
> CONFIG_CRYPTO=y
> 
> #
> # Crypto core or helper
> #
> CONFIG_CRYPTO_ALGAPI=y
> CONFIG_CRYPTO_ALGAPI2=y
> CONFIG_CRYPTO_AEAD=m
> CONFIG_CRYPTO_AEAD2=y
> CONFIG_CRYPTO_BLKCIPHER=m
> CONFIG_CRYPTO_BLKCIPHER2=y
> CONFIG_CRYPTO_HASH=y
> CONFIG_CRYPTO_HASH2=y
> CONFIG_CRYPTO_RNG2=y
> CONFIG_CRYPTO_PCOMP2=y
> CONFIG_CRYPTO_MANAGER=y
> CONFIG_CRYPTO_MANAGER2=y
> CONFIG_CRYPTO_MANAGER_TESTS=y
> CONFIG_CRYPTO_GF128MUL=m
> CONFIG_CRYPTO_NULL=m
> # CONFIG_CRYPTO_PCRYPT is not set
> CONFIG_CRYPTO_WORKQUEUE=y
> # CONFIG_CRYPTO_CRYPTD is not set
> CONFIG_CRYPTO_AUTHENC=m
> # CONFIG_CRYPTO_TEST is not set
> 
> #
> # Authenticated Encryption with Associated Data
> #
> # CONFIG_CRYPTO_CCM is not set
> # CONFIG_CRYPTO_GCM is not set
> # CONFIG_CRYPTO_SEQIV is not set
> 
> #
> # Block modes
> #
> CONFIG_CRYPTO_CBC=m
> # CONFIG_CRYPTO_CTR is not set
> # CONFIG_CRYPTO_CTS is not set
> CONFIG_CRYPTO_ECB=m
> CONFIG_CRYPTO_LRW=m
> CONFIG_CRYPTO_PCBC=m
> # CONFIG_CRYPTO_XTS is not set
> 
> #
> # Hash modes
> #
> CONFIG_CRYPTO_HMAC=y
> CONFIG_CRYPTO_XCBC=m
> # CONFIG_CRYPTO_VMAC is not set
> 
> #
> # Digest
> #
> CONFIG_CRYPTO_CRC32C=y
> # CONFIG_CRYPTO_CRC32C_INTEL is not set
> # CONFIG_CRYPTO_GHASH is not set
> CONFIG_CRYPTO_MD4=m
> CONFIG_CRYPTO_MD5=y
> CONFIG_CRYPTO_MICHAEL_MIC=m
> # CONFIG_CRYPTO_RMD128 is not set
> # CONFIG_CRYPTO_RMD160 is not set
> # CONFIG_CRYPTO_RMD256 is not set
> # CONFIG_CRYPTO_RMD320 is not set
> CONFIG_CRYPTO_SHA1=y
> CONFIG_CRYPTO_SHA256=m
> CONFIG_CRYPTO_SHA512=m
> CONFIG_CRYPTO_TGR192=m
> CONFIG_CRYPTO_WP512=m
> # CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL is not set
> 
> #
> # Ciphers
> #
> CONFIG_CRYPTO_AES=m
> CONFIG_CRYPTO_AES_X86_64=m
> # CONFIG_CRYPTO_AES_NI_INTEL is not set
> CONFIG_CRYPTO_ANUBIS=m
> CONFIG_CRYPTO_ARC4=m
> CONFIG_CRYPTO_BLOWFISH=m
> CONFIG_CRYPTO_CAMELLIA=m
> CONFIG_CRYPTO_CAST5=m
> CONFIG_CRYPTO_CAST6=m
> CONFIG_CRYPTO_DES=m
> CONFIG_CRYPTO_FCRYPT=m
> CONFIG_CRYPTO_KHAZAD=m
> # CONFIG_CRYPTO_SALSA20 is not set
> # CONFIG_CRYPTO_SALSA20_X86_64 is not set
> # CONFIG_CRYPTO_SEED is not set
> CONFIG_CRYPTO_SERPENT=m
> CONFIG_CRYPTO_TEA=m
> CONFIG_CRYPTO_TWOFISH=m
> CONFIG_CRYPTO_TWOFISH_COMMON=m
> CONFIG_CRYPTO_TWOFISH_X86_64=m
> 
> #
> # Compression
> #
> CONFIG_CRYPTO_DEFLATE=m
> # CONFIG_CRYPTO_ZLIB is not set
> # CONFIG_CRYPTO_LZO is not set
> 
> #
> # Random Number Generation
> #
> # CONFIG_CRYPTO_ANSI_CPRNG is not set
> CONFIG_CRYPTO_HW=y
> # CONFIG_CRYPTO_DEV_PADLOCK is not set
> # CONFIG_CRYPTO_DEV_HIFN_795X is not set
> CONFIG_HAVE_KVM=y
> CONFIG_HAVE_KVM_IRQCHIP=y
> CONFIG_HAVE_KVM_EVENTFD=y
> CONFIG_KVM_APIC_ARCHITECTURE=y
> CONFIG_KVM_MMIO=y
> CONFIG_VIRTUALIZATION=y
> CONFIG_KVM=m
> CONFIG_KVM_INTEL=m
> CONFIG_KVM_AMD=m
> # CONFIG_VHOST_NET is not set
> # CONFIG_VIRTIO_PCI is not set
> # CONFIG_VIRTIO_BALLOON is not set
> CONFIG_BINARY_PRINTF=y
> 
> #
> # Library routines
> #
> CONFIG_RAID6_PQ=m
> CONFIG_BITREVERSE=y
> CONFIG_GENERIC_FIND_FIRST_BIT=y
> CONFIG_GENERIC_FIND_NEXT_BIT=y
> CONFIG_GENERIC_FIND_LAST_BIT=y
> CONFIG_CRC_CCITT=m
> CONFIG_CRC16=m
> CONFIG_CRC_T10DIF=y
> CONFIG_CRC_ITU_T=m
> CONFIG_CRC32=y
> # CONFIG_CRC7 is not set
> CONFIG_LIBCRC32C=y
> CONFIG_ZLIB_INFLATE=y
> CONFIG_ZLIB_DEFLATE=m
> CONFIG_LZO_DECOMPRESS=y
> CONFIG_DECOMPRESS_GZIP=y
> CONFIG_DECOMPRESS_BZIP2=y
> CONFIG_DECOMPRESS_LZMA=y
> CONFIG_DECOMPRESS_LZO=y
> CONFIG_TEXTSEARCH=y
> CONFIG_TEXTSEARCH_KMP=m
> CONFIG_TEXTSEARCH_BM=m
> CONFIG_TEXTSEARCH_FSM=m
> CONFIG_HAS_IOMEM=y
> CONFIG_HAS_IOPORT=y
> CONFIG_HAS_DMA=y
> CONFIG_NLATTR=y

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Wed, Aug 25, 2010 at 04:11:06PM -0400, Don Zickus wrote:
...
> >  Uhhuh. NMI received for unknown reason 00 on CPU 15.
> >  Do you have a strange power saving mode enabled?
> >  Dazed and confused, but trying to continue
> 
> So I found a Nehalem box that can reliably reproduce Ingo's problem using
> something as simple 'perf top'.  But like above, I am noticing the
> samething, an extra NMI(PMI??) that comes out of nowhere.
> 
> Looking at the data above the delta between nmis is very small compared to
> the other nmis.  It almost suggests that this is an extra PMI.
> Considering there is already two cpu errata discussing extra PMIs under
> certain configurations, I wouldn't be surprised if this was a third.
> 
> Cheers,
> Don
> 

Oh. I'm not sure if it would be a good idea at all but maybe we could
use kind of Robert's idea about "pmu nmi relaxing time" ie some time
slice in which we treat nmi's as being from pmu, but not arbitrary number
but equal to the number of PMI turned off. Say we handle NMI and found
that 4 events are overflowed, we clear them, arm timer and wait for
3 unknow nmis to happen, if they are not happening during some time
period we clear this waitqueue, if they happen or partially happen
- we destroy the timer. Ie almost the same as Robert's idea but
without tsc? Just a thought.

	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Thu, Aug 26, 2010 at 12:24:58AM +0400, Cyrill Gorcunov wrote:
> On Wed, Aug 25, 2010 at 04:11:06PM -0400, Don Zickus wrote:
> ...
> > >  Uhhuh. NMI received for unknown reason 00 on CPU 15.
> > >  Do you have a strange power saving mode enabled?
> > >  Dazed and confused, but trying to continue
> > 
> > So I found a Nehalem box that can reliably reproduce Ingo's problem using
> > something as simple 'perf top'.  But like above, I am noticing the
> > samething, an extra NMI(PMI??) that comes out of nowhere.
> > 
> > Looking at the data above the delta between nmis is very small compared to
> > the other nmis.  It almost suggests that this is an extra PMI.
> > Considering there is already two cpu errata discussing extra PMIs under
> > certain configurations, I wouldn't be surprised if this was a third.
> > 
> > Cheers,
> > Don
> > 
> 
> Oh. I'm not sure if it would be a good idea at all but maybe we could
> use kind of Robert's idea about "pmu nmi relaxing time" ie some time
> slice in which we treat nmi's as being from pmu, but not arbitrary number
> but equal to the number of PMI turned off. Say we handle NMI and found
> that 4 events are overflowed, we clear them, arm timer and wait for
> 3 unknow nmis to happen, if they are not happening during some time
> period we clear this waitqueue, if they happen or partially happen
> - we destroy the timer. Ie almost the same as Robert's idea but
> without tsc? Just a thought.

The only problem is only one counter is overflowing in these cases, so we
would have to do it all the time, which may not be hard.  But I was
thinking of something similar.

For now, I am trying to force counter0 off, seeing that most of the perf
errata on nehalem have been on counter0.  Or maybe I can get 'perf top' to
use something other than counter0 by running 'perf record' first?

Cheers,
Don

> 
> 	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Wed, Aug 25, 2010 at 05:20:37PM -0400, Don Zickus wrote:
... 
> The only problem is only one counter is overflowing in these cases, so we
> would have to do it all the time, which may not be hard.  But I was
> thinking of something similar.
> 
> For now, I am trying to force counter0 off, seeing that most of the perf
> errata on nehalem have been on counter0.  Or maybe I can get 'perf top' to
> use something other than counter0 by running 'perf record' first?
> 
> Cheers,
> Don
> 
> > 
> > 	-- Cyrill
> 

Well, I never looked deep inside perf tool code itself. Perhaps the easiest
(if you want to disable counter0) is just disable it in kernel and see how
it goes, no?

	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 25.08.10 16:11:06, Don Zickus wrote:
> >  cpu #15, nmi #160, marked #0, handled = 1, time = 333392635730, delta = 11238255
> >  cpu #15, nmi #161, marked #0, handled = 1, time = 333403779380, delta = 11143650
> >  cpu #15, nmi #162, marked #0, handled = 1, time = 333415418497, delta = 11639117
> >  cpu #15, nmi #163, marked #0, handled = 1, time = 333415467084, delta = 48587
> >  cpu #15, nmi #164, marked #0, handled = 1, time = 333415501531, delta = 34447
> >  cpu #15, nmi #165, marked #0, handled = 1, time = 333459918106, delta = 44416575
> >  cpu #15, nmi #166, marked #0, handled = 0, time = 333459923167, delta = 1666
> >  cpu #15, nmi #151, marked #0, handled = 1, time = 332978597882, delta = 11447002
> >  cpu #15, nmi #152, marked #0, handled = 1, time = 332978657151, delta = 59269
> >  cpu #15, nmi #153, marked #0, handled = 1, time = 332978667847, delta = 10696
> >  cpu #15, nmi #154, marked #0, handled = 1, time = 333023125757, delta = 44457910
> >  cpu #15, nmi #155, marked #0, handled = 1, time = 333291980833, delta = 268855076
> >  cpu #15, nmi #156, marked #0, handled = 1, time = 333325663125, delta = 33682292
> >  cpu #15, nmi #157, marked #0, handled = 1, time = 333348216481, delta = 22553356
> >  cpu #15, nmi #158, marked #0, handled = 1, time = 333370168887, delta = 21952406
> >  cpu #15, nmi #159, marked #0, handled = 1, time = 333381397475, delta = 11228588
> >  Uhhuh. NMI received for unknown reason 00 on CPU 15.
> >  Do you have a strange power saving mode enabled?
> >  Dazed and confused, but trying to continue
> 
> So I found a Nehalem box that can reliably reproduce Ingo's problem using
> something as simple 'perf top'.  But like above, I am noticing the
> samething, an extra NMI(PMI??) that comes out of nowhere.

This could also be a race in the counter handling code, or we do not
proper count the number of handled counters. Maybe 2 counters actually
fired but we only noticed one counter and then accidentially cleared
the 2nd without processing it.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Thu, Aug 26, 2010 at 1:00 PM, Robert Richter <robert.richter@amd.com> wrote:
...
>
> This could also be a race in the counter handling code, or we do not
> proper count the number of handled counters. Maybe 2 counters actually
> fired but we only noticed one counter and then accidentially cleared
> the 2nd without processing it.
>
> -Robert
>

Any chance to get it tested on P4 machine since it has a bit
different design?

Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Thu, Aug 26, 2010 at 01:18:29PM +0400, Cyrill Gorcunov wrote:
> On Thu, Aug 26, 2010 at 1:00 PM, Robert Richter <robert.richter@amd.com> wrote:
> ...
> >
> > This could also be a race in the counter handling code, or we do not
> > proper count the number of handled counters. Maybe 2 counters actually
> > fired but we only noticed one counter and then accidentially cleared
> > the 2nd without processing it.
> >
> > -Robert
> >
> 
> Any chance to get it tested on P4 machine since it has a bit
> different design?

Well P4 uses a different pmu irq handler, so I am not sure it will give us
much insight.  I haven't noticed this on an AMD or an intel i5 either.

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Thu, Aug 26, 2010 at 01:18:29PM +0400, Cyrill Gorcunov wrote:
> On Thu, Aug 26, 2010 at 1:00 PM, Robert Richter <robert.richter@amd.com> wrote:
> ...
> >
> > This could also be a race in the counter handling code, or we do not
> > proper count the number of handled counters. Maybe 2 counters actually
> > fired but we only noticed one counter and then accidentially cleared
> > the 2nd without processing it.
> >
> > -Robert
> >
> 
> Any chance to get it tested on P4 machine since it has a bit
> different design?

Hmm, I take that back.  I guess I can reproduce this on my i5 that I had
using Ingo's config.

Working on Robert's assumption, I added code to perf_event_intel.c that
said if handled !=0 just add one to it (IOW always process handled as 0 or
 something >1).  That seems to working good and catches the nmis that Ingo
was seeing.

I'll keep looking for the race condition to better fix it.

Cheers,
Don

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Thu, Aug 26, 2010 at 11:22:46AM -0400, Don Zickus wrote:
> On Thu, Aug 26, 2010 at 01:18:29PM +0400, Cyrill Gorcunov wrote:
> > On Thu, Aug 26, 2010 at 1:00 PM, Robert Richter <robert.richter@amd.com> wrote:
> > ...
> > >
> > > This could also be a race in the counter handling code, or we do not
> > > proper count the number of handled counters. Maybe 2 counters actually
> > > fired but we only noticed one counter and then accidentially cleared
> > > the 2nd without processing it.
> > >
> > > -Robert
> > >
> > 
> > Any chance to get it tested on P4 machine since it has a bit
> > different design?
> 
> Hmm, I take that back.  I guess I can reproduce this on my i5 that I had
> using Ingo's config.
> 
> Working on Robert's assumption, I added code to perf_event_intel.c that
> said if handled !=0 just add one to it (IOW always process handled as 0 or
>  something >1).  That seems to working good and catches the nmis that Ingo
> was seeing.
> 
> I'll keep looking for the race condition to better fix it.
> 
> Cheers,
> Don
> 

Sounds promising, mind to post new inter-diff? Ie what you have
changed from Robert's patch.

	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Thu, Aug 26, 2010 at 07:34:04PM +0400, Cyrill Gorcunov wrote:
> > I'll keep looking for the race condition to better fix it.
> > 
> > Cheers,
> > Don
> > 
> 
> Sounds promising, mind to post new inter-diff? Ie what you have
> changed from Robert's patch.

It was really in Peter's patch, just a stupid hack for now.

Cheers,
Don


diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
index 4539b4b..9e65a7b 100644
--- a/arch/x86/kernel/cpu/perf_event_intel.c
+++ b/arch/x86/kernel/cpu/perf_event_intel.c
@@ -777,7 +777,9 @@ again:
 
 done:
 	intel_pmu_enable_all(0);
-	return handled;
+	if (!handled)
+		return handled;
+	return ++handled;
 }
 
 static struct event_constraint *

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Cyrill Gorcunov]

On Thu, Aug 26, 2010 at 12:40:31PM -0400, Don Zickus wrote:
> On Thu, Aug 26, 2010 at 07:34:04PM +0400, Cyrill Gorcunov wrote:
> > > I'll keep looking for the race condition to better fix it.
> > > 
> > > Cheers,
> > > Don
> > > 
> > 
> > Sounds promising, mind to post new inter-diff? Ie what you have
> > changed from Robert's patch.
> 
> It was really in Peter's patch, just a stupid hack for now.
> 
> Cheers,
> Don
> 
> 
> diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
> index 4539b4b..9e65a7b 100644
> --- a/arch/x86/kernel/cpu/perf_event_intel.c
> +++ b/arch/x86/kernel/cpu/perf_event_intel.c
> @@ -777,7 +777,9 @@ again:
>  
>  done:
>  	intel_pmu_enable_all(0);
> -	return handled;
> +	if (!handled)
> +		return handled;
> +	return ++handled;
>  }
>  
>  static struct event_constraint *
> 

ok, it seems it just treat any unknown nmi as being came from PMU, no?

	-- Cyrill

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 26.08.10 14:02:50, Cyrill Gorcunov wrote:
> > diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
> > index 4539b4b..9e65a7b 100644
> > --- a/arch/x86/kernel/cpu/perf_event_intel.c
> > +++ b/arch/x86/kernel/cpu/perf_event_intel.c
> > @@ -777,7 +777,9 @@ again:
> >  
> >  done:
> >  	intel_pmu_enable_all(0);
> > -	return handled;
> > +	if (!handled)
> > +		return handled;
> > +	return ++handled;
> >  }
> >  
> >  static struct event_constraint *
> > 
> 
> ok, it seems it just treat any unknown nmi as being came from PMU, no?

Yes, this just throws away all unknown nmis after a perf nmi. It
disables unknown nmi detection on this cpu type.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Peter Zijlstra]

On Fri, 2010-08-27 at 09:57 +0200, Robert Richter wrote:
> On 26.08.10 14:02:50, Cyrill Gorcunov wrote:
> > > diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
> > > index 4539b4b..9e65a7b 100644
> > > --- a/arch/x86/kernel/cpu/perf_event_intel.c
> > > +++ b/arch/x86/kernel/cpu/perf_event_intel.c
> > > @@ -777,7 +777,9 @@ again:
> > >  
> > >  done:
> > >  	intel_pmu_enable_all(0);
> > > -	return handled;
> > > +	if (!handled)
> > > +		return handled;
> > > +	return ++handled;
> > >  }
> > >  
> > >  static struct event_constraint *
> > > 
> > 
> > ok, it seems it just treat any unknown nmi as being came from PMU, no?
> 
> Yes, this just throws away all unknown nmis after a perf nmi. It
> disables unknown nmi detection on this cpu type.

Wouldn't returning 2 be more sensible, then it would only eat a few
unknowns after each pmi? (Still assuming you return 0 when there really
was nothing to do)

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 27.08.10 04:11:21, Peter Zijlstra wrote:
> On Fri, 2010-08-27 at 09:57 +0200, Robert Richter wrote:
> > On 26.08.10 14:02:50, Cyrill Gorcunov wrote:
> > > > diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
> > > > index 4539b4b..9e65a7b 100644
> > > > --- a/arch/x86/kernel/cpu/perf_event_intel.c
> > > > +++ b/arch/x86/kernel/cpu/perf_event_intel.c
> > > > @@ -777,7 +777,9 @@ again:
> > > >  
> > > >  done:
> > > >  	intel_pmu_enable_all(0);
> > > > -	return handled;
> > > > +	if (!handled)
> > > > +		return handled;
> > > > +	return ++handled;
> > > >  }
> > > >  
> > > >  static struct event_constraint *
> > > > 
> > > 
> > > ok, it seems it just treat any unknown nmi as being came from PMU, no?
> > 
> > Yes, this just throws away all unknown nmis after a perf nmi. It
> > disables unknown nmi detection on this cpu type.
> 
> Wouldn't returning 2 be more sensible, then it would only eat a few
> unknowns after each pmi? (Still assuming you return 0 when there really
> was nothing to do)

Yes, this would be the best workaround for cpus where the detection
logic does not work properly. But I think Don found a solution
already.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

On 25.08.10 06:41:30, Ingo Molnar wrote:

> > > > > Uhhuh. NMI received for unknown reason 00 on CPU 9.
> > > > > Do you have a strange power saving mode enabled?
> > > > > Dazed and confused, but trying to continue
> 
> Before the two patches i did not get these messages at all, on any of 
> the systems.

Yes, because all were eaten by NOTIFY_STOP, and that's why we also
lose NMIs from other sources, like the nmi button. The patches try to
fix this.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Ingo Molnar]

* Robert Richter <robert.richter@amd.com> wrote:

> On 25.08.10 06:41:30, Ingo Molnar wrote:
> 
> > > > > > Uhhuh. NMI received for unknown reason 00 on CPU 9.
> > > > > > Do you have a strange power saving mode enabled?
> > > > > > Dazed and confused, but trying to continue
> > 
> > Before the two patches i did not get these messages at all, on any 
> > of the systems.
> 
> Yes, because all were eaten by NOTIFY_STOP, and that's why we also 
> lose NMIs from other sources, like the nmi button. The patches try to 
> fix this.

I understand that, but obviously the patches need to fix the lack of 
messages without introducing new regressions: such as messages not seen 
before and CPUs with stuck NMIs.

Thanks,

	Ingo

Re: [PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Don Zickus]

On Fri, Aug 20, 2010 at 11:05:42AM -0400, Don Zickus wrote:
> I'll test tip later today to see if I can reproduce it.
> 
> Cheers,
> Don

Sad to say, that won't happen.  Both my amd box and nehalem box have to
many issues with your master branch.

The amd box BUGs in perf_event_nmi_handler on the new code trying to run
'perf top'

arch/x86/kernel/cpu/perf_event.c::perf_event_nmi_handler:1250

((__get_cpu_var(nmi).marked == this_nmi) &&

The BUG is attached below.  I can't figure out why

And my Nehalem box won't even boot with the that kernel, not even to
console for some reason.  Then bisecting revealed that in 2.6.35 something
with LVM changed such that the kernel can't mount my RHEL-6 lvm
partitions.  So even if I did get that kernel booting it won't mount
disks.

I'll take this as a sign to quit for now.. and try again on Monday. :-)

Cheers,
Don

-----



amd-ma78gm-01.rhts.eng.bos.redhat.com login: BUG: unable to handle kernel
paging request at ffff87ff838a5200
IP: [<ffffffff814a6370>] perf_event_nmi_handler+0xd0/0xe0
PGD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 0 
Modules linked in: autofs4 sunrpc cpufreq_ondemand powernow_k8 freq_table
mperf ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport wmi
snd_hda_codec_atihdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec
snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore
snd_page_alloc pcspkr serio_raw edac_core edac_mce_amd sg i2c_piix4 r8169
mii ahci libahci shpchp ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif
firewire_ohci firewire_core crc_itu_t ata_generic pata_acpi pata_atiixp
floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core dm_mod
[last unloaded: scsi_wait_scan]

Pid: 1865, comm: perf Not tainted 2.6.36-rc1tipperf-tip+ #28
GA-MA78GM-S2H/GA-MA78GM-S2H
RIP: 0010:[<ffffffff814a6370>]  [<ffffffff814a6370>]
perf_event_nmi_handler+0xd0/0xe0
RSP: 0018:ffff880002407e88  EFLAGS: 00010046
RAX: 0000000000000001 RBX: 000000000000000c RCX: ffffffff814a5200
RDX: ffffffff814a5200 RSI: 0000000000000001 RDI: ffff880002400000
RBP: ffff880002407e98 R08: 0000000000000001 R09: ffff880002407d48
R10: 0000000000000002 R11: 0000000000000000 R12: ffff880002407ef8
R13: 00000000fffffffc R14: 0000000000000000 R15: ffffffff81c1df80
FS:  00007f0220ac9700(0000) GS:ffff880002400000(0000)
knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff87ff838a5200 CR3: 0000000222c31000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process perf (pid: 1865, threadinfo ffff880220a78000, task
ffff88021c64a100)
Stack:
 0000000000000000 ffff880002407ef8 ffff880002407ed8 ffffffff814a8505
<0> 0000000000000000 ffff880002407f58 000000000000003d 000000000000003d
<0> ffff88000240ccc0 0000000000000001 ffff880002407ee8 ffffffff814a856a
Call Trace:
 <NMI> 
 [<ffffffff814a8505>] notifier_call_chain+0x55/0x80
 [<ffffffff814a856a>] atomic_notifier_call_chain+0x1a/0x20
 [<ffffffff814a859e>] notify_die+0x2e/0x30
 [<ffffffff814a5963>] do_nmi+0x173/0x2b0
 [<ffffffff814a5220>] nmi+0x20/0x30
 [<ffffffff810340fa>] ? native_write_msr_safe+0xa/0x10
 <<EOE>> 
 [<ffffffff8101a4b0>] x86_pmu_enable_all+0x60/0x80
 [<ffffffff8101b72c>] hw_perf_enable+0xfc/0x230
 [<ffffffff810eb1dd>] perf_enable+0x2d/0x40
 [<ffffffff810ed76d>] __perf_install_in_context+0xcd/0x190
 [<ffffffff810ed6a0>] ? __perf_install_in_context+0x0/0x190
 [<ffffffff810953bc>] smp_call_function_single+0x8c/0x160
 [<ffffffff810f23a8>] ? find_get_context+0x98/0x2b0
 [<ffffffff810edbba>] perf_install_in_context+0x9a/0xa0
 [<ffffffff810f3141>] sys_perf_event_open+0x361/0x4f0
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: 53 01 00 48 c7 c0 00 52 4a 81 65 48 8b 14 25 38 e3 00 00 3b 0c 02 0f
85 67 ff ff ff b8 01 80 00 00 c9 c3 0f 1f 84 00 00 00 00 00 <83> 3c 0f 01
74 a6 eb e9 00 00 00 00 00 00 00 00 55 48 89 e5 48 
RIP  [<ffffffff814a6370>] perf_event_nmi_handler+0xd0/0xe0
 RSP <ffff880002407e88>
CR2: ffff87ff838a5200
---[ end trace 3ddcb8e2da2c4430 ]---

> 
> 
> Ingo Molnar <mingo@elte.hu> wrote:
> 

> 
> it's not working so well, i'm getting:
> 
>  Uhhuh. NMI received for unknown reason 00 on CPU 9.
>  Do you have a strange power saving mode enabled?
>  Dazed and confused, but trying to continue
> 
> on a nehalem box, after a perf top and perf stat run.
> 
> Thanks,
> 
> 	Ingo